| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one or more |
| contributor license agreements. See the NOTICE file distributed with |
| this work for additional information regarding copyright ownership. |
| The ASF licenses this file to You under the Apache License, Version 2.0 |
| (the "License"); you may not use this file except in compliance with |
| the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="UTF-8"> |
| <!--[if IE]><meta http-equiv="X-UA-Compatible" content="IE=edge"><![endif]--> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <meta name="generator" content="Asciidoctor 1.5.8"> |
| <meta name="author" content="Apache NiFi Team"> |
| <title>Apache NiFi User Guide</title> |
| <style> |
| /* Asciidoctor default stylesheet | MIT License | http://asciidoctor.org */ |
| /* Copyright (C) 2012-2015 Dan Allen, Ryan Waldron and the Asciidoctor Project |
| |
| Permission is hereby granted, free of charge, to any person obtaining a copy |
| of this software and associated documentation files (the "Software"), to deal |
| in the Software without restriction, including without limitation the rights |
| to use, copy, modify, merge, publish, distribute, sublicense, and/or sell |
| copies of the Software, and to permit persons to whom the Software is |
| furnished to do so, subject to the following conditions: |
| |
| The above copyright notice and this permission notice shall be included in |
| all copies or substantial portions of the Software. |
| |
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR |
| IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, |
| FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE |
| AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER |
| LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, |
| OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN |
| THE SOFTWARE. */ |
| /* Remove the comments around the @import statement below when using this as a custom stylesheet */ |
| @import "https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400"; |
| article,aside,details,figcaption,figure,footer,header,hgroup,main,nav,section,summary{display:block} |
| audio,canvas,video{display:inline-block} |
| audio:not([controls]){display:none;height:0} |
| [hidden],template{display:none} |
| script{display:none!important} |
| html{font-family:sans-serif;-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%} |
| body{margin:0} |
| a{background:transparent} |
| a:focus{outline:thin dotted} |
| a:active,a:hover{outline:0} |
| h1{font-size:2em;margin:.67em 0} |
| abbr[title]{border-bottom:1px dotted} |
| b,strong{font-weight:bold} |
| dfn{font-style:italic} |
| hr{-moz-box-sizing:content-box;box-sizing:content-box;height:0} |
| mark{background:#ff0;color:#000} |
| code,kbd,pre,samp{font-family:monospace;font-size:1em} |
| pre{white-space:pre-wrap} |
| q{quotes:"\201C" "\201D" "\2018" "\2019"} |
| small{font-size:80%} |
| sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline} |
| sup{top:-.5em} |
| sub{bottom:-.25em} |
| img{border:0} |
| svg:not(:root){overflow:hidden} |
| figure{margin:0} |
| fieldset{border:1px solid silver;margin:0 2px;padding:.35em .625em .75em} |
| legend{border:0;padding:0} |
| button,input,select,textarea{font-family:inherit;font-size:100%;margin:0} |
| button,input{line-height:normal} |
| button,select{text-transform:none} |
| button,html input[type="button"],input[type="reset"],input[type="submit"]{-webkit-appearance:button;cursor:pointer} |
| button[disabled],html input[disabled]{cursor:default} |
| input[type="checkbox"],input[type="radio"]{box-sizing:border-box;padding:0} |
| input[type="search"]{-webkit-appearance:textfield;-moz-box-sizing:content-box;-webkit-box-sizing:content-box;box-sizing:content-box} |
| input[type="search"]::-webkit-search-cancel-button,input[type="search"]::-webkit-search-decoration{-webkit-appearance:none} |
| button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0} |
| textarea{overflow:auto;vertical-align:top} |
| table{border-collapse:collapse;border-spacing:0} |
| *,*:before,*:after{-moz-box-sizing:border-box;-webkit-box-sizing:border-box;box-sizing:border-box} |
| html,body{font-size:100%} |
| body{background:#fff;color:rgba(0,0,0,.8);padding:0;margin:0;font-family:"Noto Serif","DejaVu Serif",serif;font-weight:400;font-style:normal;line-height:1;position:relative;cursor:auto} |
| a:hover{cursor:pointer} |
| img,object,embed{max-width:100%;height:auto} |
| object,embed{height:100%} |
| img{-ms-interpolation-mode:bicubic} |
| #map_canvas img,#map_canvas embed,#map_canvas object,.map_canvas img,.map_canvas embed,.map_canvas object{max-width:none!important} |
| .left{float:left!important} |
| .right{float:right!important} |
| .text-left{text-align:left!important} |
| .text-right{text-align:right!important} |
| .text-center{text-align:center!important} |
| .text-justify{text-align:justify!important} |
| .hide{display:none} |
| .antialiased,body{-webkit-font-smoothing:antialiased} |
| img{display:inline-block;vertical-align:middle} |
| textarea{height:auto;min-height:50px} |
| select{width:100%} |
| p.lead,.paragraph.lead>p,#preamble>.sectionbody>.paragraph:first-of-type p{font-size:1.21875em;line-height:1.6} |
| .subheader,.admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{line-height:1.45;color:#7a2518;font-weight:400;margin-top:0;margin-bottom:.25em} |
| div,dl,dt,dd,ul,ol,li,h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6,pre,form,p,blockquote,th,td{margin:0;padding:0;direction:ltr} |
| a{color:#2156a5;text-decoration:underline;line-height:inherit} |
| a:hover,a:focus{color:#1d4b8f} |
| a img{border:none} |
| p{font-family:inherit;font-weight:400;font-size:1em;line-height:1.6;margin-bottom:1.25em;text-rendering:optimizeLegibility} |
| p aside{font-size:.875em;line-height:1.35;font-style:italic} |
| h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{font-family:"Open Sans","DejaVu Sans",sans-serif;font-weight:300;font-style:normal;color:#ba3925;text-rendering:optimizeLegibility;margin-top:1em;margin-bottom:.5em;line-height:1.0125em} |
| h1 small,h2 small,h3 small,#toctitle small,.sidebarblock>.content>.title small,h4 small,h5 small,h6 small{font-size:60%;color:#e99b8f;line-height:0} |
| h1{font-size:2.125em} |
| h2{font-size:1.6875em} |
| h3,#toctitle,.sidebarblock>.content>.title{font-size:1.375em} |
| h4,h5{font-size:1.125em} |
| h6{font-size:1em} |
| hr{border:solid #ddddd8;border-width:1px 0 0;clear:both;margin:1.25em 0 1.1875em;height:0} |
| em,i{font-style:italic;line-height:inherit} |
| strong,b{font-weight:bold;line-height:inherit} |
| small{font-size:60%;line-height:inherit} |
| code{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;font-weight:400;color:rgba(0,0,0,.9);padding-right: 1px;} |
| ul,ol,dl{font-size:1em;line-height:1.6;margin-bottom:1.25em;list-style-position:outside;font-family:inherit} |
| ul,ol,ul.no-bullet,ol.no-bullet{margin-left:1.5em} |
| ul li ul,ul li ol{margin-left:1.25em;margin-bottom:0;font-size:1em} |
| ul.square li ul,ul.circle li ul,ul.disc li ul{list-style:inherit} |
| ul.square{list-style-type:square} |
| ul.circle{list-style-type:circle} |
| ul.disc{list-style-type:disc} |
| ul.no-bullet{list-style:none} |
| ol li ul,ol li ol{margin-left:1.25em;margin-bottom:0} |
| dl dt{margin-bottom:.3125em;font-weight:bold} |
| dl dd{margin-bottom:1.25em} |
| abbr,acronym{text-transform:uppercase;font-size:90%;color:rgba(0,0,0,.8);border-bottom:1px dotted #ddd;cursor:help} |
| abbr{text-transform:none} |
| blockquote{margin:0 0 1.25em;padding:.5625em 1.25em 0 1.1875em;border-left:1px solid #ddd} |
| blockquote cite{display:block;font-size:.9375em;color:rgba(0,0,0,.6)} |
| blockquote cite:before{content:"\2014 \0020"} |
| blockquote cite a,blockquote cite a:visited{color:rgba(0,0,0,.6)} |
| blockquote,blockquote p{line-height:1.6;color:rgba(0,0,0,.85)} |
| @media only screen and (min-width:768px){h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2} |
| h1{font-size:2.75em} |
| h2{font-size:2.3125em} |
| h3,#toctitle,.sidebarblock>.content>.title{font-size:1.6875em} |
| h4{font-size:1.4375em}}table{background:#fff;margin-bottom:1.25em;border:solid 1px #dedede} |
| table thead,table tfoot{background:#f7f8f7;font-weight:bold} |
| table thead tr th,table thead tr td,table tfoot tr th,table tfoot tr td{padding:.5em .625em .625em;font-size:inherit;color:rgba(0,0,0,.8);text-align:left} |
| table tr th,table tr td{padding:.5625em .625em;font-size:inherit;color:rgba(0,0,0,.8)} |
| table tr.even,table tr.alt,table tr:nth-of-type(even){background:#f8f8f7} |
| table thead tr th,table tfoot tr th,table tbody tr td,table tr td,table tfoot tr td{display:table-cell;line-height:1.6} |
| h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2;word-spacing:-.05em} |
| h1 strong,h2 strong,h3 strong,#toctitle strong,.sidebarblock>.content>.title strong,h4 strong,h5 strong,h6 strong{font-weight:400} |
| .clearfix:before,.clearfix:after,.float-group:before,.float-group:after{content:" ";display:table} |
| .clearfix:after,.float-group:after{clear:both} |
| *:not(pre)>code{font-size:.9375em;font-style:normal!important;letter-spacing:0;word-spacing:-.15em;background-color:#f7f7f8;-webkit-border-radius:4px;border-radius:4px;line-height:1.45;text-rendering:optimizeSpeed} |
| pre,pre>code{line-height:1.45;color:rgba(0,0,0,.9);font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;font-weight:400;text-rendering:optimizeSpeed} |
| .keyseq{color:rgba(51,51,51,.8)} |
| kbd{display:inline-block;color:rgba(0,0,0,.8);font-size:.75em;line-height:1.4;background-color:#f7f7f7;border:1px solid #ccc;-webkit-border-radius:3px;border-radius:3px;-webkit-box-shadow:0 1px 0 rgba(0,0,0,.2),0 0 0 .1em white inset;box-shadow:0 1px 0 rgba(0,0,0,.2),0 0 0 .1em #fff inset;margin:-.15em .15em 0 .15em;padding:.2em .6em .2em .5em;vertical-align:middle;white-space:nowrap} |
| .keyseq kbd:first-child{margin-left:0} |
| .keyseq kbd:last-child{margin-right:0} |
| .menuseq,.menu{color:rgba(0,0,0,.8)} |
| b.button:before,b.button:after{position:relative;top:-1px;font-weight:400} |
| b.button:before{content:"[";padding:0 3px 0 2px} |
| b.button:after{content:"]";padding:0 2px 0 3px} |
| p a>code:hover{color:rgba(0,0,0,.9)} |
| #header,#content,#footnotes,#footer{width:100%;margin-left:auto;margin-right:auto;margin-top:0;margin-bottom:0;max-width:62.5em;*zoom:1;position:relative;padding-left:.9375em;padding-right:.9375em} |
| #header:before,#header:after,#content:before,#content:after,#footnotes:before,#footnotes:after,#footer:before,#footer:after{content:" ";display:table} |
| #header:after,#content:after,#footnotes:after,#footer:after{clear:both} |
| #content{margin-top:1.25em} |
| #content:before{content:none} |
| #header>h1:first-child{color:rgba(0,0,0,.85);margin-top:2.25rem;margin-bottom:0} |
| #header>h1:first-child+#toc{margin-top:8px;border-top:1px solid #ddddd8} |
| #header>h1:only-child,body.toc2 #header>h1:nth-last-child(2){border-bottom:1px solid #ddddd8;padding-bottom:8px} |
| #header .details{border-bottom:1px solid #ddddd8;line-height:1.45;padding-top:.25em;padding-bottom:.25em;padding-left:.25em;color:rgba(0,0,0,.6);display:-ms-flexbox;display:-webkit-flex;display:flex;-ms-flex-flow:row wrap;-webkit-flex-flow:row wrap;flex-flow:row wrap} |
| #header .details span:first-child{margin-left:-.125em} |
| #header .details span.email a{color:rgba(0,0,0,.85)} |
| #header .details br{display:none} |
| #header .details br+span:before{content:"\00a0\2013\00a0"} |
| #header .details br+span.author:before{content:"\00a0\22c5\00a0";color:rgba(0,0,0,.85)} |
| #header .details br+span#revremark:before{content:"\00a0|\00a0"} |
| #header #revnumber{text-transform:capitalize} |
| #header #revnumber:after{content:"\00a0"} |
| #content>h1:first-child:not([class]){color:rgba(0,0,0,.85);border-bottom:1px solid #ddddd8;padding-bottom:8px;margin-top:0;padding-top:1rem;margin-bottom:1.25rem} |
| #toc{border-bottom:1px solid #efefed;padding-bottom:.5em} |
| #toc>ul{margin-left:.125em} |
| #toc ul.sectlevel0>li>a{font-style:italic} |
| #toc ul.sectlevel0 ul.sectlevel1{margin:.5em 0} |
| #toc ul{font-family:"Open Sans","DejaVu Sans",sans-serif;list-style-type:none} |
| #toc a{text-decoration:none} |
| #toc a:active{text-decoration:underline} |
| #toctitle{color:#7a2518;font-size:1.2em} |
| @media only screen and (min-width:768px){#toctitle{font-size:1.375em} |
| body.toc2{padding-left:15em;padding-right:0} |
| #toc.toc2{margin-top:0!important;background-color:#f8f8f7;position:fixed;width:15em;left:0;top:0;border-right:1px solid #efefed;border-top-width:0!important;border-bottom-width:0!important;z-index:1000;padding:1.25em 1em;height:100%;overflow:auto} |
| #toc.toc2 #toctitle{margin-top:0;font-size:1.2em} |
| #toc.toc2>ul{font-size:.9em;margin-bottom:0} |
| #toc.toc2 ul ul{margin-left:0;padding-left:1em} |
| #toc.toc2 ul.sectlevel0 ul.sectlevel1{padding-left:0;margin-top:.5em;margin-bottom:.5em} |
| body.toc2.toc-right{padding-left:0;padding-right:15em} |
| body.toc2.toc-right #toc.toc2{border-right-width:0;border-left:1px solid #efefed;left:auto;right:0}}@media only screen and (min-width:1280px){body.toc2{padding-left:20em;padding-right:0} |
| #toc.toc2{width:20em} |
| #toc.toc2 #toctitle{font-size:1.375em} |
| #toc.toc2>ul{font-size:.95em} |
| #toc.toc2 ul ul{padding-left:1.25em} |
| body.toc2.toc-right{padding-left:0;padding-right:20em}}#content #toc{border-style:solid;border-width:1px;border-color:#e0e0dc;margin-bottom:1.25em;padding:1.25em;background:#f8f8f7;-webkit-border-radius:4px;border-radius:4px} |
| #content #toc>:first-child{margin-top:0} |
| #content #toc>:last-child{margin-bottom:0} |
| #footer{max-width:100%;background-color:rgba(0,0,0,.8);padding:1.25em} |
| #footer-text{color:rgba(255,255,255,.8);line-height:1.44} |
| .sect1{padding-bottom:.625em} |
| @media only screen and (min-width:768px){.sect1{padding-bottom:1.25em}}.sect1+.sect1{border-top:1px solid #efefed} |
| #content h1>a.anchor,h2>a.anchor,h3>a.anchor,#toctitle>a.anchor,.sidebarblock>.content>.title>a.anchor,h4>a.anchor,h5>a.anchor,h6>a.anchor{position:absolute;z-index:1001;width:1.5ex;margin-left:-1.5ex;display:block;text-decoration:none!important;visibility:hidden;text-align:center;font-weight:400} |
| #content h1>a.anchor:before,h2>a.anchor:before,h3>a.anchor:before,#toctitle>a.anchor:before,.sidebarblock>.content>.title>a.anchor:before,h4>a.anchor:before,h5>a.anchor:before,h6>a.anchor:before{content:"\00A7";font-size:.85em;display:block;padding-top:.1em} |
| #content h1:hover>a.anchor,#content h1>a.anchor:hover,h2:hover>a.anchor,h2>a.anchor:hover,h3:hover>a.anchor,#toctitle:hover>a.anchor,.sidebarblock>.content>.title:hover>a.anchor,h3>a.anchor:hover,#toctitle>a.anchor:hover,.sidebarblock>.content>.title>a.anchor:hover,h4:hover>a.anchor,h4>a.anchor:hover,h5:hover>a.anchor,h5>a.anchor:hover,h6:hover>a.anchor,h6>a.anchor:hover{visibility:visible} |
| #content h1>a.link,h2>a.link,h3>a.link,#toctitle>a.link,.sidebarblock>.content>.title>a.link,h4>a.link,h5>a.link,h6>a.link{color:#ba3925;text-decoration:none} |
| #content h1>a.link:hover,h2>a.link:hover,h3>a.link:hover,#toctitle>a.link:hover,.sidebarblock>.content>.title>a.link:hover,h4>a.link:hover,h5>a.link:hover,h6>a.link:hover{color:#a53221} |
| .audioblock,.imageblock,.literalblock,.listingblock,.stemblock,.videoblock{margin-bottom:1.25em} |
| .admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{text-rendering:optimizeLegibility;text-align:left;font-family:"Noto Serif","DejaVu Serif",serif;font-size:1rem;font-style:italic} |
| table.tableblock>caption.title{white-space:nowrap;overflow:visible;max-width:0} |
| .paragraph.lead>p,#preamble>.sectionbody>.paragraph:first-of-type p{color:rgba(0,0,0,.85)} |
| table.tableblock #preamble>.sectionbody>.paragraph:first-of-type p{font-size:inherit} |
| .admonitionblock>table{border-collapse:separate;border:0;background:none;width:100%} |
| .admonitionblock>table td.icon{text-align:center;width:80px} |
| .admonitionblock>table td.icon img{max-width:none} |
| .admonitionblock>table td.icon .title{font-weight:bold;font-family:"Open Sans","DejaVu Sans",sans-serif;text-transform:uppercase} |
| .admonitionblock>table td.content{padding-left:1.125em;padding-right:1.25em;border-left:1px solid #ddddd8;color:rgba(0,0,0,.6)} |
| .admonitionblock>table td.content>:last-child>:last-child{margin-bottom:0} |
| .exampleblock>.content{border-style:solid;border-width:1px;border-color:#e6e6e6;margin-bottom:1.25em;padding:1.25em;background:#fff;-webkit-border-radius:4px;border-radius:4px} |
| .exampleblock>.content>:first-child{margin-top:0} |
| .exampleblock>.content>:last-child{margin-bottom:0} |
| .sidebarblock{border-style:solid;border-width:1px;border-color:#e0e0dc;margin-bottom:1.25em;padding:1.25em;background:#f8f8f7;-webkit-border-radius:4px;border-radius:4px} |
| .sidebarblock>:first-child{margin-top:0} |
| .sidebarblock>:last-child{margin-bottom:0} |
| .sidebarblock>.content>.title{color:#7a2518;margin-top:0;text-align:center} |
| .exampleblock>.content>:last-child>:last-child,.exampleblock>.content .olist>ol>li:last-child>:last-child,.exampleblock>.content .ulist>ul>li:last-child>:last-child,.exampleblock>.content .qlist>ol>li:last-child>:last-child,.sidebarblock>.content>:last-child>:last-child,.sidebarblock>.content .olist>ol>li:last-child>:last-child,.sidebarblock>.content .ulist>ul>li:last-child>:last-child,.sidebarblock>.content .qlist>ol>li:last-child>:last-child{margin-bottom:0} |
| .literalblock pre,.listingblock pre:not(.highlight),.listingblock pre[class="highlight"],.listingblock pre[class^="highlight "],.listingblock pre.CodeRay,.listingblock pre.prettyprint{background:#f7f7f8} |
| .sidebarblock .literalblock pre,.sidebarblock .listingblock pre:not(.highlight),.sidebarblock .listingblock pre[class="highlight"],.sidebarblock .listingblock pre[class^="highlight "],.sidebarblock .listingblock pre.CodeRay,.sidebarblock .listingblock pre.prettyprint{background:#f2f1f1} |
| .literalblock pre,.literalblock pre[class],.listingblock pre,.listingblock pre[class]{-webkit-border-radius:4px;border-radius:4px;word-wrap:break-word;padding:1em;font-size:.8125em} |
| .literalblock pre.nowrap,.literalblock pre[class].nowrap,.listingblock pre.nowrap,.listingblock pre[class].nowrap{overflow-x:auto;white-space:pre;word-wrap:normal} |
| @media only screen and (min-width:768px){.literalblock pre,.literalblock pre[class],.listingblock pre,.listingblock pre[class]{font-size:.90625em}}@media only screen and (min-width:1280px){.literalblock pre,.literalblock pre[class],.listingblock pre,.listingblock pre[class]{font-size:1em}}.literalblock.output pre{color:#f7f7f8;background-color:rgba(0,0,0,.9)} |
| .listingblock pre.highlightjs{padding:0} |
| .listingblock pre.highlightjs>code{padding:1em;-webkit-border-radius:4px;border-radius:4px} |
| .listingblock pre.prettyprint{border-width:0} |
| .listingblock>.content{position:relative} |
| .listingblock code[data-lang]:before{display:none;content:attr(data-lang);position:absolute;font-size:.75em;top:.425rem;right:.5rem;line-height:1;text-transform:uppercase;color:#999} |
| .listingblock:hover code[data-lang]:before{display:block} |
| .listingblock.terminal pre .command:before{content:attr(data-prompt);padding-right:.5em;color:#999} |
| .listingblock.terminal pre .command:not([data-prompt]):before{content:"$"} |
| table.pyhltable{border-collapse:separate;border:0;margin-bottom:0;background:none} |
| table.pyhltable td{vertical-align:top;padding-top:0;padding-bottom:0} |
| table.pyhltable td.code{padding-left:.75em;padding-right:0} |
| pre.pygments .lineno,table.pyhltable td:not(.code){color:#999;padding-left:0;padding-right:.5em;border-right:1px solid #ddddd8} |
| pre.pygments .lineno{display:inline-block;margin-right:.25em} |
| table.pyhltable .linenodiv{background:none!important;padding-right:0!important} |
| .quoteblock{margin:0 1em 1.25em 1.5em;display:table} |
| .quoteblock>.title{margin-left:-1.5em;margin-bottom:.75em} |
| .quoteblock blockquote,.quoteblock blockquote p{color:rgba(0,0,0,.85);font-size:1.15rem;line-height:1.75;word-spacing:.1em;letter-spacing:0;font-style:italic;text-align:justify} |
| .quoteblock blockquote{margin:0;padding:0;border:0} |
| .quoteblock blockquote:before{content:"\201c";float:left;font-size:2.75em;font-weight:bold;line-height:.6em;margin-left:-.6em;color:#7a2518;text-shadow:0 1px 2px rgba(0,0,0,.1)} |
| .quoteblock blockquote>.paragraph:last-child p{margin-bottom:0} |
| .quoteblock .attribution{margin-top:.5em;margin-right:.5ex;text-align:right} |
| .quoteblock .quoteblock{margin-left:0;margin-right:0;padding:.5em 0;border-left:3px solid rgba(0,0,0,.6)} |
| .quoteblock .quoteblock blockquote{padding:0 0 0 .75em} |
| .quoteblock .quoteblock blockquote:before{display:none} |
| .verseblock{margin:0 1em 1.25em 1em} |
| .verseblock pre{font-family:"Open Sans","DejaVu Sans",sans;font-size:1.15rem;color:rgba(0,0,0,.85);font-weight:300;text-rendering:optimizeLegibility} |
| .verseblock pre strong{font-weight:400} |
| .verseblock .attribution{margin-top:1.25rem;margin-left:.5ex} |
| .quoteblock .attribution,.verseblock .attribution{font-size:.9375em;line-height:1.45;font-style:italic} |
| .quoteblock .attribution br,.verseblock .attribution br{display:none} |
| .quoteblock .attribution cite,.verseblock .attribution cite{display:block;letter-spacing:-.05em;color:rgba(0,0,0,.6)} |
| .quoteblock.abstract{margin:0 0 1.25em 0;display:block} |
| .quoteblock.abstract blockquote,.quoteblock.abstract blockquote p{text-align:left;word-spacing:0} |
| .quoteblock.abstract blockquote:before,.quoteblock.abstract blockquote p:first-of-type:before{display:none} |
| table.tableblock{max-width:100%;border-collapse:separate} |
| table.tableblock td>.paragraph:last-child p>p:last-child,table.tableblock th>p:last-child,table.tableblock td>p:last-child{margin-bottom:0} |
| table.spread{width:100%} |
| table.tableblock,th.tableblock,td.tableblock{border:0 solid #dedede} |
| table.grid-all th.tableblock,table.grid-all td.tableblock{border-width:0 1px 1px 0} |
| table.grid-all tfoot>tr>th.tableblock,table.grid-all tfoot>tr>td.tableblock{border-width:1px 1px 0 0} |
| table.grid-cols th.tableblock,table.grid-cols td.tableblock{border-width:0 1px 0 0} |
| table.grid-all *>tr>.tableblock:last-child,table.grid-cols *>tr>.tableblock:last-child{border-right-width:0} |
| table.grid-rows th.tableblock,table.grid-rows td.tableblock{border-width:0 0 1px 0} |
| table.grid-all tbody>tr:last-child>th.tableblock,table.grid-all tbody>tr:last-child>td.tableblock,table.grid-all thead:last-child>tr>th.tableblock,table.grid-rows tbody>tr:last-child>th.tableblock,table.grid-rows tbody>tr:last-child>td.tableblock,table.grid-rows thead:last-child>tr>th.tableblock{border-bottom-width:0} |
| table.grid-rows tfoot>tr>th.tableblock,table.grid-rows tfoot>tr>td.tableblock{border-width:1px 0 0 0} |
| table.frame-all{border-width:1px} |
| table.frame-sides{border-width:0 1px} |
| table.frame-topbot{border-width:1px 0} |
| th.halign-left,td.halign-left{text-align:left} |
| th.halign-right,td.halign-right{text-align:right} |
| th.halign-center,td.halign-center{text-align:center} |
| th.valign-top,td.valign-top{vertical-align:top} |
| th.valign-bottom,td.valign-bottom{vertical-align:bottom} |
| th.valign-middle,td.valign-middle{vertical-align:middle} |
| table thead th,table tfoot th{font-weight:bold} |
| tbody tr th{display:table-cell;line-height:1.6;background:#f7f8f7} |
| tbody tr th,tbody tr th p,tfoot tr th,tfoot tr th p{color:rgba(0,0,0,.8);font-weight:bold} |
| p.tableblock>code:only-child{background:none;padding:0} |
| p.tableblock{font-size:1em} |
| td>div.verse{white-space:pre} |
| ol{margin-left:1.75em} |
| ul li ol{margin-left:1.5em} |
| dl dd{margin-left:1.125em} |
| dl dd:last-child,dl dd:last-child>:last-child{margin-bottom:0} |
| ol>li p,ul>li p,ul dd,ol dd,.olist .olist,.ulist .ulist,.ulist .olist,.olist .ulist{margin-bottom:.625em} |
| ul.unstyled,ol.unnumbered,ul.checklist,ul.none{list-style-type:none} |
| ul.unstyled,ol.unnumbered,ul.checklist{margin-left:.625em} |
| ul.checklist li>p:first-child>.fa-square-o:first-child,ul.checklist li>p:first-child>.fa-check-square-o:first-child{width:1em;font-size:.85em} |
| ul.checklist li>p:first-child>input[type="checkbox"]:first-child{width:1em;position:relative;top:1px} |
| ul.inline{margin:0 auto .625em auto;margin-left:-1.375em;margin-right:0;padding:0;list-style:none;overflow:hidden} |
| ul.inline>li{list-style:none;float:left;margin-left:1.375em;display:block} |
| ul.inline>li>*{display:block} |
| .unstyled dl dt{font-weight:400;font-style:normal} |
| ol.arabic{list-style-type:decimal} |
| ol.decimal{list-style-type:decimal-leading-zero} |
| ol.loweralpha{list-style-type:lower-alpha} |
| ol.upperalpha{list-style-type:upper-alpha} |
| ol.lowerroman{list-style-type:lower-roman} |
| ol.upperroman{list-style-type:upper-roman} |
| ol.lowergreek{list-style-type:lower-greek} |
| .hdlist>table,.colist>table{border:0;background:none} |
| .hdlist>table>tbody>tr,.colist>table>tbody>tr{background:none} |
| td.hdlist1{padding-right:.75em;font-weight:bold} |
| td.hdlist1,td.hdlist2{vertical-align:top} |
| .literalblock+.colist,.listingblock+.colist{margin-top:-.5em} |
| .colist>table tr>td:first-of-type{padding:0 .75em;line-height:1} |
| .colist>table tr>td:last-of-type{padding:.25em 0} |
| .thumb,.th{line-height:0;display:inline-block;border:solid 4px #fff;-webkit-box-shadow:0 0 0 1px #ddd;box-shadow:0 0 0 1px #ddd} |
| .imageblock.left,.imageblock[style*="float: left"]{margin:.25em .625em 1.25em 0} |
| .imageblock.right,.imageblock[style*="float: right"]{margin:.25em 0 1.25em .625em} |
| .imageblock>.title{margin-bottom:0} |
| .imageblock.thumb,.imageblock.th{border-width:6px} |
| .imageblock.thumb>.title,.imageblock.th>.title{padding:0 .125em} |
| .image.left,.image.right{margin-top:.25em;margin-bottom:.25em;display:inline-block;line-height:0} |
| .image.left{margin-right:.625em} |
| .image.right{margin-left:.625em} |
| a.image{text-decoration:none} |
| span.footnote,span.footnoteref{vertical-align:super;font-size:.875em} |
| span.footnote a,span.footnoteref a{text-decoration:none} |
| span.footnote a:active,span.footnoteref a:active{text-decoration:underline} |
| #footnotes{padding-top:.75em;padding-bottom:.75em;margin-bottom:.625em} |
| #footnotes hr{width:20%;min-width:6.25em;margin:-.25em 0 .75em 0;border-width:1px 0 0 0} |
| #footnotes .footnote{padding:0 .375em;line-height:1.3;font-size:.875em;margin-left:1.2em;text-indent:-1.2em;margin-bottom:.2em} |
| #footnotes .footnote a:first-of-type{font-weight:bold;text-decoration:none} |
| #footnotes .footnote:last-of-type{margin-bottom:0} |
| #content #footnotes{margin-top:-.625em;margin-bottom:0;padding:.75em 0} |
| .gist .file-data>table{border:0;background:#fff;width:100%;margin-bottom:0} |
| .gist .file-data>table td.line-data{width:99%} |
| div.unbreakable{page-break-inside:avoid} |
| .big{font-size:larger} |
| .small{font-size:smaller} |
| .underline{text-decoration:underline} |
| .overline{text-decoration:overline} |
| .line-through{text-decoration:line-through} |
| .aqua{color:#00bfbf} |
| .aqua-background{background-color:#00fafa} |
| .black{color:#000} |
| .black-background{background-color:#000} |
| .blue{color:#0000bf} |
| .blue-background{background-color:#0000fa} |
| .fuchsia{color:#bf00bf} |
| .fuchsia-background{background-color:#fa00fa} |
| .gray{color:#606060} |
| .gray-background{background-color:#7d7d7d} |
| .green{color:#006000} |
| .green-background{background-color:#007d00} |
| .lime{color:#00bf00} |
| .lime-background{background-color:#00fa00} |
| .maroon{color:#600000} |
| .maroon-background{background-color:#7d0000} |
| .navy{color:#000060} |
| .navy-background{background-color:#00007d} |
| .olive{color:#606000} |
| .olive-background{background-color:#7d7d00} |
| .purple{color:#600060} |
| .purple-background{background-color:#7d007d} |
| .red{color:#bf0000} |
| .red-background{background-color:#fa0000} |
| .silver{color:#909090} |
| .silver-background{background-color:#bcbcbc} |
| .teal{color:#006060} |
| .teal-background{background-color:#007d7d} |
| .white{color:#bfbfbf} |
| .white-background{background-color:#fafafa} |
| .yellow{color:#bfbf00} |
| .yellow-background{background-color:#fafa00} |
| span.icon>.fa{cursor:default} |
| .admonitionblock td.icon [class^="fa icon-"]{font-size:2.5em;text-shadow:1px 1px 2px rgba(0,0,0,.5);cursor:default} |
| .admonitionblock td.icon .icon-note:before{content:"\f05a";color:#19407c} |
| .admonitionblock td.icon .icon-tip:before{content:"\f0eb";text-shadow:1px 1px 2px rgba(155,155,0,.8);color:#111} |
| .admonitionblock td.icon .icon-warning:before{content:"\f071";color:#bf6900} |
| .admonitionblock td.icon .icon-caution:before{content:"\f06d";color:#bf3400} |
| .admonitionblock td.icon .icon-important:before{content:"\f06a";color:#bf0000} |
| .conum[data-value]{display:inline-block;color:#fff!important;background-color:rgba(0,0,0,.8);-webkit-border-radius:100px;border-radius:100px;text-align:center;font-size:.75em;width:1.67em;height:1.67em;line-height:1.67em;font-family:"Open Sans","DejaVu Sans",sans-serif;font-style:normal;font-weight:bold} |
| .conum[data-value] *{color:#fff!important} |
| .conum[data-value]+b{display:none} |
| .conum[data-value]:after{content:attr(data-value)} |
| pre .conum[data-value]{position:relative;top:-.125em} |
| b.conum *{color:inherit!important} |
| .conum:not([data-value]):empty{display:none} |
| h1,h2{letter-spacing:-.01em} |
| dt,th.tableblock,td.content{text-rendering:optimizeLegibility} |
| p,td.content{letter-spacing:-.01em} |
| p strong,td.content strong{letter-spacing:-.005em} |
| p,blockquote,dt,td.content{font-size:1.0625rem} |
| p{margin-bottom:1.25rem} |
| .sidebarblock p,.sidebarblock dt,.sidebarblock td.content,p.tableblock{font-size:1em} |
| .exampleblock>.content{background-color:#fffef7;border-color:#e0e0dc;-webkit-box-shadow:0 1px 4px #e0e0dc;box-shadow:0 1px 4px #e0e0dc} |
| .print-only{display:none!important} |
| @media print{@page{margin:1.25cm .75cm} |
| *{-webkit-box-shadow:none!important;box-shadow:none!important;text-shadow:none!important} |
| a{color:inherit!important;text-decoration:underline!important} |
| a.bare,a[href^="#"],a[href^="mailto:"]{text-decoration:none!important} |
| a[href^="http:"]:not(.bare):after,a[href^="https:"]:not(.bare):after{content:"(" attr(href) ")";display:inline-block;font-size:.875em;padding-left:.25em} |
| abbr[title]:after{content:" (" attr(title) ")"} |
| pre,blockquote,tr,img{page-break-inside:avoid} |
| thead{display:table-header-group} |
| img{max-width:100%!important} |
| p,blockquote,dt,td.content{font-size:1em;orphans:3;widows:3} |
| h2,h3,#toctitle,.sidebarblock>.content>.title{page-break-after:avoid} |
| #toc,.sidebarblock,.exampleblock>.content{background:none!important} |
| #toc{border-bottom:1px solid #ddddd8!important;padding-bottom:0!important} |
| .sect1{padding-bottom:0!important} |
| .sect1+.sect1{border:0!important} |
| #header>h1:first-child{margin-top:1.25rem} |
| body.book #header{text-align:center} |
| body.book #header>h1:first-child{border:0!important;margin:2.5em 0 1em 0} |
| body.book #header .details{border:0!important;display:block;padding:0!important} |
| body.book #header .details span:first-child{margin-left:0!important} |
| body.book #header .details br{display:block} |
| body.book #header .details br+span:before{content:none!important} |
| body.book #toc{border:0!important;text-align:left!important;padding:0!important;margin:0!important} |
| body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-break-before:always} |
| .listingblock code[data-lang]:before{display:block} |
| #footer{background:none!important;padding:0 .9375em} |
| #footer-text{color:rgba(0,0,0,.6)!important;font-size:.9em} |
| .hide-on-print{display:none!important} |
| .print-only{display:block!important} |
| .hide-for-print{display:none!important} |
| .show-for-print{display:inherit!important}} |
| </style> |
| <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css"> |
| </head> |
| <body class="article"> |
| <div id="header"> |
| <h1>Apache NiFi User Guide</h1> |
| <div class="details"> |
| <span id="author" class="author">Apache NiFi Team</span><br> |
| <span id="email" class="email"><a href="mailto:dev@nifi.apache.org">dev@nifi.apache.org</a></span><br> |
| </div> |
| <div id="toc" class="toc"> |
| <div id="toctitle">Table of Contents</div> |
| <ul class="sectlevel1"> |
| <li><a href="user-guide.html#introduction">Introduction</a></li> |
| <li><a href="user-guide.html#browser-support">Browser Support</a> |
| <ul class="sectlevel2"> |
| <li><a href="user-guide.html#unsupported-browsers">Unsupported Browsers</a></li> |
| <li><a href="user-guide.html#viewing-the-ui-in-variably-sized-browsers">Viewing the UI in Variably Sized Browsers</a></li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#terminology">Terminology</a></li> |
| <li><a href="user-guide.html#User_Interface">NiFi User Interface</a></li> |
| <li><a href="user-guide.html#UI-with-multi-tenant-authorization">Accessing the UI with Multi-Tenant Authorization</a></li> |
| <li><a href="user-guide.html#logging-in">Logging In</a></li> |
| <li><a href="user-guide.html#building-dataflow">Building a DataFlow</a> |
| <ul class="sectlevel2"> |
| <li><a href="user-guide.html#adding-components-to-the-canvas">Adding Components to the Canvas</a></li> |
| <li><a href="user-guide.html#component-versioning">Component Versions</a> |
| <ul class="sectlevel3"> |
| <li><a href="user-guide.html#sorting-and-filtering-components">Sorting and Filtering Components</a></li> |
| <li><a href="user-guide.html#changing-component-versions">Changing Component Versions</a></li> |
| <li><a href="user-guide.html#understanding-version-dependencies">Understanding Version Dependencies</a></li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#Configuring_a_Processor">Configuring a Processor</a> |
| <ul class="sectlevel3"> |
| <li><a href="user-guide.html#settings-tab">Settings Tab</a></li> |
| <li><a href="user-guide.html#scheduling-tab">Scheduling Tab</a></li> |
| <li><a href="user-guide.html#properties-tab">Properties Tab</a></li> |
| <li><a href="user-guide.html#relationships-tab">Relationships Tab</a></li> |
| <li><a href="user-guide.html#comments-tab">Comments Tab</a></li> |
| <li><a href="user-guide.html#additional-help">Additional Help</a></li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#Configuring_a_ProcessGroup">Configuring a Process Group</a> |
| <ul class="sectlevel3"> |
| <li><a href="user-guide.html#General_tab_ProcessGroup">General Tab</a></li> |
| <li><a href="user-guide.html#controller-services">Controller Services</a></li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#Parameters">Parameters</a> |
| <ul class="sectlevel3"> |
| <li><a href="user-guide.html#parameter-contexts">Parameter Contexts</a></li> |
| <li><a href="user-guide.html#adding-a-parameter-to-a-parameter-context">Adding a Parameter to a Parameter Context</a></li> |
| <li><a href="user-guide.html#parameters-and-el">Parameters and Expression Language</a></li> |
| <li><a href="user-guide.html#assigning_parameter_context_to_PG">Assigning a Parameter Context to a Process Group</a></li> |
| <li><a href="user-guide.html#referencing-parameters">Referencing Parameters</a></li> |
| <li><a href="user-guide.html#parameter-providers">Parameter Providers</a></li> |
| <li><a href="user-guide.html#accessing-parameters">Accessing Parameters</a></li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#Using_Custom_Properties">Using Custom Properties with Expression Language</a> |
| <ul class="sectlevel3"> |
| <li><a href="user-guide.html#Variables">Variables</a></li> |
| <li><a href="user-guide.html#Custom_Properties">Referencing Custom Properties via nifi.properties</a></li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#Controller_Services">Controller Services</a> |
| <ul class="sectlevel3"> |
| <li><a href="user-guide.html#Management_Controller_Services">Adding Management Controller Services</a></li> |
| <li><a href="user-guide.html#Controller_Services_for_Dataflows">Adding Controller Services for Dataflows</a></li> |
| <li><a href="user-guide.html#Enabling_Disabling_Controller_Services">Enabling/Disabling Controller Services</a></li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#Reporting_Tasks">Reporting Tasks</a></li> |
| <li><a href="user-guide.html#Connecting_Components">Connecting Components</a> |
| <ul class="sectlevel3"> |
| <li><a href="user-guide.html#details-tab">Details Tab</a></li> |
| <li><a href="user-guide.html#settings">Settings</a></li> |
| <li><a href="user-guide.html#changing-configuration-and-context-menu-options">Changing Configuration and Context Menu Options</a></li> |
| <li><a href="user-guide.html#bending-connections">Bending Connections</a></li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#processor-validation">Processor Validation</a></li> |
| <li><a href="user-guide.html#site-to-site">Site-to-Site</a> |
| <ul class="sectlevel3"> |
| <li><a href="user-guide.html#configure-site-to-site-client-nifi-instance">Configure Site-to-Site client NiFi instance</a></li> |
| <li><a href="user-guide.html#configure-site-to-site-server-nifi-instance">Configure Site-to-Site Server NiFi Instance</a></li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#example-dataflow">Example Dataflow</a></li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#command-and-control-of-the-dataflow">Command and Control of the DataFlow</a> |
| <ul class="sectlevel2"> |
| <li><a href="user-guide.html#starting-a-component">Starting a Component</a></li> |
| <li><a href="user-guide.html#stopping_components">Stopping a Component</a></li> |
| <li><a href="user-guide.html#terminating_tasks">Terminating a Component’s Tasks</a></li> |
| <li><a href="user-guide.html#enablingdisabling-a-component">Enabling/Disabling a Component</a></li> |
| <li><a href="user-guide.html#Remote_Group_Transmission">Remote Process Group Transmission</a> |
| <ul class="sectlevel3"> |
| <li><a href="user-guide.html#Remote_Port_Configuration">Individual Port Transmission</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#navigating">Navigating within a DataFlow</a> |
| <ul class="sectlevel2"> |
| <li><a href="user-guide.html#component_linking">Component Linking</a></li> |
| <li><a href="user-guide.html#component_alignment">Component Alignment</a> |
| <ul class="sectlevel3"> |
| <li><a href="user-guide.html#align-vertically">Align Vertically</a></li> |
| <li><a href="user-guide.html#align-horizontally">Align Horizontally</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#search">Search Components in DataFlow</a> |
| <ul class="sectlevel2"> |
| <li><a href="user-guide.html#filters">Filters</a></li> |
| <li><a href="user-guide.html#keywords">Keywords</a></li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#monitoring">Monitoring of DataFlow</a> |
| <ul class="sectlevel2"> |
| <li><a href="user-guide.html#processor_anatomy">Anatomy of a Processor</a></li> |
| <li><a href="user-guide.html#process_group_anatomy">Anatomy of a Process Group</a></li> |
| <li><a href="user-guide.html#remote_group_anatomy">Anatomy of a Remote Process Group</a></li> |
| <li><a href="user-guide.html#Queue_Interaction">Queue Interaction</a></li> |
| <li><a href="user-guide.html#Summary_Page">Summary Page</a></li> |
| <li><a href="user-guide.html#Status_History">Historical Statistics of a Component</a></li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#versioning_dataflow">Versioning a DataFlow</a> |
| <ul class="sectlevel2"> |
| <li><a href="user-guide.html#connecting-to-a-nifi-registry">Connecting to a NiFi Registry</a></li> |
| <li><a href="user-guide.html#version_states">Version States</a></li> |
| <li><a href="user-guide.html#import-a-versioned-flow">Import a Versioned Flow</a></li> |
| <li><a href="user-guide.html#start-version-control">Start Version Control</a></li> |
| <li><a href="user-guide.html#managing_local_changes">Managing Local Changes</a> |
| <ul class="sectlevel3"> |
| <li><a href="user-guide.html#show-local-changes">Show Local Changes</a></li> |
| <li><a href="user-guide.html#revert-local-changes">Revert Local Changes</a></li> |
| <li><a href="user-guide.html#commit-local-changes">Commit Local Changes</a></li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#change-version">Change Version</a></li> |
| <li><a href="user-guide.html#stop-version-control">Stop Version Control</a></li> |
| <li><a href="user-guide.html#nested-versioned-flows">Nested Versioned Flows</a></li> |
| <li><a href="user-guide.html#parameters-in-versioned-flows">Parameters in Versioned Flows</a></li> |
| <li><a href="user-guide.html#Variables_in_Versioned_Flows">Variables in Versioned Flows</a></li> |
| <li><a href="user-guide.html#Restricted_Components_in_Versioned_Flows">Restricted Components in Versioned Flows</a> |
| <ul class="sectlevel3"> |
| <li><a href="user-guide.html#restricted-controller-service-created-in-root-process-group">Restricted Controller Service Created in Root Process Group</a></li> |
| <li><a href="user-guide.html#restricted-controller-service-created-in-process-group">Restricted Controller Service Created in Process Group</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#templates">Templates</a> |
| <ul class="sectlevel2"> |
| <li><a href="user-guide.html#Create_Template">Creating a Template</a></li> |
| <li><a href="user-guide.html#Import_Template">Importing a Template</a></li> |
| <li><a href="user-guide.html#instantiating-a-template">Instantiating a Template</a></li> |
| <li><a href="user-guide.html#Manage_Templates">Managing Templates</a> |
| <ul class="sectlevel3"> |
| <li><a href="user-guide.html#Export_Template">Exporting a Template</a></li> |
| <li><a href="user-guide.html#removing-a-template">Removing a Template</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#data_provenance">Data Provenance</a> |
| <ul class="sectlevel2"> |
| <li><a href="user-guide.html#provenance_events">Provenance Events</a></li> |
| <li><a href="user-guide.html#searching-for-events">Searching for Events</a></li> |
| <li><a href="user-guide.html#event_details">Details of an Event</a></li> |
| <li><a href="user-guide.html#replay_flowfile">Replaying a FlowFile</a></li> |
| <li><a href="user-guide.html#viewing-flowfile-lineage">Viewing FlowFile Lineage</a> |
| <ul class="sectlevel3"> |
| <li><a href="user-guide.html#find-parents">Find Parents</a></li> |
| <li><a href="user-guide.html#expanding-an-event">Expanding an Event</a></li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#writeahead-provenance">Write Ahead Provenance Repository</a> |
| <ul class="sectlevel3"> |
| <li><a href="user-guide.html#backwards-compatibility">Backwards Compatibility</a></li> |
| <li><a href="user-guide.html#older-existing-nifi-version">Older Existing NiFi Version</a></li> |
| <li><a href="user-guide.html#bootstrap-conf">Bootstrap.conf</a></li> |
| <li><a href="user-guide.html#system-properties">System Properties</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li><a href="user-guide.html#experimental_warning">Experimental Warning</a></li> |
| <li><a href="user-guide.html#other_management_features">Other Management Features</a></li> |
| </ul> |
| </div> |
| </div> |
| <div id="content"> |
| <div class="sect1"> |
| <h2 id="introduction"><a class="anchor" href="user-guide.html#introduction"></a>Introduction</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Apache NiFi is a dataflow system based on the concepts of flow-based programming. It supports |
| powerful and scalable directed graphs of data routing, transformation, and system mediation logic. NiFi has |
| a web-based user interface for design, control, feedback, and monitoring of dataflows. It is highly configurable |
| along several dimensions of quality of service, such as loss-tolerant versus guaranteed delivery, low latency versus |
| high throughput, and priority-based queuing. NiFi provides fine-grained data provenance for all data received, forked, joined |
| cloned, modified, sent, and ultimately dropped upon reaching its configured end-state.</p> |
| </div> |
| <div class="paragraph"> |
| <p>See the <a href="administration-guide.html">System Administrator’s Guide</a> for information about system requirements, installation, and configuration. Once NiFi is installed, |
| use a supported web browser to view the UI.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="browser-support"><a class="anchor" href="user-guide.html#browser-support"></a>Browser Support</h2> |
| <div class="sectionbody"> |
| <table class="tableblock frame-all grid-all stretch"> |
| <colgroup> |
| <col style="width: 50%;"> |
| <col style="width: 50%;"> |
| </colgroup> |
| <thead> |
| <tr> |
| <th class="tableblock halign-left valign-top">Browser</th> |
| <th class="tableblock halign-left valign-top">Version</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Chrome</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Current and Current - 1</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">FireFox</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Current and Current - 1</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Edge</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Current and Current - 1</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Safari</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Current and Current - 1</p></td> |
| </tr> |
| </tbody> |
| </table> |
| <div class="paragraph"> |
| <p>Current and Current - 1 indicates that the UI is supported in the current stable release of that browser and the preceding one. For instance, if |
| the current stable release is 45.X then the officially supported versions will be 45.X and 44.X.</p> |
| </div> |
| <div class="paragraph"> |
| <p>For Safari, which releases major versions much less frequently, Current and Current - 1 simply represent the two latest releases.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The supported browser versions are driven by the capabilities the UI employs and the dependencies it uses. UI features will be developed and tested |
| against the supported browsers. Any problem using a supported browser should be reported to Apache NiFi.</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="unsupported-browsers"><a class="anchor" href="user-guide.html#unsupported-browsers"></a>Unsupported Browsers</h3> |
| <div class="paragraph"> |
| <p>While the UI may run successfully in unsupported browsers, it is not actively tested against them. Additionally, the UI is designed as a desktop |
| experience and is not currently supported in mobile browsers.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="viewing-the-ui-in-variably-sized-browsers"><a class="anchor" href="user-guide.html#viewing-the-ui-in-variably-sized-browsers"></a>Viewing the UI in Variably Sized Browsers</h3> |
| <div class="paragraph"> |
| <p>In most environments, all of the UI is visible in your browser. However, the UI has a responsive design that allows you |
| to scroll through screens as needed, in smaller sized browsers or tablet environments.</p> |
| </div> |
| <div class="paragraph"> |
| <p>In environments where your browser width is less than 800 pixels and the height less than 600 pixels, portions of the |
| UI may become unavailable.</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="terminology"><a class="anchor" href="user-guide.html#terminology"></a>Terminology</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p><strong>DataFlow Manager</strong>: A DataFlow Manager (DFM) is a NiFi user who has permissions to add, remove, and modify components of a NiFi dataflow.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>FlowFile</strong>: The FlowFile represents a single piece of data in NiFi. A FlowFile is made up of two components: |
| FlowFile Attributes and FlowFile Content. |
| Content is the data that is represented by the FlowFile. Attributes are characteristics that provide information or |
| context about the data; they are made up of key-value pairs. |
| All FlowFiles have the following Standard Attributes:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>uuid</strong>: A Universally Unique Identifier that distinguishes the FlowFile from other FlowFiles in the system.</p> |
| </li> |
| <li> |
| <p><strong>filename</strong>: A human-readable filename that may be used when storing the data to disk or in an external service</p> |
| </li> |
| <li> |
| <p><strong>path</strong>: A hierarchically structured value that can be used when storing data to disk or an external service so that the data is not stored in a single directory</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Processor</strong>: The Processor is the NiFi component that is used to listen for incoming data; pull data from external sources; |
| publish data to external sources; and route, transform, or extract information from FlowFiles.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Relationship</strong>: Each Processor has zero or more Relationships defined for it. These Relationships are named to indicate the result of processing a FlowFile. |
| After a Processor has finished processing a FlowFile, it will route (or “transfer”) the FlowFile to one of the Relationships. |
| A DFM is then able to connect each of these Relationships to other components in order to specify where the FlowFile should |
| go next under each potential processing result.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Connection</strong>: A DFM creates an automated dataflow by dragging components from the Components part of the NiFi toolbar to the canvas |
| and then connecting the components together via Connections. Each connection consists of one or more Relationships. |
| For each Connection that is drawn, a DFM can determine which Relationships should be used for the Connection. |
| This allows data to be routed in different ways based on its processing outcome. Each connection houses a FlowFile Queue. |
| When a FlowFile is transferred to a particular Relationship, it is added to the queue belonging to the associated Connection.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Controller Service</strong>: Controller Services are extension points that, after being added and configured by a DFM in the User Interface, will start up when NiFi starts up and provide information for use by other components (such as processors or other controller services). A common Controller Service used by several components is the StandardSSLContextService. It provides the ability to configure keystore and/or truststore properties once and reuse that configuration throughout the application. The idea is that, rather than configure this information in every processor that might need it, the controller service provides it for any processor to use as needed.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Reporting Task</strong>: Reporting Tasks run in the background to provide statistical reports about what is happening in the NiFi instance. The DFM adds and configures Reporting Tasks in the User Interface as desired. Common reporting tasks include the ControllerStatusReportingTask, MonitorDiskUsage reporting task, MonitorMemory reporting task, and the StandardGangliaReporter.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Parameter Provider</strong>: Parameter Providers can provide parameters from an external source to Parameter Contexts. The parameters of a Parameter Provider may be fetched and applied to all referencing Parameter Contexts.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Funnel</strong>: A funnel is a NiFi component that is used to combine the data from several Connections into a single Connection.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Process Group</strong>: When a dataflow becomes complex, it often is beneficial to reason about the dataflow at a higher, more abstract level. |
| NiFi allows multiple components, such as Processors, to be grouped together into a Process Group. |
| The NiFi User Interface then makes it easy for a DFM to connect together multiple Process Groups into a logical dataflow, |
| as well as allowing the DFM to enter a Process Group in order to see and manipulate the components within the Process Group.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Port</strong>: Dataflows that are constructed using one or more Process Groups need a way to connect a Process Group to other dataflow components. |
| This is achieved by using Ports. A DFM can add any number of Input Ports and Output Ports to a Process Group and name these ports appropriately.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Remote Process Group</strong>: Just as data is transferred into and out of a Process Group, it is sometimes necessary to transfer data from one instance of NiFi to another. |
| While NiFi provides many different mechanisms for transferring data from one system to another, Remote Process Groups are often the easiest way to accomplish |
| this if transferring data to another instance of NiFi.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Bulletin</strong>: The NiFi User Interface provides a significant amount of monitoring and feedback about the current status of the application. |
| In addition to rolling statistics and the current status provided for each component, components are able to report Bulletins. |
| Whenever a component reports a Bulletin, a bulletin icon is displayed on that component. System-level bulletins are displayed on the Status bar near the top of the page. |
| Using the mouse to hover over that icon will provide a tool-tip that shows the time and severity (Debug, Info, Warning, Error) of the Bulletin, |
| as well as the message of the Bulletin. |
| Bulletins from all components can also be viewed and filtered in the Bulletin Board Page, available in the Global Menu.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Template</strong>: Often times, a dataflow is comprised of many sub-flows that could be reused. NiFi allows DFMs to select a part of the dataflow |
| (or the entire dataflow) and create a Template. This Template is given a name and can then be dragged onto the canvas just like the other components. |
| As a result, several components may be combined together to make a larger building block from which to create a dataflow. |
| These templates can also be exported as XML and imported into another NiFi instance, allowing these building blocks to be shared.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>flow.xml.gz</strong>: Everything the DFM puts onto the NiFi User Interface canvas is written, in real time, to one file called the <em>flow.xml.gz</em>. This file is located in the <code>nifi/conf</code> directory by default. |
| Any change made on the canvas is automatically saved to this file, without the user needing to click a "Save" button. |
| In addition, NiFi automatically creates a backup copy of this file in the archive directory when it is updated. |
| You can use these archived files to rollback flow configuration. To do so, stop NiFi, replace <em>flow.xml.gz</em> with a desired backup copy, then restart NiFi. |
| In a clustered environment, stop the entire NiFi cluster, replace the <em>flow.xml.gz</em> of one of nodes, and restart the node. Remove <em>flow.xml.gz</em> from other nodes. |
| Once you confirmed the node starts up as a one-node cluster, start the other nodes. The replaced flow configuration will be synchronized across the cluster. |
| The name and location of <em>flow.xml.gz</em>, and auto archive behavior are configurable. See the <a href="administration-guide.html#core-properties-br">System Administrator’s Guide</a> for further details.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="User_Interface"><a class="anchor" href="user-guide.html#User_Interface"></a>NiFi User Interface</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>The NiFi UI provides mechanisms for creating automated dataflows, as well as visualizing, |
| editing, monitoring, and administering those dataflows. The UI can be broken down into several segments, |
| each responsible for different functionality of the application. This section provides screenshots of the |
| application and highlights the different segments of the UI. Each segment is discussed in further detail later |
| in the document.</p> |
| </div> |
| <div class="paragraph"> |
| <p>When the application is started, the user is able to navigate to the UI by going to the default address of |
| <code><a href="https://localhost:8443/nifi" class="bare">https://localhost:8443/nifi</a></code> in a web browser. The default configuration generates a username and password |
| with full system administration privileges. |
| For information on securing the system, see the <a href="administration-guide.html">System Administrator’s Guide</a>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>When a DFM navigates to the UI for the first time, a blank canvas is provided on which a dataflow can be built:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/nifi-toolbar-components.png" alt="NiFi Components Toolbar"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The Components Toolbar runs across the top left portion of your screen. It consists of the components you can drag onto the |
| canvas to build your dataflow. Each component is described in more detail in <a href="user-guide.html#building-dataflow">Building a DataFlow</a>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The Status Bar is under the Components Toolbar. The Status bar provides information about the number of threads that are |
| currently active in the flow, the amount of data that currently exists in the flow, how many Remote Process Groups exist |
| on the canvas in each state (Transmitting, Not Transmitting), how many Processors exist on the canvas in each state |
| (Stopped, Running, Invalid, Disabled), how many versioned Process Groups exist on the canvas in each state (Up to date, |
| Locally modified, Stale, Locally modified and stale, Sync failure) and the timestamp at which all of this information |
| was last refreshed. Additionally, if the instance of NiFi is clustered, the Status bar shows how many nodes are in the |
| cluster and how many are currently connected.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The Operate Palette sits to the left-hand side of the screen. It consists of buttons that are |
| used by DFMs to manage the flow, as well as by administrators who manage user access |
| and configure system properties, such as how many system resources should be provided to the application.</p> |
| </div> |
| <div class="paragraph"> |
| <p>On the right side of the canvas is Search, and the Global Menu. For more information on search refer to <a href="user-guide.html#search">Search Components in DataFlow</a>. The Global Menu |
| contains options that allow you to manipulate existing components on the canvas:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/global-menu.png" alt="NiFi Global Menu"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Additionally, the UI has some features that allow you to easily navigate around the canvas. You can use the |
| Navigate Palette to pan around the canvas, and to zoom in and out. The “Birds Eye View” of the dataflow provides a high-level |
| view of the dataflow and allows you to pan across large portions of the dataflow. You can also find breadcrumbs along the |
| bottom of the screen. As you navigate into and out of Process Groups, the breadcrumbs show |
| the depth in the flow, and each Process Group that you entered to reach this depth. Each of the Process Groups listed in the |
| breadcrumbs is a link that will take you back up to that level in the flow.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/nifi-navigation.png" alt="NiFi Navigation"> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="UI-with-multi-tenant-authorization"><a class="anchor" href="user-guide.html#UI-with-multi-tenant-authorization"></a>Accessing the UI with Multi-Tenant Authorization</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Multi-tenant authorization enables multiple groups of users (tenants) to command, control, and observe different parts of the dataflow, |
| with varying levels of authorization. When an authenticated user attempts to view or modify a NiFi resource, the system checks whether the |
| user has privileges to perform that action. These privileges are defined by policies that you can apply system wide or to individual |
| components. What this means from a Dataflow Manager perspective is that once you have access to the NiFi canvas, a range of functionality |
| is visible and available to you, depending on the privileges assigned to you.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The available global access policies are:</p> |
| </div> |
| <table class="tableblock frame-all grid-all stretch"> |
| <colgroup> |
| <col style="width: 50%;"> |
| <col style="width: 50%;"> |
| </colgroup> |
| <thead> |
| <tr> |
| <th class="tableblock halign-left valign-top">Policy</th> |
| <th class="tableblock halign-left valign-top">Privilege</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">view the UI</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view the UI</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">access the controller</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view and modify the controller including Management Controller Services, Reporting Tasks, Registry Clients, Parameter Providers and nodes in the cluster</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">query provenance</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to submit a provenance search and request even lineage</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">access restricted components</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to create/modify restricted components assuming other permissions are sufficient. The restricted |
| components may indicate which specific permissions are required. Permissions can be granted for specific restrictions or be granted regardless |
| of restrictions. If permission is granted regardless of restrictions, the user can create/modify all restricted components.</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">access all policies</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view and modify the policies for all components</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">access users/groups</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view and modify the users and user groups</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">retrieve site-to-site details</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows other NiFi instances to retrieve Site-To-Site details</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">view system diagnostics</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view System Diagnostics</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">proxy user requests</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows proxy machines to send requests on the behalf of others</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">access counters</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view and modify counters</p></td> |
| </tr> |
| </tbody> |
| </table> |
| <div class="paragraph"> |
| <p>The available component-level access policies are:</p> |
| </div> |
| <table class="tableblock frame-all grid-all stretch"> |
| <colgroup> |
| <col style="width: 50%;"> |
| <col style="width: 50%;"> |
| </colgroup> |
| <thead> |
| <tr> |
| <th class="tableblock halign-left valign-top">Policy</th> |
| <th class="tableblock halign-left valign-top">Privilege</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">view the component</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view component configuration details</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">modify the component</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to modify component configuration details</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">view provenance</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view provenance events generated by this component</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">view the data</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view metadata and content for this component in FlowFile queues in outbound connections and through provenance events</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">modify the data</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to empty FlowFile queues in outbound connections and submit replays through provenance events</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">view the policies</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view the list of users who can view and modify a component</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">modify the policies</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to modify the list of users who can view and modify a component</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">retrieve data via site-to-site</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows a port to receive data from NiFi instances</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">send data via site-to-site</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Allows a port to send data from NiFi instances</p></td> |
| </tr> |
| </tbody> |
| </table> |
| <div class="paragraph"> |
| <p>If you are unable to view or modify a NiFi resource, contact your System Administrator or see Configuring Users and Access Policies in the |
| <a href="administration-guide.html">System Administrator’s Guide</a> for more information.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="logging-in"><a class="anchor" href="user-guide.html#logging-in"></a>Logging In</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>If NiFi is configured to run securely, users will be able to request access to the DataFlow. For information on configuring NiFi to run |
| securely, see the <a href="administration-guide.html">System Administrator’s Guide</a>. If NiFi supports anonymous access, users will be given access |
| accordingly and given an option to log in.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Clicking the 'login' link will open the log in page. If the user is logging in with their username/password they will be presented with |
| a form to do so. If NiFi is not configured to support anonymous access and the user is logging in with their username/password, they will |
| be immediately sent to the login form bypassing the canvas.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/login.png" alt="Log In"> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="building-dataflow"><a class="anchor" href="user-guide.html#building-dataflow"></a>Building a DataFlow</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>A DFM is able to build an automated dataflow using the NiFi UI. Simply drag components from the toolbar to the canvas, |
| configure the components to meet specific needs, and connect |
| the components together.</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="adding-components-to-the-canvas"><a class="anchor" href="user-guide.html#adding-components-to-the-canvas"></a>Adding Components to the Canvas</h3> |
| <div class="paragraph"> |
| <p>The User Interface section above outlined the different segments of the UI and pointed out a Components Toolbar. |
| This section looks at each of the Components in that toolbar:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/components.png" alt="Components"> |
| </div> |
| </div> |
| <div id="processor" class="paragraph"> |
| <p><span class="image"><img src="images/iconProcessor.png" alt="Processor" width="32"></span> |
| <strong>Processor</strong>: The Processor is the most commonly used component, as it is responsible for data ingress, egress, routing, and |
| manipulating. There are many different types of Processors. In fact, this is a very common Extension Point in NiFi, |
| meaning that many vendors may implement their own Processors to perform whatever functions are necessary for their use case. |
| When a Processor is dragged onto the canvas, the user is presented with a dialog to choose which type of Processor to use:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/add-processor.png" alt="Add Processor Dialog"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>In the top-right corner, the user is able to filter the list based on the Processor Type or the Tags associated with a Processor. |
| Processor developers have the ability to add Tags to their Processors. These tags are used in this dialog for filtering and are |
| displayed on the left-hand side in a Tag Cloud. The more Processors that exist with a particular Tag, the larger the Tag appears |
| in the Tag Cloud. Clicking a Tag in the Cloud will filter the available Processors to only those that contain that Tag. If multiple |
| Tags are selected, only those Processors that contain all of those Tags are shown. For example, if we want to show only those |
| Processors that allow us to ingest files, we can select both the <code>files</code> Tag and the <code>ingest</code> Tag:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/add-processor-with-tag-cloud.png" alt="Add Processor with Tag Cloud"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Restricted components will be marked with a |
| <span class="image"><img src="images/restricted.png" alt="Restricted"></span> |
| icon next to their name. These are components |
| that can be used to execute arbitrary unsanitized code provided by the operator through the NiFi REST API/UI or can be used to obtain |
| or alter data on the NiFi host system using the NiFi OS credentials. These components could be used by an otherwise authorized NiFi |
| user to go beyond the intended use of the application, escalate privilege, or could expose data about the internals of the NiFi process |
| or the host system. All of these capabilities should be considered privileged, and admins should be aware of these capabilities and |
| explicitly enable them for a subset of trusted users. Before a user is allowed to create and modify restricted components they must be granted access. Hovering over the <span class="image"><img src="images/restricted.png" alt="Restricted"></span> |
| icon will display the specific permissions a restricted component requires. Permissions can be |
| assigned regardless of restrictions. In this case, the user will have access to all restricted components. Alternatively, users can |
| be assigned access to specific restrictions. If the user has been granted access to all restrictions a component requires, they will |
| have access to that component assuming otherwise sufficient permissions. For more information refer to |
| <a href="user-guide.html#UI-with-multi-tenant-authorization">Accessing the UI with Multi-Tenant Authorization</a> and <a href="user-guide.html#Restricted_Components_in_Versioned_Flows">Restricted Components in Versioned Flows</a>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Clicking the "Add" button or double-clicking on a Processor Type will add the selected Processor to the canvas at the |
| location that it was dropped.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| For any component added to the canvas, it is possible to select it with the mouse and move it anywhere on the canvas. |
| Also, it is possible to select multiple items at once by either holding down the Shift key and selecting each item or by holding |
| down the Shift key and dragging a selection box around the desired components. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="paragraph"> |
| <p>Once you have dragged a Processor onto the canvas, you can interact with it by right-clicking on the Processor and |
| selecting an option from the context menu. The options available to you from the context menu vary, depending on the privileges assigned to you.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/nifi-processor-menu.png" alt="Processor Menu"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>While the options available from the context menu vary, the following options are typically available when you have full privileges to work with a Processor:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Configure</strong>: This option allows the user to establish or change the configuration of the Processor (see <a href="user-guide.html#Configuring_a_Processor">Configuring a Processor</a>).</p> |
| </li> |
| </ul> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| For Processors, Ports, Remote Process Groups, Connections and Labels, it is possible to open the configuration dialog by double-clicking on the desired component. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Start</strong> or <strong>Stop</strong>: This option allows the user to start or stop a Processor; the option will be either Start or Stop, depending on the current state of the Processor.</p> |
| </li> |
| <li> |
| <p><strong>Run Once</strong>: This option allows the user to run a selected Processor exactly once. If the Processor is prevented from executing (e.g., there are no incoming FlowFiles or the outgoing connection has back pressure applied) the Processor won’t get triggered. <strong>Execution</strong> settings apply (i.e., <strong>Primary Node</strong> and <strong>All Nodes</strong> settings will result in running the Processor only once on the Primary Node or one time on each of the nodes, respectively). Works only with <strong>Timer driven</strong> and <strong>CRON driven</strong> scheduling strategies.</p> |
| </li> |
| <li> |
| <p><strong>Enable</strong> or <strong>Disable</strong>: This option allows the user to enable or disable a Processor; the option will be either Enable or Disable, depending on the current state of the Processor.</p> |
| </li> |
| <li> |
| <p><strong>View data provenance</strong>: This option displays the NiFi Data Provenance table, with information about data provenance events for the FlowFiles routed through that Processor (see <a href="user-guide.html#data_provenance">Data Provenance</a>).</p> |
| </li> |
| <li> |
| <p><strong>Replay last event</strong>: This option will replay the last Provenance event, effectively requeuing the last FlowFile that was processed by the Processor (see <a href="user-guide.html#replay_flowfile">Replaying a FlowFile</a>).</p> |
| </li> |
| <li> |
| <p><strong>View status history</strong>: This option opens a graphical representation of the Processor’s statistical information over time.</p> |
| </li> |
| <li> |
| <p><strong>View usage</strong>: This option takes the user to the Processor’s usage documentation.</p> |
| </li> |
| <li> |
| <p><strong>View connections→Upstream</strong>: This option allows the user to see and "jump to" upstream connections that are coming into the Processor. This is particularly useful when processors connect into and out of other Process Groups.</p> |
| </li> |
| <li> |
| <p><strong>View connections→Downstream</strong>: This option allows the user to see and "jump to" downstream connections that are going out of the Processor. This is particularly useful when processors connect into and out of other Process Groups.</p> |
| </li> |
| <li> |
| <p><strong>Center in view</strong>: This option centers the view of the canvas on the given Processor.</p> |
| </li> |
| <li> |
| <p><strong>Change color</strong>: This option allows the user to change the color of the Processor, which can make the visual management of large flows easier.</p> |
| </li> |
| <li> |
| <p><strong>Create template</strong>: This option allows the user to create a template from the selected Processor.</p> |
| </li> |
| <li> |
| <p><strong>Copy</strong>: This option places a copy of the selected Processor on the clipboard, so that it may be pasted elsewhere on the canvas by right-clicking on the canvas and selecting Paste. The Copy/Paste actions also may be done using the keystrokes Ctrl-C (Command-C) and Ctrl-V (Command-V).</p> |
| </li> |
| <li> |
| <p><strong>Delete</strong>: This option allows the DFM to delete a Processor from the canvas.</p> |
| </li> |
| </ul> |
| </div> |
| <div id="input_port" class="paragraph"> |
| <p><span class="image"><img src="images/iconInputPort.png" alt="Input Port" width="32"></span> |
| <strong>Input Port</strong>: Input Ports provide a mechanism for transferring data into a Process Group. When an Input Port is dragged |
| onto the canvas, the DFM is prompted to name the Port. All Ports within a Process Group must have unique names.</p> |
| </div> |
| <div class="paragraph"> |
| <p>All components exist only within a Process Group. When a user initially navigates to the NiFi page, the user is placed |
| in the Root Process Group. If the Input Port is dragged onto the Root Process Group, the Input Port provides a mechanism |
| to receive data from remote instances of NiFi via <a href="user-guide.html#site-to-site">Site-to-Site</a>. In this case, the Input Port can be configured |
| to restrict access to appropriate users, if NiFi is configured to run securely. For information on configuring NiFi to run |
| securely, see the |
| <a href="administration-guide.html">System Administrator’s Guide</a>.</p> |
| </div> |
| <div id="output_port" class="paragraph"> |
| <p><span class="image"><img src="images/iconOutputPort.png" alt="Output Port" width="32"></span> |
| <strong>Output Port</strong>: Output Ports provide a mechanism for transferring data from a Process Group to destinations outside |
| of the Process Group. When an Output Port is dragged onto the canvas, the DFM is prompted to name the Port. All Ports |
| within a Process Group must have unique names.</p> |
| </div> |
| <div class="paragraph"> |
| <p>If the Output Port is dragged onto the Root Process Group, the Output Port provides a mechanism for sending data to |
| remote instances of NiFi via <a href="user-guide.html#site-to-site">Site-to-Site</a>. In this case, the Port acts as a queue. As remote instances |
| of NiFi pull data from the port, that data is removed from the queues of the incoming Connections. If NiFi is configured |
| to run securely, the Output Port can be configured to restrict access to appropriate users. For information on configuring |
| NiFi to run securely, see the |
| <a href="administration-guide.html">System Administrator’s Guide</a>.</p> |
| </div> |
| <div id="process_group" class="paragraph"> |
| <p><span class="image"><img src="images/iconProcessGroup.png" alt="Process Group" width="32"></span> |
| <strong>Process Group</strong>: Process Groups can be used to logically group a set of components so that the dataflow is easier to understand |
| and maintain. When a Process Group is dragged onto the canvas, the DFM is prompted to name the Process Group. The Process Group will |
| then be nested within that parent group.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Once you have dragged a Process Group onto the canvas, you can interact with it by right-clicking on the Process Group and selecting an option from the |
| context menu. The options available to you from the context menu vary, depending on the privileges assigned to you.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/nifi-process-group-menu.png" alt="Process Group Menu"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>While the options available from the context menu vary, the following options are typically available when you have full privileges to work with the Process Group:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Configure</strong>: This option allows the user to establish or change the configuration of the Process Group.</p> |
| </li> |
| <li> |
| <p><strong>Variables</strong>: This option allows the user to create or configure variables within the NiFi UI.</p> |
| </li> |
| <li> |
| <p><strong>Enter group</strong>: This option allows the user to enter the Process Group.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| It is also possible to double-click on the Process Group to enter it. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Start</strong>: This option allows the user to start a Process Group.</p> |
| </li> |
| <li> |
| <p><strong>Stop</strong>: This option allows the user to stop a Process Group.</p> |
| </li> |
| <li> |
| <p><strong>Enable</strong>: This option allows the user to enable all Processors in the Process Group.</p> |
| </li> |
| <li> |
| <p><strong>Disable</strong>: This option allows the user to disable all Processors in the Process Group.</p> |
| </li> |
| <li> |
| <p><strong>View status history</strong>: This option opens a graphical representation of the Process Group’s statistical information over time.</p> |
| </li> |
| <li> |
| <p><strong>View connections→Upstream</strong>: This option allows the user to see and "jump to" upstream connections that are coming into the Process Group.</p> |
| </li> |
| <li> |
| <p><strong>View connections→Downstream</strong>: This option allows the user to see and "jump to" downstream connections that are going out of the Process Group.</p> |
| </li> |
| <li> |
| <p><strong>Center in view</strong>: This option centers the view of the canvas on the given Process Group.</p> |
| </li> |
| <li> |
| <p><strong>Group</strong>: This option allows the user to create a new Process Group that contains the selected Process Group and any other components selected on the canvas.</p> |
| </li> |
| <li> |
| <p><strong>Download flow definition</strong>: This option allows the user to download the flow definition of the process group as a JSON file. The file can be used as a backup or imported into a <a href="https://nifi.apache.org/registry.html" target="_blank" rel="noopener">NiFi Registry</a> using the <a href="toolkit-guide.html#nifi_CLI">NiFi CLI</a>. There are two options when downloading a flow definition:</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>→ <strong>Without external services</strong>: Controller services referenced by the selected process group but outside its scope (e.g., services in a parent group) <em>will not be</em> included in the flow definition as services.</p> |
| </li> |
| <li> |
| <p>→ <strong>With external services</strong>: Controller services referenced by the selected process group but outside its scope (e.g., services in a parent group) <em>will be</em> included in the flow definition.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p><strong>Create template</strong>: This option allows the user to create a template from the selected Process Group.</p> |
| </li> |
| <li> |
| <p><strong>Copy</strong>: This option places a copy of the selected Process Group on the clipboard, so that it may be pasted elsewhere on the canvas by right-clicking on the canvas and selecting Paste. The Copy/Paste actions also may be done using the keystrokes Ctrl-C (Command-C) and Ctrl-V (Command-V).</p> |
| </li> |
| <li> |
| <p><strong>Empty all queues</strong>: This option allows the user to empty all queues in the selected Process Group. All FlowFiles from all connections waiting at the time of the request will be removed.</p> |
| </li> |
| <li> |
| <p><strong>Delete</strong>: This option allows the DFM to delete a Process Group.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>(Note: If "Download flow definition" is selected for a versioned process group, there is no versioning information in the download. In other words, the resulting contents of the JSON file is the same whether the process group is versioned or not.)</p> |
| </div> |
| <div id="remote_process_group" class="paragraph"> |
| <p><span class="image"><img src="images/iconRemoteProcessGroup.png" alt="Remote Process Group" width="32"></span> |
| <strong>Remote Process Group</strong>: Remote Process Groups appear and behave similar to Process Groups. However, the Remote Process Group (RPG) references a remote instance of NiFi. When an RPG is dragged onto the canvas, rather than being prompted for a name, the DFM is prompted for the URL of the remote NiFi instance. If the remote NiFi is a clustered instance, adding two or more cluster node URLs is recommended so that an initial connection can be made even if one of the nodes is unavailable. Multiple URLs can be specified in a comma-separated format.</p> |
| </div> |
| <div class="paragraph"> |
| <p>When data is transferred to a clustered instance of NiFi via an RPG, the RPG will first connect to the remote instance whose URL is configured to determine which nodes are in the cluster and how busy each node is. This information is then used to load balance the data that is pushed to each node. The remote instances are then interrogated periodically to determine information about any nodes that are dropped from or added to the cluster and to recalculate the load balancing based on each node’s load. For more information, see the section on <a href="user-guide.html#site-to-site">Site-to-Site</a>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Once you have dragged a Remote Process Group onto the canvas, you can interact with it by right-clicking on the Remote Process Group and selecting an option from the context menu. The options available to you from the menu vary, depending on the privileges assigned to you.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/nifi-rpg-menu.png" alt="Remote Process Group Menu"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The following options are typically available when you have full privileges to work with the Remote Process Group:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Configure</strong>: This option allows the user to establish or change the configuration of the Remote Process Group.</p> |
| </li> |
| <li> |
| <p><strong>Enable transmission</strong>: Makes the transmission of data between NiFi instances active (see <a href="user-guide.html#Remote_Group_Transmission">Remote Process Group Transmission</a>).</p> |
| </li> |
| <li> |
| <p><strong>Disable transmission</strong>: Disables the transmission of data between NiFi instances.</p> |
| </li> |
| <li> |
| <p><strong>View status history</strong>: This option opens a graphical representation of the Remote Process Group’s statistical information over time.</p> |
| </li> |
| <li> |
| <p><strong>View connections→Upstream</strong>: This option allows the user to see and "jump to" upstream connections that are coming into the Remote Process Group.</p> |
| </li> |
| <li> |
| <p><strong>View connections→Downstream</strong>: This option allows the user to see and "jump to" downstream connections that are going out of the Remote Process Group.</p> |
| </li> |
| <li> |
| <p><strong>Refresh remote</strong>: This option refreshes the view of the status of the remote NiFi instance.</p> |
| </li> |
| <li> |
| <p><strong>Manage remote ports</strong>: This option allows the user to see input ports and/or output ports that exist on the remote instance of NiFi that the Remote Process Group is connected to. Note that if the Site-to-Site configuration is secure, only the ports that the connecting NiFi has been given accessed to will be visible.</p> |
| </li> |
| <li> |
| <p><strong>Center in view</strong>: This option centers the view of the canvas on the given Remote Process Group.</p> |
| </li> |
| <li> |
| <p><strong>Go to</strong>: This option opens a view of the remote NiFi instance in a new tab of the browser. Note that if the Site-to-Site configuration is secure, the user must have access to the remote NiFi instance in order to view it.</p> |
| </li> |
| <li> |
| <p><strong>Group</strong>: This option allows the user to create a new Process Group that contains the selected Remote Process Group and any other components selected on the canvas.</p> |
| </li> |
| <li> |
| <p><strong>Create template</strong>: This option allows the user to create a template from the selected Remote Process Group.</p> |
| </li> |
| <li> |
| <p><strong>Copy</strong>: This option places a copy of the selected Process Group on the clipboard, so that it may be pasted elsewhere on the canvas by right-clicking on the canvas and selecting Paste. The Copy/Paste actions also may be done using the keystrokes Ctrl-C (Command-C) and Ctrl-V (Command-V).</p> |
| </li> |
| <li> |
| <p><strong>Delete</strong>: This option allows the DFM to delete a Remote Process Group from the canvas.</p> |
| </li> |
| </ul> |
| </div> |
| <div id="funnel" class="paragraph"> |
| <p><span class="image"><img src="images/iconFunnel.png" alt="Funnel"></span> |
| <strong>Funnel</strong>: Funnels are used to combine the data from many Connections into a single Connection. This has two advantages. |
| First, if many Connections are created with the same destination, the canvas can become cluttered if those Connections |
| have to span a large space. By funneling these Connections into a single Connection, that single Connection can then be |
| drawn to span that large space instead. Secondly, Connections can be configured with FlowFile Prioritizers. Data from |
| several Connections can be funneled into a single Connection, providing the ability to Prioritize all of the data on that |
| one Connection, rather than prioritizing the data on each Connection independently.</p> |
| </div> |
| <div id="template" class="paragraph"> |
| <p><span class="image"><img src="images/iconTemplate.png" alt="Template"></span> |
| <strong>Template</strong>: Templates can be created by DFMs from sections of the flow, or they can be imported from other |
| dataflows. These Templates provide larger building blocks for creating a complex flow quickly. When the Template is |
| dragged onto the canvas, the DFM is provided a dialog to choose which Template to add to the canvas:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/instantiate-template.png" alt="Instantiate Template Dialog"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Clicking the drop-down box shows all available Templates. Any Template that was created with a description will show a question mark |
| icon, indicating that there is more information. Hovering over the icon with the mouse will show this description:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/instantiate-template-description.png" alt="Instantiate Template Dialog"> |
| </div> |
| </div> |
| <div id="label" class="paragraph"> |
| <p><span class="image"><img src="images/iconLabel.png" alt="Label"></span> |
| <strong>Label</strong>: Labels are used to provide documentation to parts of a dataflow. When a Label is dropped onto the canvas, |
| it is created with a default size. The Label can then be resized by dragging the handle in the bottom-right corner. |
| The Label has no text when initially created. The text of the Label can be added by right-clicking on the Label and |
| choosing <code>Configure</code>.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="component-versioning"><a class="anchor" href="user-guide.html#component-versioning"></a>Component Versions</h3> |
| <div class="paragraph"> |
| <p>You have access to information about the version of your Processors, Controller Services, and Reporting Tasks. |
| This is especially useful when you are working within a clustered environment with multiple NiFi instances running |
| different versions of a component or if you have upgraded to a newer version of a processor. The Add Processor, |
| Add Controller Service, and Add Reporting Task dialogs include a column identifying the component version, as well |
| as the name of the component, the organization or group that created the component, and the NAR bundle that contains |
| the component.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/add-processor-version-example.png" alt="Add Processor Version Example"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Each component displayed on the canvas also contains this information.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/processor-version-information-example.png" alt="Processor Version Information Example"> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="sorting-and-filtering-components"><a class="anchor" href="user-guide.html#sorting-and-filtering-components"></a>Sorting and Filtering Components</h4> |
| <div class="paragraph"> |
| <p>When you are adding a component, you can sort on version number or filter based on originating source.</p> |
| </div> |
| <div class="paragraph"> |
| <p>To sort based on version, click the version column to display in ascending or descending version order.</p> |
| </div> |
| <div class="paragraph"> |
| <p>To filter based on source group, click the source drop-down in the upper left of your Add Component dialog, |
| and select the group you want to view.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/add-processor-version-sort-filter.png" alt="Add Processor Version Sort and Filter"> |
| </div> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="changing-component-versions"><a class="anchor" href="user-guide.html#changing-component-versions"></a>Changing Component Versions</h4> |
| <div class="paragraph"> |
| <p>To change a component version, perform the following steps.</p> |
| </div> |
| <div class="olist arabic"> |
| <ol class="arabic"> |
| <li> |
| <p>Right-click the component on the canvas to display configuration options.</p> |
| </li> |
| <li> |
| <p>Select Change version.</p> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/processor-change-version.png" alt="Processor Change Version"> |
| </div> |
| </div> |
| </li> |
| <li> |
| <p>In the Component Version dialog, select the version you want to run from the Version drop-down menu.</p> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/component-version-dialog.png" alt="Component Version"> |
| </div> |
| </div> |
| </li> |
| </ol> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="understanding-version-dependencies"><a class="anchor" href="user-guide.html#understanding-version-dependencies"></a>Understanding Version Dependencies</h4> |
| <div class="paragraph"> |
| <p>When you are configuring a component, you can also view information about version dependencies.</p> |
| </div> |
| <div class="olist arabic"> |
| <ol class="arabic"> |
| <li> |
| <p>Right-click your component and select Configure to display the Configure dialog for your component.</p> |
| </li> |
| <li> |
| <p>Click the Properties tab.</p> |
| </li> |
| <li> |
| <p>Click the information icon to view any version dependency information.</p> |
| </li> |
| </ol> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/configure-processor-with-version-information.png" alt="Configuration Version Requirements"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>In the following example, MyProcessor version 1.0 is configured properly with the controller service StandardMyService version 1.0:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/processor-cs-version-match.png" alt="Processor and Controller Service Version Match"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>If the version of MyProcessor is changed to an incompatible version (MyProcessor 2.0), validation errors will be displayed on the processor:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/processor-cs-version-mismatch-warnings.png" alt="Processor and Controller Service Version Mismatch Warnings"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>and an error message will be displayed in the processor’s controller service configuration since the service is no longer valid:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/processor-cs-version-mismatch-config.png" alt="Processor and Controller Service Version Mismatch Property"> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="Configuring_a_Processor"><a class="anchor" href="user-guide.html#Configuring_a_Processor"></a>Configuring a Processor</h3> |
| <div class="paragraph"> |
| <p>To configure a processor, right-click on the Processor and select the <code>Configure</code> option from the context menu. Alternatively, just double-click on the Processor. The configuration dialog is opened with four |
| different tabs, each of which is discussed below. Once you have finished configuring the Processor, you can apply |
| the changes by clicking "Apply" or cancel all changes by clicking "Cancel".</p> |
| </div> |
| <div class="paragraph"> |
| <p>Note that after a Processor has been started, the context menu shown for the Processor no longer has a <code>Configure</code> |
| option but rather has a <code>View Configuration</code> option. Processor configuration cannot be changed while the Processor is |
| running. You must first stop the Processor and wait for all of its active tasks to complete before configuring |
| the Processor again.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Note that entering certain control characters are not supported and will be automatically filtered out when entered. The following characters and any |
| unpaired Unicode surrogate codepoints will not be retained in any configuration:</p> |
| </div> |
| <div class="literalblock"> |
| <div class="content"> |
| <pre>[#x0], [#x1], [#x2], [#x3], [#x4], [#x5], [#x6], [#x7], [#x8], [#xB], [#xC], [#xE], [#xF], [#x10], [#x11], [#x12], [#x13], [#x14], [#x15], [#x16], [#x17], [#x18], [#x19], [#x1A], [#x1B], [#x1C], [#x1D], [#x1E], [#x1F], [#xFFFE], [#xFFFF]</pre> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="settings-tab"><a class="anchor" href="user-guide.html#settings-tab"></a>Settings Tab</h4> |
| <div class="paragraph"> |
| <p>The first tab in the Processor Configuration dialog is the Settings tab:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/settings-tab.png" alt="Settings Tab"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>This tab contains several different configuration items. First, it allows the DFM to change the name of the Processor. |
| The name of a Processor by default is the same as the Processor type. Next to the Processor Name is a checkbox, indicating |
| whether the Processor is Enabled. When a Processor is added to the canvas, it is enabled. If the |
| Processor is disabled, it cannot be started. The disabled state is used to indicate that when a group of Processors is started, |
| such as when a DFM starts an entire Process Group, this (disabled) Processor should be excluded.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Below the Name configuration, the Processor’s unique identifier is displayed along with the Processor’s type and NAR bundle. These values cannot be modified.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Next are two dialogues for configuring 'Penalty Duration' and 'Yield Duration'. During the normal course of processing a |
| piece of data (a FlowFile), an event may occur that indicates that the data cannot be processed at this time but the |
| data may be processable at a later time. When this occurs, the Processor may choose to Penalize the FlowFile. This will |
| prevent the FlowFile from being Processed for some period of time. For example, if the Processor is to push the data |
| to a remote service, but the remote service already has a file with the same name as the filename that the Processor |
| is specifying, the Processor may penalize the FlowFile. The 'Penalty Duration' allows the DFM to specify how long the |
| FlowFile should be penalized. The default value is <code>30 seconds</code>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Similarly, the Processor may determine that some situation exists such that the Processor can no longer make any progress, |
| regardless of the data that it is processing. For example, if a Processor is to push data to a remote service and that |
| service is not responding, the Processor cannot make any progress. As a result, the Processor should 'yield', which will |
| prevent the Processor from being scheduled to run for some period of time. That period of time is specified by setting |
| the 'Yield Duration'. The default value is <code>1 second</code>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The last configurable option on the left-hand side of the Settings tab is the Bulletin level. Whenever the Processor writes |
| to its log, the Processor also will generate a Bulletin. This setting indicates the lowest level of Bulletin that should be |
| shown in the User Interface. By default, the Bulletin level is set to <code>WARN</code>, which means it will display all warning and error-level |
| bulletins.</p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="scheduling-tab"><a class="anchor" href="user-guide.html#scheduling-tab"></a>Scheduling Tab</h4> |
| <div class="paragraph"> |
| <p>The second tab in the Processor Configuration dialog is the Scheduling Tab:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/scheduling-tab.png" alt="Scheduling Tab"> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="scheduling-strategy"><a class="anchor" href="user-guide.html#scheduling-strategy"></a>Scheduling Strategy</h5> |
| <div class="paragraph"> |
| <p>The first configuration option is the Scheduling Strategy. There are three possible options for scheduling components:</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Timer driven</strong>: This is the default mode. The Processor will be scheduled to run on a regular interval. The interval |
| at which the Processor is run is defined by the 'Run Schedule' option (see below).</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Event driven</strong>: When this mode is selected, the Processor will be triggered to run by an event, and that event occurs when FlowFiles enter Connections |
| feeding this Processor. This mode is currently considered experimental and is not supported by all Processors. When this mode is |
| selected, the 'Run Schedule' option is not configurable, as the Processor is not triggered to run periodically but |
| as the result of an event. Additionally, this is the only mode for which the 'Concurrent Tasks' |
| option can be set to 0. In this case, the number of threads is limited only by the size of the Event-Driven Thread Pool that |
| the administrator has configured.</p> |
| </div> |
| <div class="admonitionblock warning"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-warning" title="Warning"></i> |
| </td> |
| <td class="content"> |
| <div class="title">Experimental</div> |
| <div class="paragraph"> |
| <p>This implementation is marked <a href="user-guide.html#experimental_warning"><strong>experimental</strong></a> as of Apache NiFi 1.10.0 (October 2019). The API, configuration, and internal behavior may change without warning, and such changes may occur during a minor release. Use at your own risk.</p> |
| </div> |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="paragraph"> |
| <p><strong>CRON driven</strong>: When using the CRON driven scheduling mode, the Processor is scheduled to run periodically, similar to the |
| Timer driven scheduling mode. However, the CRON driven mode provides significantly more flexibility at the expense of |
| increasing the complexity of the configuration. The CRON driven scheduling value is a string of six required fields and one |
| optional field, each separated by a space. These fields are:</p> |
| </div> |
| <table class="tableblock frame-all grid-all stretch"> |
| <colgroup> |
| <col style="width: 50%;"> |
| <col style="width: 50%;"> |
| </colgroup> |
| <thead> |
| <tr> |
| <th class="tableblock halign-left valign-top">Field</th> |
| <th class="tableblock halign-left valign-top">Valid values</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Seconds</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">0-59</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Minutes</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">0-59</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Hours</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">0-23</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Day of Month</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">1-31</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Month</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">1-12 or JAN-DEC</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Day of Week</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">1-7 or SUN-SAT</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Year (optional)</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">empty, 1970-2099</p></td> |
| </tr> |
| </tbody> |
| </table> |
| <div class="paragraph"> |
| <p>You typically specify values one of the following ways:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Number</strong>: Specify one or more valid value. You can enter more than one value using a comma-separated list.</p> |
| </li> |
| <li> |
| <p><strong>Range</strong>: Specify a range using the <number>-<number> syntax.</p> |
| </li> |
| <li> |
| <p><strong>Increment</strong>: Specify an increment using <start value>/<increment> syntax. For example, in the Minutes field, 0/15 indicates the minutes 0, 15, 30, and 45.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>You should also be aware of several valid special characters:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>*  — Indicates that all values are valid for that field.</p> |
| </li> |
| <li> |
| <p>?  — Indicates that no specific value is specified. This special character is valid in the Days of Month and Days of Week field.</p> |
| </li> |
| <li> |
| <p>L  — You can append L to one of the Day of Week values, to specify the last occurrence of this day in the month. For |
| example, 1L indicates the last Sunday of the month.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>For example:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>The string <code>0 0 13 * * ?</code> indicates that you want to schedule the processor to run at 1:00 PM every day.</p> |
| </li> |
| <li> |
| <p>The string <code>0 20 14 ? * MON-FRI</code> indicates that you want to schedule the processor to run at 2:20 PM every Monday through Friday.</p> |
| </li> |
| <li> |
| <p>The string <code>0 15 10 ? * 6L 2011-2017</code> indicates that you want to schedule the processor to run at 10:15 AM, on the last Friday of every month, between 2011 and 2017.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>For additional information and examples, see the <a href="https://www.quartz-scheduler.org/documentation/quartz-2.2.2/tutorials/tutorial-lesson-06.html">cron triggers tutorial</a> in the <a href="https://www.quartz-scheduler.org/documentation/" target="_blank" rel="noopener">Quartz Documentation</a>.</p> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="concurrent-tasks"><a class="anchor" href="user-guide.html#concurrent-tasks"></a>Concurrent Tasks</h5> |
| <div class="paragraph"> |
| <p>Next, the Scheduling tab provides a configuration option named 'Concurrent Tasks'. This controls how many threads the Processor |
| will use. Said a different way, this controls how many FlowFiles should be processed by this Processor at the same time. Increasing |
| this value will typically allow the Processor to handle more data in the same amount of time. However, it does this by using system |
| resources that then are not usable by other Processors. This essentially provides a relative weighting of Processors — it controls |
| how much of the system’s resources should be allocated to this Processor instead of other Processors. This field is available for |
| most Processors. There are, however, some types of Processors that can only be scheduled with a single Concurrent task.</p> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="run-schedule"><a class="anchor" href="user-guide.html#run-schedule"></a>Run Schedule</h5> |
| <div class="paragraph"> |
| <p>The 'Run Schedule' dictates how often the Processor should be scheduled to run. The valid values for this field depend on the selected |
| Scheduling Strategy (see above). If using the Event driven Scheduling Strategy, this field is not available. When using the Timer driven |
| Scheduling Strategy, this value is a time duration specified by a number followed by a time unit. For example, <code>1 second</code> or <code>5 mins</code>. |
| The default value of <code>0 sec</code> means that the Processor should run as often as possible as long as it has data to process. This is true |
| for any time duration of 0, regardless of the time unit (e.g., <code>0 sec</code>, <code>0 mins</code>, <code>0 days</code>). For an explanation of values that are |
| applicable for the CRON driven Scheduling Strategy, see the description of the CRON driven Scheduling Strategy itself.</p> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="execution"><a class="anchor" href="user-guide.html#execution"></a>Execution</h5> |
| <div class="paragraph"> |
| <p>The Execution setting is used to determine on which node(s) the Processor will be |
| scheduled to execute. Selecting 'All Nodes' will result in this Processor being scheduled on every node in the cluster. Selecting |
| 'Primary Node' will result in this Processor being scheduled on the Primary Node only. Processors that have been configured for 'Primary Node' execution are identified by a "P" next to the processor icon:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/primary-node-processor.png" alt="Primary Node Processor"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>To quickly identify 'Primary Node' processors, the "P" icon is also shown in the Processors tab on the Summary page:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/primary-node-processors-summary.png" alt="Primary Node Processors in Summary Page"> |
| </div> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="run-duration"><a class="anchor" href="user-guide.html#run-duration"></a>Run Duration</h5> |
| <div class="paragraph"> |
| <p>The right-hand side of the Scheduling tab contains a slider for choosing the 'Run Duration'. This controls how long the Processor should be scheduled |
| to run each time that it is triggered. On the left-hand side of the slider, it is marked 'Lower latency' while the right-hand side |
| is marked 'Higher throughput'. When a Processor finishes running, it must update the repository in order to transfer the FlowFiles to |
| the next Connection. Updating the repository is expensive, so the more work that can be done at once before updating the repository, |
| the more work the Processor can handle (Higher throughput). However, this means that the next Processor cannot start processing |
| those FlowFiles until the previous Process updates this repository. As a result, the latency will be longer (the time required to process |
| the FlowFile from beginning to end will be longer). As a result, the slider provides a spectrum from which the DFM can choose to favor |
| Lower Latency or Higher Throughput.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="properties-tab"><a class="anchor" href="user-guide.html#properties-tab"></a>Properties Tab</h4> |
| <div class="paragraph"> |
| <p>The Properties tab provides a mechanism to configure Processor-specific behavior. There are no default properties. Each type of Processor |
| must define which Properties make sense for its use case. Below, we see the Properties tab for a RouteOnAttribute Processor:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/properties-tab.png" alt="Properties Tab"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>This Processor, by default, has only a single property: 'Routing Strategy'. The default value is 'Route to Property name'. Next to |
| the name of this property is a small question mark symbol ( |
| <span class="image"><img src="images/iconInfo.png" alt="Info"></span> |
| ). This help symbol is seen in other places throughout the User Interface, and it indicates that more information is available. |
| Hovering over this symbol with the mouse will provide additional details about the property and the default value, as well as |
| historical values that have been set for the Property.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Clicking on the value for the property will allow a DFM to change the value. Depending on the values that are allowed for the property, |
| the user is either provided a drop-down from which to choose a value or is given a text area to type a value:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/edit-property-dropdown.png" alt="Edit Property with Dropdown"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>In the top-right corner of the tab is a button for adding a New Property. Clicking this button will provide the DFM with a dialog to |
| enter the name and value of a new property. Not all Processors allow User-Defined properties. In processors that do not allow them, |
| the Processor becomes invalid when User-Defined properties are applied.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Selected Processors support configurable Sensitive Value status for User-Defined properties. Processors must indicate |
| support for configurable Sensitive Value status, otherwise the Sensitive Value selection will be disabled in the |
| Add Property dialog.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/add-property-sensitive-value-dialog.png" alt="Add Property with Sensitive Value status"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Selecting <code>Yes</code> for the Sensitive Value setting instructs NiFi to handle the property value as |
| sensitive for configuration persistence and framework operations. NiFi encrypts Sensitive Values when storing the flow |
| configuration and does not include Sensitive Values in exported Flow Definitions.</p> |
| </div> |
| <div class="paragraph"> |
| <p>RouteOnAttribute allows User-Defined properties and will not be valid until the user has added a property.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/edit-property-textarea.png" alt="Edit Property with Text Area"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Note that after a User-Defined property has been added, an icon will appear on the right-hand side of that row ( |
| <span class="image"><img src="images/iconDelete.png" alt="Delete Icon"></span> |
| ). Clicking it will remove the User-Defined property from the Processor.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Some processors also have an Advanced User Interface (UI) built into them. For example, the UpdateAttribute processor has an Advanced UI. To access the Advanced UI, click the "Advanced" button that appears at the bottom of the Configure Processor window. Only processors that have an Advanced UI will have this button.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Some processors have properties that refer to other components, such as Controller Services, which also need to be configured. For example, the GetHTTP processor has an SSLContextService property, which refers to the StandardSSLContextService controller service. When DFMs want to configure this property but have not yet created and configured the controller service, they have the option to create the service on the spot, as depicted in the image below. For more information about configuring Controller Services, see the <a href="user-guide.html#Controller_Services">Controller Services</a> section.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/create-service-ssl-context.png" alt="Create Service"></span></p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="relationships-tab"><a class="anchor" href="user-guide.html#relationships-tab"></a>Relationships Tab</h4> |
| <div class="paragraph"> |
| <p>The Relationships tab contains an 'Automatically Terminate / Retry Relationships' section. Each of the Relationships that is defined by the Processor is listed here, along with its description.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/relationships-tab.png" alt="Relationships Tab"> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="automatically-terminate"><a class="anchor" href="user-guide.html#automatically-terminate"></a>Automatically Terminate</h5> |
| <div class="paragraph"> |
| <p>In order for a Processor to be considered valid and able to run, each Relationship defined by the Processor must be either connected to a downstream component or auto-terminated. If a Relationship is auto-terminated, any FlowFile that is routed to that Relationship will be removed from the flow and its processing considered complete. Any Relationship that is already connected to a downstream component cannot be auto-terminated. The Relationship must first be removed from any Connection that uses it. Additionally, for any Relationship that is selected to be auto-terminated, the auto-termination status will be cleared (turned off) if the Relationship is added to a Connection.</p> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="automatically-retry"><a class="anchor" href="user-guide.html#automatically-retry"></a>Automatically Retry</h5> |
| <div class="paragraph"> |
| <p>Users can also configure whether or not FlowFiles routed to a given Relationship should be retried.</p> |
| </div> |
| <div class="sect5"> |
| <h6 id="number-of-retry-attempts"><a class="anchor" href="user-guide.html#number-of-retry-attempts"></a>Number of Retry Attempts</h6> |
| <div class="paragraph"> |
| <p>For relationships set to retry, this number indicates how many times a FlowFile will attempt to reprocess before it is routed elsewhere.</p> |
| </div> |
| </div> |
| <div class="sect5"> |
| <h6 id="retry-back-off-policy"><a class="anchor" href="user-guide.html#retry-back-off-policy"></a>Retry Back Off Policy</h6> |
| <div class="paragraph"> |
| <p>When a FlowFile is to be retried, the user can configure the backoff policy with two options:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Penalize - Retry attempts will occur in time, but the processor will continue to process other FlowFiles.</p> |
| </li> |
| <li> |
| <p>Yield - No other FlowFile processing will occur until all retry attempts have been made.</p> |
| </li> |
| </ul> |
| </div> |
| </div> |
| <div class="sect5"> |
| <h6 id="retry-maximum-back-off-period"><a class="anchor" href="user-guide.html#retry-maximum-back-off-period"></a>Retry Maximum Back Off Period</h6> |
| <div class="paragraph"> |
| <p>Initial retries are based on the Penalty/Yield Duration time specified in the Settings tab. The duration time is repeatedly doubled for every subsequent retry attempt. This number indicates the maximum allowable time period before another retry attempt occurs.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| If both terminate and retry are selected, any retry logic will happen first, then auto-termination. |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="comments-tab"><a class="anchor" href="user-guide.html#comments-tab"></a>Comments Tab</h4> |
| <div class="paragraph"> |
| <p>The last tab in the Processor configuration dialog is the Comments tab. This tab simply provides an area for users to include |
| whatever comments are appropriate for this component. Use of the Comments tab is optional:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/comments-tab.png" alt="Comments Tab"> |
| </div> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="additional-help"><a class="anchor" href="user-guide.html#additional-help"></a>Additional Help</h4> |
| <div class="paragraph"> |
| <p>You can access additional documentation about each Processor’s usage by right-clicking on the Processor and selecting 'Usage' from the context menu. Alternatively, select Help from the Global Menu in the top-right corner of the UI to display a Help page with all of the documentation, including usage documentation for all the Processors that are available. Click on the desired Processor to view usage documentation.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="Configuring_a_ProcessGroup"><a class="anchor" href="user-guide.html#Configuring_a_ProcessGroup"></a>Configuring a Process Group</h3> |
| <div class="paragraph"> |
| <p>To configure a Process Group, right-click on the Process Group and select the <code>Configure</code> option from the context menu. The configuration dialog is opened with two tabs: General and Controller Services.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/process-group-configuration-window.png" alt="Configure Process Group"> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="General_tab_ProcessGroup"><a class="anchor" href="user-guide.html#General_tab_ProcessGroup"></a>General Tab</h4> |
| <div class="paragraph"> |
| <p>This tab contains several different configuration items. First is the Process Group Name. This is the name that is shown at the top of the Process Group on the canvas as well as in the breadcrumbs at the bottom of the UI. For the Root Process Group (i.e., the highest level group), this is also the name that is shown as the title of the browser tab. Note that this information is visible to any other NiFi instance that connects remotely to this instance (using Remote Process Groups, a.k.a., <a href="user-guide.html#site-to-site">Site-to-Site</a>).</p> |
| </div> |
| <div class="paragraph"> |
| <p>The next configuration element is the Process Group Parameter Context, which is used to provide parameters to components of the flow. From this drop-down, the user is able to choose which Parameter Context should be bound to this Process Group and can optionally create a new one to bind to the Process Group. For more information refer to <a href="user-guide.html#Parameters">Parameters</a> and <a href="user-guide.html#parameter-contexts">Parameter Contexts</a>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The third element in the configuration dialog is the Process Group Comments. This provides a mechanism to add any useful information about the Process Group.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The next two elements, Process Group FlowFile Concurrency and Process Group Outbound Policy, are covered in the following sections.</p> |
| </div> |
| <div class="sect4"> |
| <h5 id="Flowfile_Concurrency"><a class="anchor" href="user-guide.html#Flowfile_Concurrency"></a>FlowFile Concurrency</h5> |
| <div class="paragraph"> |
| <p>FlowFile Concurrency is used to control how data is brought into the Process Group. There are three options available:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Unbounded (the default)</p> |
| </li> |
| <li> |
| <p>Single FlowFile Per Node</p> |
| </li> |
| <li> |
| <p>Single Batch Per Node</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>When the FlowFile Concurrency is set to "Unbounded", the Input Ports in the Process Group will ingest data as quickly as they |
| are able, provided that back pressure does not prevent them from doing so.</p> |
| </div> |
| <div class="paragraph"> |
| <p>When the FlowFile Concurrency is configured to "Single FlowFile Per Node", the Input Ports will only allow a single FlowFile through at at time. |
| Once that FlowFile enters the Process Group, no additional FlowFiles will be brought in until all FlowFiles have left the Process Group (either by |
| being removed from the system/auto-terminated, or by exiting through an Output Port). This will often result in slower performance, as it reduces |
| the parallelization that NiFi uses to process the data. However, there are several reasons that a user may want to use this approach. A common use case |
| is one in which each incoming FlowFile contains references to several other data items, such as a list of files in a directory. The user may want to |
| process the entire listing before allowing any other data to enter the Process Group.</p> |
| </div> |
| <div class="paragraph"> |
| <p>When the FlowFile Concurrency is configured to "Single Batch Per Node", the Input Ports will behave similarly to the way that they behave in the |
| "Single FlowFile Per Node" mode, but when a FlowFile is ingested, the Input Ports will continue to ingest all data until all of the queues feeding |
| the Input Ports have been emptied. At that point, they will not bring any more data into the Process Group until all data has finished processing and |
| has left the Process Group (see <a href="user-guide.html#Connecting_Batch_Oriented_Groups">Connecting Batch-Oriented Process Groups</a>).</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| The FlowFile Concurrency controls only when data will be pulled into the Process Group from an Input Port. It does not prevent a Processor within the |
| Process Group from ingesting data from outside of NiFi. |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="Outbound_Policy"><a class="anchor" href="user-guide.html#Outbound_Policy"></a>Outbound Policy</h5> |
| <div class="paragraph"> |
| <p>While the FlowFile Concurrency dictates how data should be brought into the Process Group, the Outbound Policy controls the flow of data out of the Process Group. |
| There are two available options available:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Stream When Available (the default)</p> |
| </li> |
| <li> |
| <p>Batch Output</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>When the Outbound Policy is configured to "Stream When Available", |
| data that arrives at an Output Port is immediately transferred out of the Process Group, assuming that no back pressure is applied.</p> |
| </div> |
| <div class="paragraph"> |
| <p>When the Outbound Policy is configured to "Batch Output", the Output Ports will not transfer data out of the Process Group until |
| all data that is in the Process Group is queued up at an Output Port (i.e., no data leaves the Process Group until all of the data has finished processing). |
| It doesn’t matter whether the data is all queued up for the same Output Port, or if some data is queued up for Output Port A while other data is queued up |
| for Output Port B. These conditions are both considered the same in terms of the completion of the FlowFile processing.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Using an Outbound Policy of "Batch Output" along with a FlowFile Concurrency of "Single FlowFile Per Node" allows a user to easily ingest a single FlowFile |
| (which in and of itself may represent a batch of data) and then wait until all processing of that FlowFile has completed before continuing on to the next step |
| in the dataflow (i.e., the next component outside of the Process Group). Additionally, when using this mode, each FlowFile that is transferred out of the Process Group |
| will be given a series of attributes named "batch.output.<Port Name>" for each Output Port in the Process Group. The value will be equal to the number of FlowFiles |
| that were routed to that Output Port for this batch of data. For example, consider a case where a single FlowFile is split into 5 FlowFiles: two FlowFiles go to Output Port A, one goes |
| to Output Port B, two go to Output Port C, and no FlowFiles go to Output Port D. In this case, each FlowFile will have attributes <code>batch.output.A = 2</code>, |
| <code>batch.output.B = 1</code>, <code>batch.output.C = 2</code>, <code>batch.output.D = 0</code>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The Outbound Policy of "Batch Output" doesn’t provide any benefits when used in conjunction with a FlowFile Concurrency of "Unbounded". |
| As a result, the Outbound Policy is ignored if the FlowFile Concurrency is set to "Unbounded".</p> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="Connecting_Batch_Oriented_Groups"><a class="anchor" href="user-guide.html#Connecting_Batch_Oriented_Groups"></a>Connecting Batch-Oriented Process Groups</h5> |
| <div class="paragraph"> |
| <p>A common use case in NiFi is to perform some batch-oriented process and only after that process completes, perform another process on that same batch of data.</p> |
| </div> |
| <div class="paragraph"> |
| <p>NiFi makes this possible by encapsulating each of these processes in its own Process Group. The Outbound Policy of the first Process Group should be configured as "Batch Output" |
| while the FlowFile Concurrency should be either "Single FlowFile Per Node" or "Single Batch Per Node". With this configuration, the first Process Group |
| will process an entire batch of data (which will either be a single FlowFile or many FlowFiles depending on the FlowFile Concurrency) as a coherent batch of data. |
| When processing has completed for that batch of data, the data will be held until all FlowFiles are finished processing and ready to leave the Process Group. At that point, the data can be transferred out of the Process Group as a batch. This configuration - when a Process Group is configured with an Outbound Policy of "Batch Output" |
| and an Output Port is connected directly to the Input Port of a Process Group with a FlowFile Concurrency of "Single Batch Per Node" - is treated as a slightly special case. |
| The receiving Process Group will ingest data not only until its input queues are empty but until they are empty AND the source Process Group has transferred all of the data from that |
| batch out of the Process Group. This allows a collection of FlowFiles to be transferred as a single batch of data between Process Groups, even if those FlowFiles |
| are spread across multiple ports.</p> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="Flowfile_Concurrency_Caveats"><a class="anchor" href="user-guide.html#Flowfile_Concurrency_Caveats"></a>Caveats</h5> |
| <div class="paragraph"> |
| <p>When using a FlowFile Concurrency of "Single FlowFile Per Node", there are a couple of caveats to consider.</p> |
| </div> |
| <div class="paragraph"> |
| <p>First, an Input Port is free to bring data into the Process Group if there is no data queued up in that Process Group on the same node. |
| This means that in a 5-node cluster, for example, there may be up to 5 incoming FlowFiles being processed simultaneously. Additionally, |
| if a connection is configured to use <a href="user-guide.html#Load_Balancing">Load Balancing</a>, it may transfer data to another node in the cluster, allowing data to enter |
| the Process Group while that FlowFile is still being processed. As a result, it is not recommended to use Load-Balanced Connections |
| within a Process Group that is not configured for "Unbounded" FlowFile Concurrency.</p> |
| </div> |
| <div class="paragraph"> |
| <p>When using the Outbound Policy of "Batch Output", it is important to consider back pressure. Consider a case where no data will be transferred |
| out of a Process Group until all data is finished processing. Also consider that the connection to Output Port A has a back pressure threshold |
| of 10,000 FlowFiles (the default). If that queue reaches the threshold of 10,000, the upstream Processor will no longer be triggered. As a result, |
| data will not finish processing, and the flow will end in a deadlock, as the Output Port will not run until the processing completes and |
| the Processor will not run until the Output Port runs. To avoid this, if a large number of FlowFiles are expected to be generated from a single |
| input FlowFile, it is recommended that back pressure for Connections ending in an Output Port be configured in such a way to allow for the |
| largest expected number of FlowFiles or back pressure for those Connections be disabled all together (by setting the Back Pressure Threshold to 0). |
| See <a href="user-guide.html#Backpressure">Back Pressure</a> for more information.</p> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="Default_Connection_Settings"><a class="anchor" href="user-guide.html#Default_Connection_Settings"></a>Default Settings for Connections</h5> |
| <div class="paragraph"> |
| <p>The final three elements in the Process Group configuration dialog are for Default FlowFile Expiration, Default Back Pressure Object Threshold, and |
| Default Back Pressure Data Size Threshold. These settings configure the default values when creating a new Connection. Each Connection represents a queue, |
| and every queue has settings for FlowFile expiration, back pressure object count, and back pressure data size. The settings specified here will affect the |
| default values for all new Connections created within the Process Group; it will not affect existing Connections. Child Process Groups created within the |
| configured Process Group will inherit the default settings. Again, existing Process Groups will not be affected. If not overridden with these options, the |
| root Process Group obtains its default back pressure settings from <code>nifi.properties</code>, and has a default FlowFile expiration of "0 sec" (i.e., do not expire).</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| Setting the Default FlowFile Expiration to a non-zero value may lead to data loss due to a FlowFile expiring as its time limit is reached. |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="controller-services"><a class="anchor" href="user-guide.html#controller-services"></a>Controller Services</h4> |
| <div class="paragraph"> |
| <p>The Controller Services tab in the Process Group configuration dialog is covered in <a href="user-guide.html#Controller_Services_for_Dataflows">Adding Controller Services for Dataflows</a>.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="Parameters"><a class="anchor" href="user-guide.html#Parameters"></a>Parameters</h3> |
| <div class="paragraph"> |
| <p>The values of properties in the flow, including sensitive properties, can be parameterized using Parameters. Parameters are created and configured within the NiFi UI. Any property can be configured to reference a Parameter with the following conditions:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>A sensitive property can only reference a Sensitive Parameter</p> |
| </li> |
| <li> |
| <p>A non-sensitive property can only reference a Non-Sensitive Parameter</p> |
| </li> |
| <li> |
| <p>Properties that reference Controller Services can not use Parameters</p> |
| </li> |
| <li> |
| <p>Parameters cannot be referenced in Reporting Tasks or in Management Controller Services</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>The UI indicates whether a Parameter can be used for a property value.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/el-param-support-help-text.png" alt="Expression Language and Parameters Help Text"> |
| </div> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| Parameters have numerous advantages over <a href="user-guide.html#Variables">Variables</a>. In addition to sensitive value support, Parameters offer more granular control over access policies. Additionally, properties that reference Parameters are validated against the substituted value, unlike most properties that reference Variables using Expression Language. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="sect3"> |
| <h4 id="parameter-contexts"><a class="anchor" href="user-guide.html#parameter-contexts"></a>Parameter Contexts</h4> |
| <div class="paragraph"> |
| <p>Parameters are created within Parameter Contexts. Parameter Contexts are globally defined/accessible to the NiFi instance. Access policies can be applied to Parameter Contexts to determine which users can create them. Once created, policies to read and write to a specific Parameter Context can also be applied (see <a href="user-guide.html#accessing-parameters">Accessing Parameters</a> for more information).</p> |
| </div> |
| <div class="sect4"> |
| <h5 id="creating-a-parameter-context"><a class="anchor" href="user-guide.html#creating-a-parameter-context"></a>Creating a Parameter Context</h5> |
| <div class="paragraph"> |
| <p>To create a Parameter Context, select Parameter Contexts from the Global Menu:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/parameter-contexts-selection.png" alt="Global Menu - Parameter Contexts"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>In the Parameter Contexts window, click the <code>+</code> button in the upper-right corner and the Add Parameter Context window opens. The window has two tabs: Settings and Parameters.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/parameter-contexts-settings.png" alt="Parameter Contexts - Settings"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>On the "Settings" tab, add a name for the Parameter Context and a description if desired. Select "Apply" to save the Parameter Context or select the "Parameters" tab to add parameters to the context.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="adding-a-parameter-to-a-parameter-context"><a class="anchor" href="user-guide.html#adding-a-parameter-to-a-parameter-context"></a>Adding a Parameter to a Parameter Context</h4> |
| <div class="paragraph"> |
| <p>Parameters can be added during Parameter Context creation or added to existing Parameter Contexts.</p> |
| </div> |
| <div class="paragraph"> |
| <p>During Parameter Context creation, select the "Parameters" tab. Click the <code>+</code> button to open the Add Parameter window.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/add-parameter-during-parameter-context-creation.png" alt="Add Parameter"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>To add parameters to an existing Parameter Context, open the Parameter Context window and click the Edit button (<span class="image"><img src="images/iconEdit.png" alt="Edit"></span>) in the row of the desired Parameter Context.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/edit-parameter-context.png" alt="Edit Parameter Context"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>On the "Parameters" tab, click the <code>+</code> button to open the Add Parameter window.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The Add Parameter window has the following settings:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Name</strong> - A name that is used to denote the Parameter. Only alpha-numeric characters (a-z, A-Z, 0-9), hyphens ( - ), underscores ( _ ), periods ( . ), and spaces are allowed.</p> |
| </li> |
| <li> |
| <p><strong>Value</strong> - The value that will be used when the Parameter is referenced. If a Parameter makes use of the Expression Language, it is important to note that the Expression Language will be evaluated |
| in the context of the component that references the Parameter. Please see the <a href="user-guide.html#parameters-and-el">Parameters and Expression Language</a> section below for more information.</p> |
| </li> |
| <li> |
| <p><strong>Set empty string</strong> - Check to explicitly set the value of the Parameter to an empty string. Unchecked by default. (Note: If checked but a value is set, the checkbox is ignored.)</p> |
| </li> |
| <li> |
| <p><strong>Sensitive Value</strong> - Set to "Yes" if the Parameter’s Value should be considered sensitive. If sensitive, the value of the Parameter will not be shown in the UI once applied. The default setting is "No". Sensitive Parameters can only be referenced by sensitive properties and Non-Sensitive Parameters by non-sensitive properties. Once a Parameter is created, its sensitivity flag cannot be changed.</p> |
| </li> |
| <li> |
| <p><strong>Description</strong> - A description that explains what the Parameter is, how it is to be used, etc. This field is optional.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>Once these settings are configured, select "Apply". The Referencing Components lists the components referenced by the currently selected parameter. Add additional Parameters or edit any existing Parameters.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/update-parameter-context.png" alt="Update Parameter Context"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>To complete the process, select "Apply" from the Parameter Context window. The following operations are performed to validate all components that reference the added or modified parameters: Stopping/Restarting affected Processors, Disabling/Re-enabling affected Controller Services, Updating Parameter Context.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/parameters-validate-affected-components.png" alt="Validate Affected Components"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The Referencing Components section now lists an aggregation of all the components referenced by the set of parameters added/edited/deleted, organized by process group.</p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="parameters-and-el"><a class="anchor" href="user-guide.html#parameters-and-el"></a>Parameters and Expression Language</h4> |
| <div class="paragraph"> |
| <p>When adding a Parameter that makes use of the Expression Language, it is important to understand the context in which the Expression Language will be evaluated. The expression is always evaluated |
| in the context of the Processor or Controller Service that references the Parameter. Take, for example, a scenario where a Parameter with the name <code>Time</code> is added with a value of <code>${now()}</code>. The |
| Expression Language results in a call to determine the system time when it is evaluated. When added as a Parameter, the system time is not evaluated when the Parameter is added, but rather when a |
| Processor or Controller Service evaluates the Expression. That is, if a Processor has a Property whose value is set to <code>#{Time}</code> it will function in exactly the same manner as if the Property’s |
| value were set to <code>${now()}</code>. Each time that the property is referenced, it will produce a different timestamp.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Furthermore, some Properties do not allow for Expression Language, while others allow for Expression Language but do not evaluate expressions against FlowFile attributes. To help understand how |
| this works, consider a Parameter named <code>File</code> whose value is <code>${filename}</code>. Then consider three different properties, each with a different Expression Language Scope and a FlowFile whose filename |
| is <code>test.txt</code>. If each of those Properties is set to <code>#{File}</code>, then the follow table illustrates the resultant value.</p> |
| </div> |
| <table class="tableblock frame-all grid-all stretch"> |
| <colgroup> |
| <col style="width: 25%;"> |
| <col style="width: 25%;"> |
| <col style="width: 25%;"> |
| <col style="width: 25%;"> |
| </colgroup> |
| <thead> |
| <tr> |
| <th class="tableblock halign-left valign-top">Configured Property Value</th> |
| <th class="tableblock halign-left valign-top">Expression Language Scope</th> |
| <th class="tableblock halign-left valign-top">Effective Property Value</th> |
| <th class="tableblock halign-left valign-top">Notes</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">#{File}</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">FlowFile Attributes</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">test.txt</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">The filename is resolved by looking at the <code>filename</code> attribute.</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">#{File}</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Variable Registry Only</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><em>Empty String</em></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">FlowFile attributes are not in scope, and we assume there is no Variable in the Variable Registry named "filename"</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">#{File}</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">None</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">${filename}</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">The literal text "${filename}" will be unevaluated.</p></td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| <div class="sect3"> |
| <h4 id="assigning_parameter_context_to_PG"><a class="anchor" href="user-guide.html#assigning_parameter_context_to_PG"></a>Assigning a Parameter Context to a Process Group</h4> |
| <div class="paragraph"> |
| <p>For a component to reference a Parameter, its Process Group must first be assigned a Parameter Context. Once assigned, processors and controller services within that Process Group may only reference Parameters within that Parameter Context.</p> |
| </div> |
| <div class="paragraph"> |
| <p>A Process Group can only be assigned one Parameter Context, while a given Parameter Context can be assigned to multiple Process Groups.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| A user can only set the Parameter Context of a Process Group to one of the Parameter Contexts that the user has the view policy for. Additionally, in order to set the Parameter Context, the user must have the modify policy for the Process Group. See <a href="user-guide.html#accessing-parameters">Accessing Parameters</a> for more information. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="paragraph"> |
| <p>To assign a Parameter Context to a Process Group, click Configure, either from the Operate Palette or from the Process Group context menu.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/process-group-configuration-parameters.png" alt="Configure Process Group Parameter Context"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>In the Flow Configuration window, select the "General" tab. From the Process Group Parameter Context drop-down menu, select an existing Parameter Context or create a new one.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/process-group-parameter-context-menu.png" alt="Process Group Parameter Context Menu"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Select "Apply" to save the configuration changes. The Process Group context menu now includes a "Parameters" option which allows quick access to the Update Parameter Context window for the assigned Parameter Context.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/context-menu-parameters-option.png" alt="Context Menu Parameters Option"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>If the Parameter Context for a Process Group is changed, all components that reference any Parameters in that Process Group will be stopped, validated, and restarted assuming the components were previously running and are still valid.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| If a Parameter Context is unset from a Process Group, it does <strong>NOT</strong> inherit the Parameter Context from the parent Process Group. Instead, no Parameters can be referenced. Any component that does already reference a Parameter will become invalid. |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="referencing-parameters"><a class="anchor" href="user-guide.html#referencing-parameters"></a>Referencing Parameters</h4> |
| <div class="sect4"> |
| <h5 id="parameter-reference-syntax"><a class="anchor" href="user-guide.html#parameter-reference-syntax"></a>Parameter Reference Syntax</h5> |
| <div class="paragraph"> |
| <p>To configure an eligible property to reference a Parameter, use the <code>#</code> symbol as the start, with the Parameter’s name enclosed in curly braces:</p> |
| </div> |
| <div class="paragraph"> |
| <p><code>#{Parameter.Name}</code></p> |
| </div> |
| <div class="paragraph"> |
| <p>This can be escaped using an additional <code>#</code> character at the beginning. To illustrate this, assume that the Parameter <code>abc</code> has a value of <code>xxx</code> and Parameter <code>def</code> has a value of <code>yyy</code>. Then, the following user-defined property values will evaluate to these effective values:</p> |
| </div> |
| <table class="tableblock frame-all grid-all stretch"> |
| <colgroup> |
| <col style="width: 33.3333%;"> |
| <col style="width: 33.3333%;"> |
| <col style="width: 33.3334%;"> |
| </colgroup> |
| <tbody> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><strong>User-Entered Literal Property Value</strong></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><strong>Effective Property Value</strong></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><strong>Explanation</strong></p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>#{abc}</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>xxx</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Simple substitution</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>#{abc}/data</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>xxx/data</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Simple substitution with additional literal data</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>#{abc}/#{def}</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>xxx/yyy</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Multiple substitution with additional literal data</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>#{abc</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>#{abc</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">No { } for parameter replacement</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>#abc</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>#abc</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">No { } for parameter replacement</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>##{abc}</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>#{abc}</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Escaped # for literal interpretation</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>###{abc}</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>#xxx</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Escaped # for literal interpretation, followed by simple substitution</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>####{abc}</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>##{abc}</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Escaped # for literal interpretation, twice</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>#####{abc}</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>##xxx</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Escaped # for literal interpretation, twice, followed by simple substitution</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>#{abc/data}</code></p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Exception thrown on property set operation</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock"><code>/</code> not a valid parameter name character</p></td> |
| </tr> |
| </tbody> |
| </table> |
| <div class="paragraph"> |
| <p>When referencing a Parameter from within <a href="expression-language-guide.html">Expression Language</a>, the Parameter reference is evaluated first. As an example, to replace <code>xxx</code> with <code>zzz</code> for the <code>abc</code> Parameter:</p> |
| </div> |
| <div class="paragraph"> |
| <p><code>${ #{abc}:replace('xxx', 'zzz') }</code></p> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="referencing-and-creating-parameters-during-component-configuration"><a class="anchor" href="user-guide.html#referencing-and-creating-parameters-during-component-configuration"></a>Referencing and Creating Parameters During Component Configuration</h5> |
| <div class="paragraph"> |
| <p>Parameters can be easily referenced or created as you configure the components in your flow. For example, assume a process group has the Parameter Context "Kafka Settings" assigned to it. "Kafka Settings" contains the parameters <code>kafka.broker</code> and <code>kafka.topic1</code>.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/existing-parameters-example.png" alt="Existing Parameters"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>To reference <code>kafka.broker</code> as the value for the "Kafka Brokers" property in the PublishKafka processor, clear the default value and begin a new entry with the start delimiter <code>#{</code>. Next use the keystroke <code>control+space</code> to show the list of available parameters:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/autocomplete-parameter-example.png" alt="Autocomplete Parameter Example"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Select <code>kafka.broker</code> and complete the entry with a closing curly brace <code>}</code>.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/existing-parameter-selected.png" alt="Existing Parameter Selected"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Help text describing this process is displayed when hovering over the Expression Language and Parameters eligibility indicators.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/el-param-support-help-text.png" alt="Expression Language and Parameters Help Text"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Parameters can also be created on the fly. For example, to create a parameter for the "Topic Name" property, select the "Convert to Parameter" icon (<span class="image"><img src="images/iconConvertToParameter.png" alt="Convert to Parameter"></span>) in that property’s row. This icon will only be available if the user has appropriate permissions to modify the Parameter Context (see <a href="user-guide.html#accessing-parameters">Accessing Parameters</a> for more information).</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/convert-property-to-parameter.png" alt="Convert Property to Parameter"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The Add Parameter dialog will open. Configure the new parameter as desired.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/configure-parameter-on-the-fly.png" alt="Configure Parameter On the Fly"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Select "Apply". The process group’s Parameter Context will be updated and the new parameter will be referenced by the property with the proper syntax applied automatically.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/new-parameter-referenced.png" alt="New Parameter Referenced"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Properties values that are selectable can also reference parameters. In addition to applying the "Convert to Parameter" method described earlier, the option "Reference parameter.." is available in the value drop-down menu.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/reference-parameter-option.png" alt="Reference Parameter Option"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Selecting "Reference parameter…​" will display a drop-down list of available parameters, determined by the parameter context assigned to the component’s process group and the user’s access policies.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/reference-parameter-available-parameters.png" alt="Reference Parameter Available Parameters"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Hovering over the question mark icon (<span class="image"><img src="images/iconInfo.png" alt="Info"></span>) displays the parameter’s description.</p> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="using-parameters-with-sensitive-properties"><a class="anchor" href="user-guide.html#using-parameters-with-sensitive-properties"></a>Using Parameters with Sensitive Properties</h5> |
| <div class="paragraph"> |
| <p>Sensitive properties may only reference sensitive Parameters. This is important for <a href="user-guide.html#versioning_dataflow">versioned flows</a>. The value of the sensitive Parameter itself will NOT be sent to the flow registry, only the fact that the property references the sensitive Parameter. For more information see <a href="user-guide.html#parameters-in-versioned-flows">Parameters in Versioned Flows</a>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The value of a sensitive property must be set to a single Parameter reference. For example, values of <code>#{password}123</code> and <code>#{password}#{suffix}</code> are not allowed. Sending <code>#{password}123</code> would lead to exposing part of the sensitive property’s value. This is in contrast to a non-sensitive property, where a value such as <code>#{path}/child/file.txt</code> is valid.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="parameter-providers"><a class="anchor" href="user-guide.html#parameter-providers"></a>Parameter Providers</h4> |
| <div class="paragraph"> |
| <p>Parameter Providers allow parameters to be stored in sources external to NiFi (e.g. HashiCorp Vault). The parameters of a Parameter Provider can be fetched and applied to all referencing Parameter Contexts.</p> |
| </div> |
| <div class="paragraph"> |
| <p>To add a Parameter Provider, select Controller Settings from the Global Menu.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/controller-settings-selection.png" alt="Global Menu - Controller Settings"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>This displays the NiFi Settings window. Select the Parameter Providers tab and click the <code>+</code> button in the upper-right corner to create a new Parameter Provider.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/parameter-providers-tab.png" alt="Parameter Providers Tab"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The Add Parameter Provider window opens. This window is similar to the Add Processor window. It provides a list of the available Parameter Providers on the right and a tag cloud, showing the most common category tags used for Parameter Providers, on the left. The DFM may click any tag in the tag cloud in order to narrow down the list of Parameter Providers to those that fit the categories desired. The DFM may also use the Filter field at the top-right of the window to search for the desired Parameter Provider or use the Source drop-down at the top-left to filter the list by the group who created them. Upon selecting a Parameter Provider from the list, the DFM can see a description of the provider below. Select the desired parameter provider and click Add, or simply double-click the name of the provider to add it.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/add-parameter-provider-window.png" alt="Add Parameter Provider Window"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Once a Parameter Provider has been added, the DFM may configure it by clicking the "Edit" button in the far-right column. Other buttons in this column include "Fetch Parameters", "Remove" and "Access Policies".</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/parameter-provider-edit-buttons.png" alt="Parameter Providers Edit Buttons"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>You can obtain information about Parameter Providers by clicking the "View Details", "Usage", and "Alerts" buttons in the left-hand column.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/parameter-provider-tasks-info-buttons.png" alt="Parameter Providers Information Buttons"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>When the DFM clicks the "Edit" button, a Configure Parameter Provider window opens. It has three tabs: Settings, Properties, and Comments. This window is similar to the Configure Processor window. The Settings tab provides a place for the DFM to give the Parameter Provider a unique name (if desired). It also lists the UUID, Type, and Bundle information for the provider and displays a list of other components (e.g. parameter contexts) that reference the parameter provider. The DFM may hover the mouse over the question mark icons to see more information about each setting.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/configure-parameter-provider-settings.png" alt="Configure Parameter Provider Settings"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The Properties tab lists the various properties that may be configured for the parameter provider. The DFM may hover the mouse over the question mark icons to see more information about each property.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/configure-parameter-provider-properties.png" alt="Configure Parameter Provider Properties"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The Comments tab is just an open-text field, where the DFM may include comments about the provider. After configuring the Parameter Provider, click "Apply" to save the configuration and close the window, or click "Cancel" to discard the changes and close the window.</p> |
| </div> |
| <div class="paragraph"> |
| <p>When you want to fetch parameters from the Parameter Provider, click the "Fetch" button (<span class="image"><img src="images/iconFetch.png" alt="Fetch Button"></span>).</p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="accessing-parameters"><a class="anchor" href="user-guide.html#accessing-parameters"></a>Accessing Parameters</h4> |
| <div class="paragraph"> |
| <p>User privileges to Parameters are managed via access policies on the following levels:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Parameter Context</p> |
| </li> |
| <li> |
| <p>Process Group</p> |
| </li> |
| <li> |
| <p>Component</p> |
| </li> |
| </ul> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| For additional information on how to configure access policies, see the <a href="administration-guide.html#access-policies">Access Policies</a> section in the System Administrator’s Guide. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="sect4"> |
| <h5 id="parameter-context-access-policies"><a class="anchor" href="user-guide.html#parameter-context-access-policies"></a>Parameter Context Access Policies</h5> |
| <div class="paragraph"> |
| <p>For a user to see Parameter Contexts, they must be added to either the "access the controller" view policy or the "access parameter contexts" view policy. For a user to modify Parameter Contexts, they must also be added to the corresponding modify policies. These policies are accessed via "Policies" from the Global Menu. See the <a href="administration-guide.html#global-access-policies">Global Access Policies</a> section in the System Administrator’s Guide for more information.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| The "access parameter contexts" policies are inherited from the "access the controller" policies unless overridden. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="paragraph"> |
| <p>View and modify policies can also be set on individual parameter contexts to determine which users can view or add parameters to the context. Select "Parameter Contexts" from the Global Menu. Select the "Access Policies" button (<span class="image"><img src="images/iconAccessPolicies.png" alt="Access Policies"></span>) in the row of the desired parameter context to manage these policies.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/individual-parameter-context-polices.png" alt="Individual Parameter Context Policies"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>See the <a href="administration-guide.html#component-level-access-policies">Component Level Access Policies</a> section in the System Administrator’s Guide for more information.</p> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="process-group-access-policies"><a class="anchor" href="user-guide.html#process-group-access-policies"></a>Process Group Access Policies</h5> |
| <div class="paragraph"> |
| <p>A user can only set the Parameter Context of a Process Group to one of the Parameter Contexts that the user has the view policy for. Additionally, in order to set the Parameter Context, the user must have the modify policy for the Process Group. The Process Group access policies can be managed by highlighting the Process Group and selecting the "Access Policies" button (<span class="image"><img src="images/iconAccessPolicies.png" alt="Access Policies"></span>) from the Operate Palette.</p> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="component-access-policies"><a class="anchor" href="user-guide.html#component-access-policies"></a>Component Access Policies</h5> |
| <div class="paragraph"> |
| <p>To reference Parameters or convert properties to a Parameter in a component, a user needs to have the view and modify policies for the component. These policies are inherited if the user has view and modify policies to the component’s process group, but these policies can be overridden on the component level.</p> |
| </div> |
| <div class="paragraph"> |
| <p>In order to modify a Parameter, a user must have view and modify policies for any and all components that reference that Parameter. This is needed because changing the Parameter requires that the components be stopped/started and also because by taking that action, the user is modifying the behavior of the component.</p> |
| </div> |
| <div class="paragraph"> |
| <p>See the <a href="administration-guide.html#component-level-access-policies">Component Level Access Policies</a> section in the System Administrator’s Guide for more information.</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="Using_Custom_Properties"><a class="anchor" href="user-guide.html#Using_Custom_Properties"></a>Using Custom Properties with Expression Language</h3> |
| <div class="paragraph"> |
| <p>You can use NiFi Expression Language to reference FlowFile attributes, compare them to other values, and manipulate their values when you are creating and configuring dataflows. For more information on Expression Language, see the <a href="expression-language-guide.html">Expression Language Guide</a>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>In addition to using FlowFile attributes, system properties, and environment properties within Expression |
| Language, you can also define custom properties for Expression Language use. Defining custom properties |
| gives you more flexibility in handling and processing dataflows. You can also create custom properties |
| for connection, server, and service properties, for easier dataflow configuration.</p> |
| </div> |
| <div class="paragraph"> |
| <p>NiFi properties have resolution precedence of which you should be aware when creating custom properties:
</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Processor-specific attributes</p> |
| </li> |
| <li> |
| <p>FlowFile properties
</p> |
| </li> |
| <li> |
| <p>FlowFile attributes
</p> |
| </li> |
| <li> |
| <p>From Variable Registry:
</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>User defined properties (custom properties)</p> |
| </li> |
| <li> |
| <p>System properties
</p> |
| </li> |
| <li> |
| <p>Operating System environment variables</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>When you are creating custom properties, ensure that each custom property contains a distinct property value, so that it is not overridden by existing environment properties, system properties, or FlowFile attributes.</p> |
| </div> |
| <div class="paragraph"> |
| <p>There are two ways to use and manage custom properties with Expression Language:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Variables</strong>: Variables are created and configured within the NiFi UI. They can be used in any field that supports Expression Language. Variables cannot be used for sensitive properties. NiFi automatically picks up new or modified variables. Variables are defined at the Process Group level, as a result, the access policies for viewing and changing variables are derived from the access policies of the Process Group. See <a href="user-guide.html#Variables">Variables</a> for more information.</p> |
| </li> |
| <li> |
| <p><strong>Custom Properties File</strong>: Key/value pairs are defined in a custom properties file that is referenced via the <code>nifi.variable.registry.properties</code> in <em>nifi.properties</em>. NiFi must be restarted for updates to be picked up. See <a href="user-guide.html#Custom_Properties">Referencing Custom Properties via nifi.properties</a> for more information.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="admonitionblock warning"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-warning" title="Warning"></i> |
| </td> |
| <td class="content"> |
| Custom properties via Variables and the <em>nifi.properties</em> file are still supported for compatibility purposes but do not have the same power as Parameters such as support for sensitive properties and more granular control over who can create, modify or use them. Variables and the <code>nifi.variable.registry.properties</code> property will be removed in a future release. As a result, it is highly recommended to switch to Parameters. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="paragraph"> |
| <p>Expression Language support for a property is indicated in the UI.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/el-support-help-text.png" alt="Expression Language Help Text"> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="Variables"><a class="anchor" href="user-guide.html#Variables"></a>Variables</h4> |
| <div class="admonitionblock warning"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-warning" title="Warning"></i> |
| </td> |
| <td class="content"> |
| Custom properties via Variables and the <em>nifi.properties</em> file are still supported for compatibility purposes but do not have the same power as Parameters such as support for sensitive properties and more granular control over who can create, modify or use them. Variables and the <code>nifi.variable.registry.properties</code> property will be removed in a future release. As a result, it is highly recommended to switch to Parameters. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="paragraph"> |
| <p>Variables are created and configured within the NiFi UI. They can be used in any field that supports Expression Language. Variables cannot be used for sensitive properties. Variables are defined at the Process Group level, as a result, the access policies for viewing and changing variables are derived from the access policies of the Process Group. Variable values cannot reference other variables or make use of Expression Language.</p> |
| </div> |
| <div class="sect4"> |
| <h5 id="variables-window"><a class="anchor" href="user-guide.html#variables-window"></a>Variables Window</h5> |
| <div class="paragraph"> |
| <p>To access the Variables window, right-click on the canvas with nothing selected:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/variables-context_menu-rpg.png" alt="Variables in Context Menu for RPG"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Select "Variables" from the Context Menu:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/variables_window_empty.png" alt="Empty Variables Window"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>"Variables" is also available in the right-click Context Menu when a process group is selected:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/variables-context_menu-pg.png" alt="Variables in Context Menu for PG"> |
| </div> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="creating-a-variable"><a class="anchor" href="user-guide.html#creating-a-variable"></a>Creating a Variable</h5> |
| <div class="paragraph"> |
| <p>In the Variables window, click the <code>+</code> button to create a new variable. Add a name:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/variable-name.png" alt="Variable Name Creation"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>and a value:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/variable-value.png" alt="Variable Value Creation"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Select "Apply":</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/new_variable-apply.png" alt="New Variable Applied"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Steps to update the variable are performed (Identifying components affected, Stopping affected Processors, etc.). For example, the Referencing Processors section now lists the "PutFile-Root" processor. Selecting the name of the processor in the list will navigate to that processor on the canvas. Looking at the properties of the processor, <code>${putfile_dir}</code> is referenced by the Directory property:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/variable-putfile-property.png" alt="Processor Property Using Variable"> |
| </div> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="variable-scope"><a class="anchor" href="user-guide.html#variable-scope"></a>Variable Scope</h5> |
| <div class="paragraph"> |
| <p>Variables are scoped by the Process Group they are defined in and are available to any Processor defined at that level and below (i.e., any descendant Processors).</p> |
| </div> |
| <div class="paragraph"> |
| <p>Variables in a descendant group override the value in a parent group. More specifically, if a variable <code>x</code> is declared at the root group and also declared inside a process group, components inside the process group will use the value of <code>x</code> defined in the process group.</p> |
| </div> |
| <div class="paragraph"> |
| <p>For example, in addition to the <code>putfile_dir</code> variable that exists at the root process group, assume another <code>putfile_dir</code> variable was created within Process Group A. If one of the components within Process Group A references <code>putfile_dir</code>, both variables will be listed, but the <code>putfile_dir</code> from the root group will have a strikethrough indicating that is is being overridden:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/variable-overridden.png" alt="Variable Overridden"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>A variable can only be modified for the process group it was created in, which is listed at the top of the Variables window. To modify a variable defined in a different process group, select the "arrow" icon in that variable’s row:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/variable_window-goto.png" alt="Variable Go To"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>which will navigate to the Variables window for that process group:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/variable_window-rpg.png" alt="Variables Window for RPG"> |
| </div> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="variable-permissions"><a class="anchor" href="user-guide.html#variable-permissions"></a>Variable Permissions</h5> |
| <div class="paragraph"> |
| <p>Variable permissions are based solely on the privileges configured on the corresponding Process Group.</p> |
| </div> |
| <div class="paragraph"> |
| <p>For example, if a user does not have access to View a process group, the Variables window can not be viewed for that process group:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/variable_insufficient-permissions.png" alt="Insufficient Permissions to View Variables"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>If a user has access to View a process group but does not have access to Modify the process group, the variables can be viewed but not modified.</p> |
| </div> |
| <div class="paragraph"> |
| <p>For information on how to manage privileges on components, see the <a href="administration-guide.html#access-policies">Access Policies</a> section in the System Administrator’s Guide.</p> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="referencing-controller-services"><a class="anchor" href="user-guide.html#referencing-controller-services"></a>Referencing Controller Services</h5> |
| <div class="paragraph"> |
| <p>In addition to Referencing Processors, the Variables window also displays Referencing Controller Services:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/variables-window_controller-services.png" alt="Referencing Controller Services"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Selecting the name of the controller service will navigate to that controller service in the Configuration window:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/variable_nav-controller_services.png" alt="Controller Service Using Variable"> |
| </div> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="unauthorized-referencing-components"><a class="anchor" href="user-guide.html#unauthorized-referencing-components"></a>Unauthorized Referencing Components</h5> |
| <div class="paragraph"> |
| <p>When View or Modify privileges are not given to a component that references a variable, the UUID of the component will be displayed in the Variables window:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/variables-window_unauthorized.png" alt="Unauthorized Referencing Components"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>In the above example, the variable <code>property1</code> is referenced by a processor that "user1" is not able to view:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/variable-unauthorized-ref-processor-canvas.png" alt="Unauthorized Referencing Processor"> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="Custom_Properties"><a class="anchor" href="user-guide.html#Custom_Properties"></a>Referencing Custom Properties via nifi.properties</h4> |
| <div class="admonitionblock warning"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-warning" title="Warning"></i> |
| </td> |
| <td class="content"> |
| Custom properties via Variables and the <em>nifi.properties</em> file are still supported for compatibility purposes but do not have the same power as Parameters such as support for sensitive properties and more granular control over who can create, modify or use them. Variables and the <code>nifi.variable.registry.properties</code> property will be removed in a future release. As a result, it is highly recommended to switch to Parameters. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="paragraph"> |
| <p>Identify one or more sets of key/value pairs, and give them to your system administrator.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Once the new custom properties have been added, ensure that the <code>nifi.variable.registry.properties</code> |
| field in the <em>nifi.properties</em> file is updated with the custom properties location.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| NiFi must be restarted for these updates to be picked up. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="paragraph"> |
| <p>For more information, see the <a href="administration-guide.html#custom_properties">Custom Properties</a> section in the System Administrator’s Guide.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="Controller_Services"><a class="anchor" href="user-guide.html#Controller_Services"></a>Controller Services</h3> |
| <div class="paragraph"> |
| <p>Controller Services are shared services that can be used by reporting tasks, processors, and other services to utilize for configuration or task execution.</p> |
| </div> |
| <div class="admonitionblock important"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-important" title="Important"></i> |
| </td> |
| <td class="content"> |
| Controller Services defined on the controller level are limited to reporting tasks and other services defined there. Controller Services for use by processors in your dataflow must be defined in the configuration of the root process group or sub-process group(s) where they will be used. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| If your NiFi instance is secured, your ability to view and add Controller Services is dependent on the privileges assigned to you. If you do not have access to one or more Controller Services, you are not able to see or access it in the UI. Access privileges can be assigned on a global or Controller Service-specific basis (see <a href="user-guide.html#UI-with-multi-tenant-authorization">Accessing the UI with Multi-Tenant Authorization</a> for more information). |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="sect3"> |
| <h4 id="Management_Controller_Services"><a class="anchor" href="user-guide.html#Management_Controller_Services"></a>Adding Management Controller Services</h4> |
| <div class="paragraph"> |
| <p>To add a Management Controller Service, select Controller Settings from the Global Menu.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/controller-settings-selection.png" alt="Global Menu - Controller Settings"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>This displays the NiFi Settings window. The window has five tabs: General, Management Controller Services, Reporting Tasks, Registry Clients and Parameter Providers. The General tab provides settings for the overall maximum thread counts of the instance.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/settings-general-tab.png" alt="Controller Settings General Tab"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>To the right of the General tab is the Management Controller Services tab. From this tab, the DFM may click the <code>+</code> button in the upper-right corner to create a new Controller Service.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/controller-services-tab.png" alt="Controller Services Tab"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The Add Controller Service window opens. This window is similar to the Add Processor window. It provides a list of the available Controller Services on the right and a tag cloud, showing the most common category tags used for Controller Services, on the left. The DFM may click any tag in the tag cloud in order to narrow down the list of Controller Services to those that fit the categories desired. The DFM may also use the Filter field at the top-right of the window to search for the desired Controller Service or use the Source drop-down at the top-left to filter the list by the group who created them. Upon selecting a Controller Service from the list, the DFM can see a description of the service below. Select the desired controller service and click Add, or simply double-click the name of the service to add it.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/add-controller-service-window.png" alt="Add Controller Service Window"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Once you have added a Controller Service, you can configure it by clicking the "Configure" button in the |
| far-right column. Other buttons in this column include "Enable", "Remove" and "Access Policies".</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/controller-services-configure-buttons.png" alt="Controller Services Buttons"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>You can obtain information about Controller Services by clicking the "Usage" and "Alerts" buttons in the left-hand column.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/controller-services-info-buttons.png" alt="Controller Services Information Buttons"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>When the DFM clicks the "Configure" button, a Configure Controller Service window opens. It has three tabs: Settings, Properties,and Comments. This window is similar to the Configure Processor window.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The Settings tab provides a place for the DFM to give the Controller Service a unique name (if desired). It also lists the UUID, Type, Bundle and Support information for the service and provides a list of other components (reporting tasks or other controller services) that reference the service.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Finally, the Bulletin level is able to be modified. Whenever the Controller Service writes to its log, the Controller Service will also generate a Bulletin. This setting indicates the lowest level of Bulletin that should be shown in the User Interface. By default, the Bulletin level is set to WARN, which means it will display all warning and error-level bulletins.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/configure-controller-service-settings.png" alt="Configure Controller Service Settings"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The Properties tab lists the various properties that apply to the particular controller service. As with configuring processors, the DFM may hover over the question mark icons to see more information about each property.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/configure-controller-service-properties.png" alt="Configure Controller Service Properties"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The Comments tab is just an open-text field, where the DFM may include comments about the service. After configuring a Controller Service, click "Apply" to save the configuration and close the window, or click "Cancel" to discard the changes and close the window.</p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="Controller_Services_for_Dataflows"><a class="anchor" href="user-guide.html#Controller_Services_for_Dataflows"></a>Adding Controller Services for Dataflows</h4> |
| <div class="paragraph"> |
| <p>To add a Controller Service for a dataflow, you can either right click a Process Group and select Configure, or click Configure from the Operate Palette.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/process-group-configuration-options.png" alt="Process Group Configuration Options"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>When you click Configure from the Operate Palette with nothing selected on your canvas, you add a Controller Service for your Root Process Group. That Controller Service is then available to all nested Process Groups in your dataflow. When you select a Process Group on the canvas and then click Configure from either the Operate Palette or the Process Group context menu, the service will be available to all Processors and Controller Services defined in that Process Group and below.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/process-group-controller-services-scope.png" alt="Process Group Controller Services Scope"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Use the following steps to add a Controller Service:</p> |
| </div> |
| <div class="olist arabic"> |
| <ol class="arabic"> |
| <li> |
| <p>Click Configure, either from the Operate Palette, or from the Process Group context menu. This displays the process group Configuration window. The window has two tabs: General and Controller Services. The <a href="user-guide.html#General_tab_ProcessGroup">General Tab</a> is for settings that pertain to general information about the process group.</p> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/process-group-configuration-window.png" alt="Process Group Configuration Window"> |
| </div> |
| </div> |
| </li> |
| <li> |
| <p>From the Process Group Configuration page, select the Controller Services tab.</p> |
| </li> |
| <li> |
| <p>Click the <code>+</code> button to display the Add Controller Service dialog.</p> |
| </li> |
| <li> |
| <p>Select the Controller Service desired, and click Add.</p> |
| </li> |
| <li> |
| <p>Perform any necessary Controller Service configuration tasks by clicking the Configure icon (<span class="image"><img src="images/iconConfigure.png" alt="Configure"></span>) in the right-hand column.</p> |
| </li> |
| </ol> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="Enabling_Disabling_Controller_Services"><a class="anchor" href="user-guide.html#Enabling_Disabling_Controller_Services"></a>Enabling/Disabling Controller Services</h4> |
| <div class="paragraph"> |
| <p>After a Controller Service has been configured, it must be enabled in order to run. Do this using the "Enable" button (<span class="image"><img src="images/iconEnable.png" alt="Enable Button"></span>) in the far-right column of the Controller Services tab. In order to modify an existing/running controller service, the DFM needs to stop/disable it (as well as all referencing reporting tasks and controller services). Do this using the "Disable" button (<span class="image"><img src="images/iconDisable.png" alt="Disable Button"></span>). Rather than having to hunt down each component that is referenced by that controller service, the DFM has the ability to stop/disable them when disabling the controller service in question. When enabling a controller service, the DFM has the option to either start/enable the controller service and all referencing components or start/enable only the controller service itself.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/enable-controller-service-scope.png" alt="Enable Controller Service Scope"></span></p> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="Reporting_Tasks"><a class="anchor" href="user-guide.html#Reporting_Tasks"></a>Reporting Tasks</h3> |
| <div class="paragraph"> |
| <p>Reporting Tasks run in the background to provide statistical reports about what is happening in the NiFi instance. The DFM adds and configures Reporting Tasks similar to the process for Controller Services. To add a Reporting Task, select Controller Settings from the Global Menu.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/controller-settings-selection.png" alt="Global Menu - Controller Settings"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>This displays the NiFi Settings window. Select the Reporting Tasks tab and click the <code>+</code> button in the upper-right corner to create a new Reporting Task.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/reporting-tasks-tab.png" alt="Reporting Tasks Tab"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The Add Reporting Task window opens. This window is similar to the Add Processor window. It provides a list of the available Reporting Tasks on the right and a tag cloud, showing the most common category tags used for Reporting Tasks, on the left. The DFM may click any tag in the tag cloud in order to narrow down the list of Reporting Tasks to those that fit the categories desired. The DFM may also use the Filter field at the top-right of the window to search for the desired Reporting Task or use the Source drop-down at the top-left to filter the list by the group who created them. Upon selecting a Reporting Task from the list, the DFM can see a description of the task below. Select the desired reporting task and click Add, or simply double-click the name of the service to add it.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/add-reporting-task-window.png" alt="Add Reporting Task Window"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Once a Reporting Task has been added, the DFM may configure it by clicking the "Edit" button in the far-right column. Other buttons in this column include "Start", "Remove", "State" and "Access Policies".</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/reporting-tasks-edit-buttons.png" alt="Reporting Tasks Edit Buttons"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>You can obtain information about Reporting Tasks by clicking the "View Details", "Usage", and "Alerts" buttons in the left-hand column.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/reporting-tasks-info-buttons.png" alt="Reporting Tasks Information Buttons"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>When the DFM clicks the "Edit" button, a Configure Reporting Task window opens. It has three tabs: Settings, Properties, and Comments. This window is similar to the Configure Processor window. The Settings tab provides a place for the DFM to give the Reporting Task a unique name (if desired). It also lists the UUID, Type, and Bundle information for the task and provides settings for the task’s Scheduling Strategy and Run Schedule (similar to the same settings in a processor). The DFM may hover the mouse over the question mark icons to see more information about each setting.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/configure-reporting-task-settings.png" alt="Configure Reporting Task Settings"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The Properties tab lists the various properties that may be configured for the task. The DFM may hover the mouse over the question mark icons to see more information about each property.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/configure-reporting-task-properties.png" alt="Configure Reporting Task Properties"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The Comments tab is just an open-text field, where the DFM may include comments about the task. After configuring the Reporting Task, click "Apply" to save the configuration and close the window, or click "Cancel" to discard the changes and close the window.</p> |
| </div> |
| <div class="paragraph"> |
| <p>When you want to run the Reporting Task, click the "Start" button (<span class="image"><img src="images/iconStart.png" alt="Start Button"></span>).</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="Connecting_Components"><a class="anchor" href="user-guide.html#Connecting_Components"></a>Connecting Components</h3> |
| <div class="paragraph"> |
| <p>Once processors and other components have been added to the canvas and configured, the next step is to connect them |
| to one another so that NiFi knows what to do with each FlowFile after it has been processed. This is accomplished by creating a |
| Connection between each component. When the user hovers the mouse over the center of a component, a new Connection icon ( |
| <span class="image"><img src="images/addConnect.png" alt="Connection Bubble"></span> |
| ) appears:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/processor-connection-bubble.png" alt="Processor with Connection Bubble"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The user drags the Connection bubble from one component to another until the second component is highlighted. When the user |
| releases the mouse, a 'Create Connection' dialog appears. This dialog consists of two tabs: 'Details' and 'Settings'. They are |
| discussed in detail below. Note that it is possible to draw a connection so that it loops back on the same processor. This can be |
| useful if the DFM wants the processor to try to re-process FlowFiles if they go down a failure Relationship. To create this type of looping |
| connection, simply drag the connection bubble away and then back to the same processor until it is highlighted. Then release the mouse |
| and the same 'Create Connection' dialog appears.</p> |
| </div> |
| <div class="sect3"> |
| <h4 id="details-tab"><a class="anchor" href="user-guide.html#details-tab"></a>Details Tab</h4> |
| <div class="paragraph"> |
| <p>The Details tab of the 'Create Connection' dialog provides information about the source and destination components, including the component name, the |
| component type, and the Process Group in which the component lives:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/create-connection.png" alt="Create Connection"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Additionally, this tab provides the ability to choose which Relationships should be included in this Connection. At least one |
| Relationship must be selected. If only one Relationship is available, it is automatically selected.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| If multiple Connections are added with the same Relationship, any FlowFile that is routed to that Relationship will |
| automatically be 'cloned', and a copy will be sent to each of those Connections. |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="settings"><a class="anchor" href="user-guide.html#settings"></a>Settings</h4> |
| <div class="paragraph"> |
| <p>The Settings tab provides the ability to configure the Connection’s Name, FlowFile Expiration, Back Pressure Thresholds, Load Balance Strategy and Prioritization:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/connection-settings.png" alt="Connection Settings"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The Connection name is optional. If not specified, the name shown for the Connection will be names of the Relationships that are active for the Connection.</p> |
| </div> |
| <div class="sect4"> |
| <h5 id="Flowfile_Expiration"><a class="anchor" href="user-guide.html#Flowfile_Expiration"></a>FlowFile Expiration</h5> |
| <div class="paragraph"> |
| <p>FlowFile expiration is a concept by which data that cannot be processed in a timely fashion can be automatically removed from the flow. |
| This is useful, for example, when the volume of data is expected to exceed the volume that can be sent to a remote site. |
| In this case, the expiration can be used in conjunction with Prioritizers to ensure that the highest priority data is |
| processed first and then anything that cannot be processed within a certain time period (one hour, for example) can be dropped. The expiration period is based on the time that the data entered the NiFi instance. In other words, if the file expiration on a given connection is set to '1 hour', and a file that has been in the NiFi instance for one hour reaches that connection, it will expire. The default |
| value of <code>0 sec</code> indicates that the data will never expire. When a file expiration other than '0 sec' is set, a small clock icon appears on the connection label, so the DFM can see it at-a-glance when looking at a flow on the canvas.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/file_expiration_clock.png" alt="File Expiration Indicator"></span></p> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="Backpressure"><a class="anchor" href="user-guide.html#Backpressure"></a>Back Pressure</h5> |
| <div class="paragraph"> |
| <p>NiFi provides two configuration elements for Back Pressure. These thresholds indicate how much data should be |
| allowed to exist in the queue before the component that is the source of the Connection is no longer scheduled to run. |
| This allows the system to avoid being overrun with data. The first option provided is the "Back pressure object threshold." |
| This is the number of FlowFiles that can be in the queue before back pressure is applied. The second configuration option |
| is the "Back pressure data size threshold." This specifies the maximum amount of data (in size) that should be queued up before |
| applying back pressure. This value is configured by entering a number followed by a data size (<code>B</code> for bytes, <code>KB</code> for |
| kilobytes, <code>MB</code> for megabytes, <code>GB</code> for gigabytes, or <code>TB</code> for terabytes).</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| By default each new connection added will have a default Back Pressure Object Threshold of <code>10,000 objects</code> and Back Pressure Data Size Threshold of <code>1 GB</code>. |
| These defaults can be changed by modifying the appropriate properties in the <em>nifi.properties</em> file. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="paragraph"> |
| <p>When back pressure is enabled, small progress bars appear on the connection label, so the DFM can see it at-a-glance when looking at a flow on the canvas. The progress bars change color based on the queue percentage: Green (0-60%), Yellow (61-85%) and Red (86-100%).</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/back_pressure_indicators.png" alt="Back Pressure Indicator Bars"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Hovering your mouse over a bar displays the exact percentage.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/back_pressure_indicator_hover.png" alt="Back Pressure Indicator Hover Text"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>When the queue is completely full, the Connection is highlighted in red.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/back_pressure_full.png" alt="Back Pressure Queue Full"></span></p> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="Load_Balancing"><a class="anchor" href="user-guide.html#Load_Balancing"></a>Load Balancing</h5> |
| <div class="sect5"> |
| <h6 id="load_balance_strategy"><a class="anchor" href="user-guide.html#load_balance_strategy"></a>Load Balance Strategy</h6> |
| <div class="paragraph"> |
| <p>To distribute the data in a flow across the nodes in the cluster, NiFi offers the following load balance strategies:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Do not load balance</strong>: Do not load balance FlowFiles between nodes in the cluster. This is the default.</p> |
| </li> |
| <li> |
| <p><strong>Partition by attribute</strong>: Determines which node to send a given FlowFile to based on the value of a user-specified FlowFile Attribute. All FlowFiles that have the same value for the Attribute will be sent to the same node in the cluster. If the destination node is disconnected from the cluster or if unable to communicate, the data does not fail over to another node. The data will queue, waiting for the node to be available again. Additionally, if a node joins or leaves the cluster necessitating a rebalance of the data, consistent hashing is applied to avoid having to redistribute all of the data.</p> |
| </li> |
| <li> |
| <p><strong>Round robin</strong>: FlowFiles will be distributed to nodes in the cluster in a round-robin fashion. If a node is disconnected from the cluster or if unable to communicate with a node, the data that is |
| queued for that node will be automatically redistributed to another node(s). If a node is not able to receive the data as fast other nodes in the cluster, the node may also be skipped for one or |
| more iterations in order to maximize throughput of data distribution across the cluster.</p> |
| </li> |
| <li> |
| <p><strong>Single node</strong>: All FlowFiles will be sent to a single node in the cluster. Which node they are sent to is not configurable. If the node is disconnected from the cluster or if unable to communicate with the node, the data that is queued for that node will remain queued until the node is available again.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| In addition to the UI settings, there are <a href="administration-guide.html#cluster_node_properties">Cluster Node Properties</a> related to load balancing that must also be configured in <em>nifi.properties</em>. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| NiFi persists the nodes that are in a cluster across restarts. This prevents the redistribution of data until all of the nodes have connected. If the cluster is shutdown and a node is not intended to be brought back up, the user is responsible for removing the node from the cluster via the "Cluster" dialog in the UI (see <a href="administration-guide.html#managing_nodes">Managing Nodes</a> for more information). |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <div class="sect5"> |
| <h6 id="load-balance-compression"><a class="anchor" href="user-guide.html#load-balance-compression"></a>Load Balance Compression</h6> |
| <div class="paragraph"> |
| <p>After selecting the load balance strategy, the user can configure whether or not data should be compressed when being transferred between nodes in the cluster.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/load_balance_compression_options.png" alt="Load Balance Compression Options"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The following compression options are available:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Do not compress</strong>: FlowFiles will not be compressed. This is the default.</p> |
| </li> |
| <li> |
| <p><strong>Compress attributes only</strong>: FlowFile attributes will be compressed, but FlowFile contents will not.</p> |
| </li> |
| <li> |
| <p><strong>Compress attributes and content</strong>: FlowFile attributes and contents will be compressed.</p> |
| </li> |
| </ul> |
| </div> |
| </div> |
| <div class="sect5"> |
| <h6 id="load-balance-indicator"><a class="anchor" href="user-guide.html#load-balance-indicator"></a>Load Balance Indicator</h6> |
| <div class="paragraph"> |
| <p>When a load balance strategy has been implemented for a connection, a load balance indicator (<span class="image"><img src="images/iconLoadBalance.png" alt="Load Balance Icon"></span>) will appear on the connection:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/load_balance_configured_connection.png" alt="Connection Configured with Load Balance Strategy"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Hovering over the icon will display the connection’s load balance strategy and compression configuration. The icon in this state also indicates that all data in the connection has been distributed across the cluster.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/load_balance_distributed_connection.png" alt="Distributed Load Balance Connection"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>When data is actively being transferred between the nodes in the cluster, the load balance indicator will change orientation and color:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/load_balance_active_connection.png" alt="Active Load Balance Connection"></span></p> |
| </div> |
| </div> |
| <div class="sect5"> |
| <h6 id="cluster-connection-summary"><a class="anchor" href="user-guide.html#cluster-connection-summary"></a>Cluster Connection Summary</h6> |
| <div class="paragraph"> |
| <p>To see where data has been distributed among the cluster nodes, select Summary from the Global Menu. Then select the "Connections" tab and the "View Connection Details" icon for a source:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/summary_connections.png" alt="NiFi Summary Connections"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>This will open the Cluster Connection Summary dialog, which shows the data on each node in the cluster:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/cluster_connection_summary.png" alt="Cluster Connection Summary Dialog"></span></p> |
| </div> |
| </div> |
| </div> |
| <div class="sect4"> |
| <h5 id="prioritization"><a class="anchor" href="user-guide.html#prioritization"></a>Prioritization</h5> |
| <div class="paragraph"> |
| <p>The right-hand side of the tab provides the ability to prioritize the data in the queue so that higher priority data is |
| processed first. Prioritizers can be dragged from the top ('Available prioritizers') to the bottom ('Selected prioritizers'). |
| Multiple prioritizers can be selected. The prioritizer that is at the top of the 'Selected prioritizers' list is the highest |
| priority. If two FlowFiles have the same value according to this prioritizer, the second prioritizer will determine which |
| FlowFile to process first, and so on. If a prioritizer is no longer desired, it can then be dragged from the 'Selected |
| prioritizers' list to the 'Available prioritizers' list.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The following prioritizers are available:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>FirstInFirstOutPrioritizer</strong>: Given two FlowFiles, the one that reached the connection first will be processed first.</p> |
| </li> |
| <li> |
| <p><strong>NewestFlowFileFirstPrioritizer</strong>: Given two FlowFiles, the one that is newest in the dataflow will be processed first.</p> |
| </li> |
| <li> |
| <p><strong>OldestFlowFileFirstPrioritizer</strong>: Given two FlowFiles, the one that is oldest in the dataflow will be processed first. 'This is the default scheme that is used if no prioritizers are selected'.</p> |
| </li> |
| <li> |
| <p><strong>PriorityAttributePrioritizer</strong>: Given two FlowFiles, an attribute called “priority” will be extracted. The one that has the lowest priority value will be processed first.</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Note that an UpdateAttribute processor should be used to add the "priority" attribute to the FlowFiles before they reach a connection that has this prioritizer set.</p> |
| </li> |
| <li> |
| <p>If only one has that attribute it will go first.</p> |
| </li> |
| <li> |
| <p>Values for the "priority" attribute can be alphanumeric, where "a" will come before "z" and "1" before "9"</p> |
| </li> |
| <li> |
| <p>If "priority" attribute cannot be parsed as a long, unicode string ordering will be used. For example: "99" and "100" will be ordered so the FlowFile with "99" comes first, but "A-99" and "A-100" will sort so the FlowFile with "A-100" comes first.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| </ul> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| With a <a href="user-guide.html#load_balance_strategy">Load Balance Strategy</a> configured, the connection has a queue per node in addition to the local queue. The prioritizer will sort the data in each queue independently. |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="changing-configuration-and-context-menu-options"><a class="anchor" href="user-guide.html#changing-configuration-and-context-menu-options"></a>Changing Configuration and Context Menu Options</h4> |
| <div class="paragraph"> |
| <p>After a connection has been drawn between two components, the connection’s configuration may be changed, and the connection may be moved to a new destination; however, the processors on either side of the connection must be stopped before a configuration or destination change may be made.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/nifi-connection.png" alt="Connection"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>To change a connection’s configuration or interact with the connection in other ways, right-click on the connection to open the connection context menu.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/nifi-connection-menu.png" alt="Connection Menu"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The following options are available:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Configure</strong>: This option allows the user to change the configuration of the connection.</p> |
| </li> |
| <li> |
| <p><strong>View status history</strong>: This option opens a graphical representation of the connection’s statistical information over time.</p> |
| </li> |
| <li> |
| <p><strong>List queue</strong>: This option lists the queue of FlowFiles that may be waiting to be processed.</p> |
| </li> |
| <li> |
| <p><strong>Go to source</strong>: This option can be useful if there is a long distance between the connection’s source and destination components on the canvas. By clicking this option, the view of the canvas will jump to the source of the connection.</p> |
| </li> |
| <li> |
| <p><strong>Go to destination</strong>: Similar to the "Go to source" option, this option changes the view to the destination component on the canvas and can be useful if there is a long distance between two connected components.</p> |
| </li> |
| <li> |
| <p><strong>Bring to front</strong>: This option brings the connection to the front of the canvas if something else (such as another connection) is overlapping it.</p> |
| </li> |
| <li> |
| <p><strong>Empty queue</strong>: This option allows the DFM to clear the queue of FlowFiles that may be waiting to be processed. This option can be especially useful during testing, when the DFM is not concerned about deleting data from the queue. When this option is selected, users must confirm that they want to delete the data in the queue.</p> |
| </li> |
| <li> |
| <p><strong>Delete</strong>: This option allows the DFM to delete a connection between two components. Note that the components on both sides of the connection must be stopped and the connection must be empty before it can be deleted.</p> |
| </li> |
| </ul> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="bending-connections"><a class="anchor" href="user-guide.html#bending-connections"></a>Bending Connections</h4> |
| <div class="paragraph"> |
| <p>To add a bend point (or elbow) to an existing connection, simply double-click on the connection in the spot where you want the bend point to be. Then, you can use the mouse to grab the bend point and drag it so that the connection is bent in the desired way. You can add as many bend points as you want. You can also use the mouse to drag and move the label on the connection to any existing bend point. To remove a bend point, simply double-click it again.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/nifi-connection-bend-points.png" alt="Connection Bend Points"></span></p> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="processor-validation"><a class="anchor" href="user-guide.html#processor-validation"></a>Processor Validation</h3> |
| <div class="paragraph"> |
| <p>Before trying to start a Processor, it’s important to make sure that the Processor’s configuration is valid. |
| A status indicator is shown in the top-left of the Processor. If the Processor is invalid, the indicator |
| will show a yellow Warning indicator with an exclamation mark indicating that there is a problem:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/invalid-processor.png" alt="Invalid Processor"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>In this case, hovering over the indicator icon with the mouse will provide a tooltip showing all of the validation |
| errors for the Processor. Once all of the validation errors have been addressed, the status indicator will change |
| to a Stop icon, indicating that the Processor is valid and ready to be started but currently is not running:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/valid-processor.png" alt="Valid Processor"> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="site-to-site"><a class="anchor" href="user-guide.html#site-to-site"></a>Site-to-Site</h3> |
| <div class="paragraph"> |
| <p>When sending data from one instance of NiFi to another, there are many different protocols that can be used. The preferred |
| protocol, though, is the NiFi Site-to-Site Protocol. Site-to-Site makes it easy to securely and efficiently transfer data to/from nodes in |
| one NiFi instance or data producing application to nodes in another NiFi instance or other consuming application.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Using Site-to-Site provides the following benefits:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Easy to configure</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>After entering the URL(s) of the remote NiFi instance/cluster, the available ports (endpoints) are automatically discovered and provided in a drop-down list.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p>Secure</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Site-to-Site optionally makes use of Certificates in order to encrypt data and provide authentication and authorization. Each port can be configured |
| to allow only specific users, and only those users will be able to see that the port even exists. For information on configuring the Certificates, |
| see the |
| <a href="administration-guide.html#security-configuration">Security Configuration</a> section of the |
| <a href="administration-guide.html">System Administrator’s Guide</a>.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p>Scalable</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>As nodes in the remote cluster change, those changes are automatically detected and data is scaled out across all nodes in the cluster.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p>Efficient</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Site-to-Site allows batches of FlowFiles to be sent at once in order to avoid the overhead of establishing connections and making multiple |
| round-trip requests between peers.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p>Reliable</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Checksums are automatically produced by both the sender and receiver and compared after the data has been transmitted, in order |
| to ensure that no corruption has occurred. If the checksums don’t match, the transaction will simply be canceled and tried again.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p>Automatically load balanced</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>As nodes come online or drop out of the remote cluster, or a node’s load becomes heavier or lighter, the amount of data that is directed |
| to that node will automatically be adjusted.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p>FlowFiles maintain attributes</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>When a FlowFile is transferred over this protocol, all of the FlowFile’s attributes |
| are automatically transferred with it. This can be very advantageous in many situations, as all of the context and enrichment |
| that has been determined by one instance of NiFi travels with the data, making for easy routing of the data and allowing users |
| to easily inspect the data.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p>Adaptable</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>As new technologies and ideas emerge, the protocol for handling Site-to-Site communications are able to change with them. When a connection is |
| made to a remote NiFi instance, a handshake is performed in order to negotiate which protocol and which version of the protocol will be used. |
| This allows new capabilities to be added while still maintaining backward compatibility with all older instances. Additionally, if a vulnerability |
| or deficiency is ever discovered in a protocol, it allows a newer version of NiFi to forbid communication over the compromised versions of the protocol.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>Site-to-Site is a protocol transferring data between two NiFi instances. Both end can be a standalone NiFi or a NiFi cluster. In this section, the NiFi instance initiates the communications is called <em>Site-to-Site client NiFi instance</em> and the other end as <em>Site-to-Site server NiFi instance</em> to clarify what configuration needed on each NiFi instances.</p> |
| </div> |
| <div class="paragraph"> |
| <p>A NiFi instance can be both client and server for Site-to-Site protocol, however, it can only be a client or server within a specific Site-to-Site communication. For example, if there are three NiFi instances A, B and C. A pushes data to B, and B pulls data from C. <em>A — push → B ← pull — C</em>. Then B is not only a <em>server</em> in the communication between A and B, but also a <em>client</em> in B and C.</p> |
| </div> |
| <div class="paragraph"> |
| <p>It is important to understand which NiFi instance will be the client or server in order to design your data flow, and configure each instance accordingly. Here is a summary of what components run on which side based on data flow direction:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Push: a client <em>sends</em> data to a Remote Process Group, the server <em>receives</em> it with an Input Port</p> |
| </li> |
| <li> |
| <p>Pull: a client <em>receives</em> data from a Remote Process Group, the server <em>sends</em> data through an Output Port</p> |
| </li> |
| </ul> |
| </div> |
| <div class="sect3"> |
| <h4 id="configure-site-to-site-client-nifi-instance"><a class="anchor" href="user-guide.html#configure-site-to-site-client-nifi-instance"></a>Configure Site-to-Site client NiFi instance</h4> |
| <div id="Site-to-Site_Remote_Process_Group" class="paragraph"> |
| <p><strong>Remote Process Group</strong>: In order to communicate with a remote NiFi instance via Site-to-Site, simply drag a <a href="user-guide.html#remote_process_group">Remote Process Group</a> onto the canvas and enter the URL(s) of the remote NiFi instance (for more information on the components of a Remote Process Group, see the <a href="user-guide.html#Remote_Group_Transmission">Remote Process Group Transmission</a> section of this guide.) The URL is the same URL you would use to go to that instance’s User Interface or in the case of a cluster, the URLs of the cluster nodes. At this point, you can drag a connection to or from the Remote Process Group in the same way you would drag a connection to or from a Processor or a local Process Group. When you drag the connection, you will have a chance to choose which Port to connect to. Note that it may take up to one minute for the Remote Process Group to determine which ports are available.</p> |
| </div> |
| <div class="paragraph"> |
| <p>If the connection is dragged starting from the Remote Process Group, the ports shown will be the Output Ports of the remote group, |
| as this indicates that you will be pulling data from the remote instance. If the connection instead ends on the Remote Process Group, |
| the ports shown will be the Input Ports of the remote group, as this implies that you will be pushing data to the remote instance.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| If the remote instance is configured to use secure data transmission, you will see only ports that you are authorized to |
| communicate with. For information on configuring NiFi to run securely, see the |
| <a href="administration-guide.html">System Administrator’s Guide</a>. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div id="Site-to-Site_Transport_Protocol" class="paragraph"> |
| <p><strong>Transport Protocol</strong>: On a Remote Process Group creation or configuration dialog, you can choose Transport Protocol to use for Site-to-Site communication as shown in the following image:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/configure-remote-process-group.png" alt="Configure Remote Process Group"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>By default, it is set to <em>RAW</em> which uses raw socket communication using a dedicated port. <em>HTTP</em> transport protocol is especially useful if the remote |
| NiFi instance is in a restricted network that only allow access through HTTP(S) protocol or only accessible from a specific HTTP Proxy server. |
| For accessing through a HTTP Proxy Server, BASIC and DIGEST authentication are supported.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Local Network Interface</strong>: In some cases, it may be desirable to prefer one network interface over another. For example, if a wired interface and a wireless |
| interface both exist, the wired interface may be preferred. This can be configured by specifying the name of the network interface to use in this box. If the |
| value entered is not valid, the Remote Process Group will not be valid and will not communicate with other NiFi instances until this is resolved.</p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="configure-site-to-site-server-nifi-instance"><a class="anchor" href="user-guide.html#configure-site-to-site-server-nifi-instance"></a>Configure Site-to-Site Server NiFi Instance</h4> |
| <div class="paragraph"> |
| <p><strong>Retrieve Site-to-Site Details</strong>: If your NiFi is running securely, in order for another NiFi instance to retrieve information from your instance, it needs to be added to the Global Access "retrieve site-to-site details" policy. This will allow the other instance to query your instance for details such as name, description, available peers (nodes when clustered), statistics, OS port information and available Input and Output ports. Utilizing Input and Output ports in a secured instance requires additional policy configuration as described below.</p> |
| </div> |
| <div id="Site-to-Site_Input_Port" class="paragraph"> |
| <p><strong>Input Port</strong>: In order to allow another NiFi instance to push data to your local instance, you can simply drag an <a href="user-guide.html#input_port">Input Port</a> onto the Root Process Group of your canvas. After entering a name for the port, it will be added to your flow. You can now right-click on the Input Port and choose Configure in order to adjust the name and the number of concurrent tasks that are used for the port.</p> |
| </div> |
| <div class="paragraph"> |
| <p>To create an Input Port for Site-to-Site in a child Process Group, enter the name for the port and select "Remote connections (site-to-site)" from the Receive From drop-down menu.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/add-input-port-S2S.png" alt="Add Input Port for Site-to-Site"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>If Site-to-Site is configured to run securely, you will need to manage the input port’s "receive data via site-to-site" component access policy. Only those users who have been added to the policy will be able to communicate with the port.</p> |
| </div> |
| <div id="Site-to-Site_Output_Port" class="paragraph"> |
| <p><strong>Output Port</strong>: Similar to an Input Port, a DataFlow Manager may choose to add an <a href="user-guide.html#output_port">Output Port</a> to the Root Process Group. The Output Port allows an authorized NiFi instance to remotely connect to your instance and pull data from the Output Port. After dragging an Output Port onto the canvas, right-click and choose Configure to adjust the name and how many concurrent tasks are allowed. Manage the output port’s "receive data via site-to-site" component access policy to control which users are authorized to pull data from the instance being configured.</p> |
| </div> |
| <div class="paragraph"> |
| <p>To create an Output Port for Site-to-Site in a child Process Group, enter the name for the port and select "Remote connections (site-to-site)" from the Send To drop-down menu.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/add-output-port-S2S.png" alt="Add Output Port for Site-to-Site"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>In addition to other instances of NiFi, some other applications may use a Site-to-Site client in order to push data to or receive data from a NiFi instance.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| For information on how to enable and configure Site-to-Site on a NiFi instance, see the |
| <a href="administration-guide.html#site_to_site_properties">Site-to-Site Properties</a> section of the |
| <a href="administration-guide.html">System Administrator’s Guide</a>. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| For information on how to configure access policies, see the |
| <a href="administration-guide.html#access-policies">Access Policies</a> section of the |
| <a href="administration-guide.html">System Administrator’s Guide</a>. |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="example-dataflow"><a class="anchor" href="user-guide.html#example-dataflow"></a>Example Dataflow</h3> |
| <div class="paragraph"> |
| <p>This section has described the steps required to build a dataflow. Now, to put it all together. The following example dataflow |
| consists of just two processors: GenerateFlowFile and LogAttribute. These processors are normally used for testing, but they can also be used |
| to build a quick flow for demonstration purposes and see NiFi in action.</p> |
| </div> |
| <div class="paragraph"> |
| <p>After you drag the GenerateFlowFile and LogAttribute processors to the canvas and connect them (using the guidelines provided above), configure them as follows:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Generate FlowFile</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>On the Scheduling tab, set Run schedule to: 5 sec. Note that the GenerateFlowFile processor can create many FlowFiles very quickly; that’s why setting the Run schedule is important so that this flow does not overwhelm the system NiFi is running on.</p> |
| </li> |
| <li> |
| <p>On the Properties tab, set File Size to: 10 KB</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p>Log Attribute</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>On the Settings tab, under Auto-terminate relationships, select the checkbox next to Success. This will terminate FlowFiles after this processor has successfully processed them.</p> |
| </li> |
| <li> |
| <p>Also on the Settings tab, set the Bulletin level to Info. This way, when the dataflow is running, this processor will display the bulletin icon (see <a href="user-guide.html#processor_anatomy">Anatomy of a Processor</a>), and the user may hover over it with the mouse to see the attributes that the processor is logging.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>The dataflow should look like the following:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/simple-flow.png" alt="Simple Flow"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Now see the following section on how to start and stop the dataflow. When the dataflow is running, be sure to note the statistical information that is displayed on the face of each processor (see <a href="user-guide.html#processor_anatomy">Anatomy of a Processor</a>).</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="command-and-control-of-the-dataflow"><a class="anchor" href="user-guide.html#command-and-control-of-the-dataflow"></a>Command and Control of the DataFlow</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>When a component is added to the NiFi canvas, it is in the Stopped state. In order to cause the component to |
| be triggered, the component must be started. Once started, the component can be stopped at any time. From a |
| Stopped state, the component can be configured, started, or disabled.</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="starting-a-component"><a class="anchor" href="user-guide.html#starting-a-component"></a>Starting a Component</h3> |
| <div class="paragraph"> |
| <p>In order to start a component, the following conditions must be met:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>The component’s configuration must be valid.</p> |
| </li> |
| <li> |
| <p>All defined Relationships for the component must be connected to another component or auto-terminated.</p> |
| </li> |
| <li> |
| <p>The component must be stopped.</p> |
| </li> |
| <li> |
| <p>The component must be enabled.</p> |
| </li> |
| <li> |
| <p>The component must have no active tasks. For more information about active tasks, see the "Anatomy of …​" |
| sections under <a href="user-guide.html#monitoring">Monitoring of DataFlow</a> (<a href="user-guide.html#processor_anatomy">Anatomy of a Processor</a>, <a href="user-guide.html#process_group_anatomy">Anatomy of a Process Group</a>, <a href="user-guide.html#remote_group_anatomy">Anatomy of a Remote Process Group</a>).</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>Components can be started by selecting all of the components to start and then clicking the "Start" button ( |
| <span class="image"><img src="images/buttonStart.png" alt="Start"></span> |
| ) in the |
| Operate Palette or by right-clicking a single component and choosing Start from the context menu.</p> |
| </div> |
| <div class="paragraph"> |
| <p>If starting a Process Group, all components within that Process Group (including child Process Groups) will |
| be started, with the exception of those components that are invalid or disabled.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Once started, the status indicator of a Processor will change to a Play symbol ( |
| <span class="image"><img src="images/iconRun.png" alt="Run"></span> |
| ).</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="stopping_components"><a class="anchor" href="user-guide.html#stopping_components"></a>Stopping a Component</h3> |
| <div class="paragraph"> |
| <p>A component can be stopped any time that it is running. A component is stopped by right-clicking on the component |
| and clicking Stop from the context menu, or by selecting the component and clicking the "Stop" button ( |
| <span class="image"><img src="images/buttonStop.png" alt="Stop"></span> |
| ) in the Operate Palette.</p> |
| </div> |
| <div class="paragraph"> |
| <p>If a Process Group is stopped, all of the components within the Process Group (including child Process Groups) |
| will be stopped.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Once stopped, the status indicator of a component will change to the Stop symbol ( |
| <span class="image"><img src="images/iconStop.png" alt="Stop"></span> |
| ).</p> |
| </div> |
| <div class="paragraph"> |
| <p>Stopping a component does not interrupt its currently running tasks. Rather, it stops scheduling new tasks to |
| be performed. The number of active tasks is shown in the top-right corner of the Processor (See <a href="user-guide.html#processor_anatomy">Anatomy of a Processor</a> |
| for more information). See <a href="user-guide.html#terminating_tasks">Terminating a Component’s Tasks</a> for how to terminate the running tasks.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="terminating_tasks"><a class="anchor" href="user-guide.html#terminating_tasks"></a>Terminating a Component’s Tasks</h3> |
| <div class="paragraph"> |
| <p>When a component is stopped, it does not interrupt the currently running tasks. This allows for the current execution to complete while no new |
| tasks are scheduled, which is the desired behavior in many cases. In some cases, it is desirable to terminate the running tasks, particularly |
| in cases where a task has hung and is no longer responsive, or while developing new flows.</p> |
| </div> |
| <div class="paragraph"> |
| <p>To be able to terminate the running task(s), the component must first be stopped (see <a href="user-guide.html#stopping_components">Stopping a Component</a>). Once the component is in the |
| Stopped state, the Terminate option will become available only if there are tasks still running (see <a href="user-guide.html#processor_anatomy">Anatomy of a Processor</a>). The Terminate option |
| (<span class="image"><img src="images/iconTerminate.png" alt="Terminate"></span>) can be accessed via the context menu or the Operate Palette while the component is selected.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The number of tasks that are actively being terminated will be displayed in parentheses next to the number of active tasks (<span class="image"><img src="images/terminated-thread.png" alt="Terminated-Threads"></span>). For example, if there is one active task at the time that Terminate is selected, this will display "0 (1)" - meaning |
| 0 active tasks and 1 task being terminated.</p> |
| </div> |
| <div class="paragraph"> |
| <p>A task may not terminate immediately, as different components may respond to the Terminate command differently. However, the components can be |
| reconfigured and started/stopped regardless of whether there are tasks still in the terminating state.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="enablingdisabling-a-component"><a class="anchor" href="user-guide.html#enablingdisabling-a-component"></a>Enabling/Disabling a Component</h3> |
| <div class="paragraph"> |
| <p>When a component is enabled, it is able to be started. Users may choose to disable components when they are part of a |
| dataflow that is still being assembled, for example. Typically, if a component is not intended to be run, the component |
| is disabled, rather than being left in the Stopped state. This helps to distinguish between components that are |
| intentionally not running and those that may have been stopped temporarily (for instance, to change the component’s |
| configuration) and inadvertently were never restarted.</p> |
| </div> |
| <div class="paragraph"> |
| <p>When it is desirable to re-enable a component, it can be enabled by selecting the component and |
| clicking the "Enable" button (<span class="image"><img src="images/buttonEnable.png" alt="Enable"></span> |
| ) in the Operate Palette. This is available only when the selected component or components are disabled. |
| Alternatively, a component can be enabled by checking the checkbox next to the "Enabled" option in |
| the Settings tab of the Processor configuration dialog or the configuration dialog for a Port.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Once enabled, the component’s status indicator will change to either Invalid ( |
| <span class="image"><img src="images/iconAlert.png" alt="Invalid"></span> |
| ) or Stopped ( |
| <span class="image"><img src="images/iconStop.png" alt="Stopped"></span> |
| ), depending on whether or not the component is valid.</p> |
| </div> |
| <div class="paragraph"> |
| <p>A component is then disabled by selecting the component and clicking the "Disable" button ( |
| <span class="image"><img src="images/buttonDisable.png" alt="Disable"></span> |
| ) in the Operate Palette, or by clearing the checkbox next to the "Enabled" option in the Settings tab |
| of the Processor configuration dialog or the configuration dialog for a Port.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Only Ports and Processors can be enabled and disabled.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="Remote_Group_Transmission"><a class="anchor" href="user-guide.html#Remote_Group_Transmission"></a>Remote Process Group Transmission</h3> |
| <div class="paragraph"> |
| <p>Remote Process Groups provide a mechanism for sending data to or retrieving data from a remote instance |
| of NiFi. When a Remote Process Group (RPG) is added to the canvas, it is added with the Transmission Disabled, |
| as indicated by the icon ( |
| <span class="image"><img src="images/iconTransmissionInactive.png" alt="Transmission Disabled"></span> |
| ) in the top-left corner. When Transmission is Disabled, it can be enabled by right-clicking on the |
| RPG and clicking the "Enable transmission" menu item. This will cause all ports for which there is a Connection |
| to begin transmitting data. This will cause the status indicator to then change to the Transmission Enabled icon ( |
| <span class="image"><img src="images/iconTransmissionActive.png" alt="Transmission Enabled"></span> |
| ).</p> |
| </div> |
| <div class="paragraph"> |
| <p>If there are problems communicating with the Remote Process Group, a Warning indicator ( |
| <span class="image"><img src="images/iconAlert.png" alt="Warning"></span> |
| ) may instead be present in the top-left corner. Hovering over this Warning indicator with the mouse will provide |
| more information about the problem.</p> |
| </div> |
| <div class="sect3"> |
| <h4 id="Remote_Port_Configuration"><a class="anchor" href="user-guide.html#Remote_Port_Configuration"></a>Individual Port Transmission</h4> |
| <div class="paragraph"> |
| <p>There are times when the DFM may want to either enable or disable transmission for only a specific port within the Remote Process Group. This can be accomplished by right-clicking on the Remote Process Group and choosing the "Manage remote ports" menu item. This provides a configuration dialog from which ports can be configured:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/remote-group-ports-dialog.png" alt="Remote Process Group Ports"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The left-hand side lists all of the Input Ports that the remote instance of NiFi allows data to be sent to. The right-hand side lists all of the Output Ports from which this instance is able to pull data. If the remote instance is using secure communications (the URL of the NiFi instance begins with <code>https://</code>, rather than <code>http://</code>), any ports that the remote instance has not made available to this instance will not be shown.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| If a port that is expected to be shown is not shown in this dialog, ensure that the instance has proper permissions and that the Remote Process Group’s flow is current. This can be checked by closing the Remote Process Group Ports dialog and looking at the bottom-left corner of the Remote Process Group. The date and time when the flow was last refreshed is displayed. If the flow appears to be outdated, it can be updated by right-clicking on the Remote Process Group and selecting "Refresh remote". (See <a href="user-guide.html#remote_group_anatomy">Anatomy of a Remote Process Group</a> for more information). |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="paragraph"> |
| <p>Each port is shown with its Name, its Description, configured number of Concurrent Tasks, and whether or not data sent to this port will be Compressed. Additionally, the port’s configured Batch Settings (Count, Size and Duration) are displayed. To the left of this information is a toggle switch to turn the port on or off. Ports that have no connections attached to them are grayed out:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/remote-port-connection-status.png" alt="Remote Port Statuses"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The on/off toggle switch provides a mechanism to enable and disable transmission for each port in the Remote Process Group independently. Those ports that are connected but are not currently transmitting can be configured by clicking the pencil icon (<span class="image"><img src="images/iconEdit.png" alt="Edit"></span>) below the on/off toggle switch. Clicking this icon will allow the DFM to change the number of Concurrent Tasks, whether or not compression should be used when transmitting data to or from this port, and Batch Settings.</p> |
| </div> |
| <div class="paragraph"> |
| <p>For an Input Port, the batch settings control how NiFi sends data to the remote input port in a transaction. NiFi will transfer FlowFiles, as they are queued in incoming relationships, until any of the limits (Count, Size, Duration) is met. If none of the settings are configured, a 500 milliseconds batch duration is used by default.</p> |
| </div> |
| <div class="paragraph"> |
| <p>For an Output Port, the batch settings tells the remote NiFi how NiFi prefers to receive data from the remote output port in a transaction. The remote NiFi will use the specified settings (Count, Size, Duration) to control the transfer of FlowFiles. If none of the settings are configured, a 5 seconds batch duration is used by default.</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="navigating"><a class="anchor" href="user-guide.html#navigating"></a>Navigating within a DataFlow</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>NiFi provides various mechanisms for getting around a dataflow. The <a href="user-guide.html#User_Interface">NiFi User Interface</a> section describes various ways to navigate around the NiFi canvas; however, once a flow exists on the canvas, there are additional ways to get from one component to another. When multiple Process Groups exist in a flow, breadcrumbs appear at the bottom of the screen, providing a way to navigate between them. In addition, to enter a Process Group that is currently visible on the canvas, simply double-click it, thereby "drilling down" into it. To leave a Process Group there are multiple ways: from the context menu opened by right-click on the canvas; with the 'Leave group' button on the 'Navigation' panel; with 'esc' key if no modal or context menu is open. Connections also provide a way to jump from one location to another within the flow. Right-click on a connection and select "Go to source" or "Go to destination" in order to jump to one end of the connection or another. This can be very useful in large, complex dataflows, where the connection lines may be long and span large areas of the canvas. Finally, all components provide the ability to jump forward or backward within the flow. Right-click any component (e.g., a processor, process group, port, etc.) and select either "Upstream connections" or "Downstream connections". A dialog window will open, showing the available upstream or downstream connections that the user may jump to. This can be especially useful when trying to follow a dataflow in a backward direction. It is typically easy to follow the path of a dataflow from start to finish, drilling down into nested process groups; however, it can be more difficult to follow the dataflow in the other direction.</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="component_linking"><a class="anchor" href="user-guide.html#component_linking"></a>Component Linking</h3> |
| <div class="paragraph"> |
| <p>A hyperlink can be used to navigate directly to a component on the NiFi canvas. This is especially useful when <a href="administration-guide.html#multi-tenant-authorization">Multi-Tenant Authorization</a> is configured. For example, a URL can be given to a user to direct them to the specific process group to which they have privileges.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The default URL for a NiFI instance is <code><a href="https://localhost:8443/nifi" class="bare">https://localhost:8443/nifi</a></code>, which points to the root process group. When a component is selected on the canvas, the URL is updated with the component’s process group id and component id in the form <code><a href="https://localhost:8443/nifi/?processGroupId=<UUID>&componentIds=<UUIDs>" class="bare">https://localhost:8443/nifi/?processGroupId=<UUID>&componentIds=<UUIDs></a>;</code>. In the following screenshot, the GenerateFlowFile processor in the process group PG1 is the selected component:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/component-linking-processor.png" alt="Component Linking Processor Example"> |
| </div> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| Linking to multiple components on the canvas is supported, with the restriction that the length of the URL cannot exceed a 2000 character limit. |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="component_alignment"><a class="anchor" href="user-guide.html#component_alignment"></a>Component Alignment</h3> |
| <div class="paragraph"> |
| <p>Components on the NiFi canvas can be aligned to more precisely arrange your dataflow. To do this, first select all the components you want to align. Then right-click to see the context menu and select “Align vertically” or “Align horizontally” depending on your desired result.</p> |
| </div> |
| <div class="sect3"> |
| <h4 id="align-vertically"><a class="anchor" href="user-guide.html#align-vertically"></a>Align Vertically</h4> |
| <div class="paragraph"> |
| <p>Here is an example of aligning components vertically on your canvas. With all components selected/highlighted, right-click:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/align-vertically-before.png" alt="Align Vertically Example Before"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>and select "Align vertically" to achieve these results:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/align-vertically-after.png" alt="Align Vertically Example After"></span></p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="align-horizontally"><a class="anchor" href="user-guide.html#align-horizontally"></a>Align Horizontally</h4> |
| <div class="paragraph"> |
| <p>Here is an example of aligning components horizontally on your canvas. With all components selected/highlighted, right-click:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/align-horizontally-before.png" alt="Align Horizontally Example Before"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>and select "Align horizontally" to achieve these results:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/align-horizontally-after.png" alt="Align Horizontally Example Before"></span></p> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="search"><a class="anchor" href="user-guide.html#search"></a>Search Components in DataFlow</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>NiFi UI provides searching functionality in order to help easily find components on the canvas. You can use search to find components by name, type, identifier, configuration properties, and their values. Search also makes it possible to refine and narrow the search result based on certain conditions using Filters and Keywords.</p> |
| </div> |
| <div class="exampleblock"> |
| <div class="title">Example 1: The result will contain components that match for "processor1".</div> |
| <div class="content"> |
| <div class="paragraph"> |
| <p>processor1</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="filters"><a class="anchor" href="user-guide.html#filters"></a>Filters</h3> |
| <div class="paragraph"> |
| <p>Filters can be added to the search box as key-value pairs where the keys are predefined and check certain conditions based on the given value. The syntax is "key:value".</p> |
| </div> |
| <div class="exampleblock"> |
| <div class="title">Example 2: The search will be executed under Process Groups (directly or via contained Process Groups) containing the string "myGroup" in their name or id. The result will contain components that match for "processor1".</div> |
| <div class="content"> |
| <div class="paragraph"> |
| <p>group:myGroup processor1</p> |
| </div> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Filters can be used together with other search terms and multiple filters can be used. The only constraint is that the search must start with the filters. Unknown filters or known filters with unknown values are ignored. If the same filter key appears multiple times, the first will be used. The order of different filters has no effect on the result.</p> |
| </div> |
| <div class="exampleblock"> |
| <div class="title">Example 3: Search will be restricted to the currently active process group (and process groups within that). The result will contain components that match for "import" but property matches will be excluded.</div> |
| <div class="content"> |
| <div class="paragraph"> |
| <p>scope:here properties:exclude import</p> |
| </div> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The supported filters are the following:</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>scope</strong>: This filter narrows the scope of the search based on the user’s currently active Process Group. The only valid value is "here". The usage of this filter looks like "scope:here". Any other value is considered as invalid, thus the filter will be ignored during search.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>group</strong>: This filter narrows the scope of the search based on the provided Process Group name or id. Search will be restricted to groups (and their components - including subgroups and their components) the names or ids of which match the filter value. If no group matches the filter, the result list will be empty.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>properties</strong>: With this, users can prevent property matches to appear in the search result. Valid values are: "no", "none", "false", "exclude" and "0".</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="keywords"><a class="anchor" href="user-guide.html#keywords"></a>Keywords</h3> |
| <div class="paragraph"> |
| <p>Users can use pre-defined (case-insensitive) keywords in the search box that will check certain conditions.</p> |
| </div> |
| <div class="exampleblock"> |
| <div class="title">Example 4: "disabled" will be treated both as keyword and regular search term. The result will contain disabled Ports and Processors as all other components that match for "disabled" in any way.</div> |
| <div class="content"> |
| <div class="paragraph"> |
| <p>disabled</p> |
| </div> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Keywords can be used with filters (see below) but not with other search terms (otherwise they won’t be treated as keywords) and only one keyword can be used at a time. Note however that keywords will also be treated as general search terms at the same time.</p> |
| </div> |
| <div class="exampleblock"> |
| <div class="title">Example 5: Search will be restricted to the currently selected process group (and its sub process groups). "invalid" here (as it is alone after the filter) will be treated both as a keyword and a regular search term. The result will contain invalid Processors and Ports as well as all other components that match for "invalid" in any way.</div> |
| <div class="content"> |
| <div class="paragraph"> |
| <p>scope:here invalid</p> |
| </div> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The supported keywords are the following:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Scheduled state</strong></p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>disabled</strong>: Adds disabled Ports and Processors to the result list.</p> |
| </li> |
| <li> |
| <p><strong>invalid</strong>: Adds Ports and Processors to the result list where the component is invalid.</p> |
| </li> |
| <li> |
| <p><strong>running</strong>: Adds running Ports and Processors to the result list.</p> |
| </li> |
| <li> |
| <p><strong>stopped</strong>: Adds stopped Ports and Processors to the result list.</p> |
| </li> |
| <li> |
| <p><strong>validating</strong>: Adds Processors to the result list that are validating at the time.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p><strong>Scheduling strategy</strong></p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>event</strong>: Adds Processors to the result list where the Scheduling Strategy is "Event Driven".</p> |
| </li> |
| <li> |
| <p><strong>timer</strong>: Adds Processors to the result list where the Scheduling Strategy is "Timer Driven".</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p><strong>Execution</strong></p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>primary:</strong> Adds Processors to the result list that are set to run on the primary node only (whether if the Processor is currently running or not).</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p><strong>Back pressure</strong></p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>back pressure</strong>: Adds Connections to the result list that are applying back pressure at the time.</p> |
| </li> |
| <li> |
| <p><strong>pressure</strong>: See "back pressure".</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p><strong>Expiration</strong></p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>expiration</strong>: Adds Connections to the result list that contain expired FlowFiles.</p> |
| </li> |
| <li> |
| <p><strong>expires</strong>: See "expiration".</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p><strong>Transmission</strong></p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>not transmitting</strong>: Adds Remote Process Groups to the result list that are not transmitting data at the time.</p> |
| </li> |
| <li> |
| <p><strong>transmitting</strong>: Adds Remote Process Groups to the result list that are transmitting data at the time.</p> |
| </li> |
| <li> |
| <p><strong>transmission disabled</strong>: See "not transmitting".</p> |
| </li> |
| <li> |
| <p><strong>transmitting enabled</strong>: See "transmitting".</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="monitoring"><a class="anchor" href="user-guide.html#monitoring"></a>Monitoring of DataFlow</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>NiFi provides a great deal of information about the DataFlow in order to monitor its |
| health and status. The Status bar provides information about the overall system health |
| (see <a href="user-guide.html#User_Interface">NiFi User Interface</a>). Processors, Process Groups, and Remote Process Groups |
| provide fine-grained details about their operations. Connections and Process Groups provide information |
| about the amount of data in their queues. The Summary Page provides information about all of the components |
| on the canvas in a tabular format and also provides System Diagnostics that include disk usage, |
| CPU utilization, and Java Heap and Garbage Collection information. In a clustered environment, this |
| information is available per-node or as aggregates across the entire cluster. We will explore each of these |
| monitoring artifacts below.</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="processor_anatomy"><a class="anchor" href="user-guide.html#processor_anatomy"></a>Anatomy of a Processor</h3> |
| <div class="paragraph"> |
| <p>NiFi provides a significant amount of information about each Processor on the canvas. The following diagram |
| shows the anatomy of a Processor:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/processor-anatomy.png" alt="Anatomy of a Processor"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The image outlines the following elements:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Processor Type</strong>: NiFi provides several different types of Processors in order to allow for a wide range |
| of tasks to be performed. Each type of Processor is designed to perform one specific task. The Processor |
| type (PutFile, in this example) describes the task that this Processor performs. In this case, the |
| Processor writes a FlowFile to disk - or "Puts" a FlowFile to a File.</p> |
| </li> |
| <li> |
| <p><strong>Bulletin Indicator</strong>: When a Processor logs that some event has occurred, it generates a Bulletin to notify |
| those who are monitoring NiFi via the User Interface. The DFM is able to configure which |
| bulletins should be displayed in the User Interface by updating the "Bulletin level" field in the |
| "Settings" tab of the Processor configuration dialog. The default value is <code>WARN</code>, which means that only |
| warnings and errors will be displayed in the UI. This icon is not present unless a Bulletin exists for this |
| Processor. When it is present, hovering over the icon with the mouse will provide a tooltip explaining the |
| message provided by the Processor as well as the Bulletin level. If the instance of NiFi is clustered, |
| it will also show the Node that emitted the Bulletin. Bulletins automatically expire after five minutes.</p> |
| </li> |
| <li> |
| <p><strong>Status Indicator</strong>: Shows the current Status of the Processor. The following indicators are possible:</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><span class="image"><img src="images/iconRun.png" alt="Running"></span> |
| <strong>Running</strong>: The Processor is currently running.</p> |
| </li> |
| <li> |
| <p><span class="image"><img src="images/iconStop.png" alt="Stopped"></span> |
| <strong>Stopped</strong>: The Processor is valid and enabled but is not running.</p> |
| </li> |
| <li> |
| <p><span class="image"><img src="images/iconAlert.png" alt="Invalid"></span> |
| <strong>Invalid</strong>: The Processor is enabled but is not currently valid and cannot be started. |
| Hovering over this icon will provide a tooltip indicating why the Processor is not valid.</p> |
| </li> |
| <li> |
| <p><span class="image"><img src="images/iconDisable.png" alt="Disabled"></span> |
| <strong>Disabled</strong>: The Processor is not running and cannot be started until it has been enabled. |
| This status does not indicate whether or not the Processor is valid.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p><strong>Processor Name</strong>: This is the user-defined name of the Processor. By default, the name of the Processor is |
| the same as the Processor Type. In the example, this value is "Copy to /review".</p> |
| </li> |
| <li> |
| <p><strong>Active Tasks</strong>: The number of tasks that this Processor is currently executing. This number is constrained |
| by the "Concurrent tasks" setting in the "Scheduling" tab of the Processor configuration dialog. |
| Here, we can see that the Processor is currently performing one task. If the NiFi instance is clustered, |
| this value represents the number of tasks that are currently executing across all nodes in the cluster.</p> |
| </li> |
| <li> |
| <p><strong>5-Minute Statistics</strong>: The Processor shows several different statistics in tabular form. Each of these |
| statistics represents the amount of work that has been performed in the past five minutes. If the NiFi |
| instance is clustered, these values indicate how much work has been done by all of the Nodes combined |
| in the past five minutes. These metrics are:</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>In</strong>: The amount of data that the Processor has pulled from the queues of its incoming Connections. |
| This value is represented as <count> (<size>) where <count> is the number of FlowFiles that have been |
| pulled from the queues and <size> is the total size of those FlowFiles' content. In this example, |
| the Processor has pulled 29 FlowFiles from the input queues, for a total of 14.16 megabytes (MB).</p> |
| </li> |
| <li> |
| <p><strong>Read/Write</strong>: The total size of the FlowFile content that the Processor has read from disk and written |
| to disk. This provides valuable information about the I/O performance that this Processor requires. |
| Some Processors may only read the data without writing anything while some will not read the data but |
| will only write data. Others will neither read nor write data, and some Processors will both read |
| and write data. In this example, we see that in the past five minutes, this Processor has read 4.88 |
| MB of the FlowFile content and has written 4.88 MB as well. This is what we would expect, |
| since this Processor simply copies the contents of a FlowFile to disk. Note, however, that this is |
| not the same as the amount of data that it pulled from its input queues. This is because some of |
| the files that it pulled from the input queues already exist in the output directory, and the |
| Processor is configured to route FlowFiles to failure when this occurs. Therefore, for those files |
| which already existed in the output directory, data was neither read nor written to disk.</p> |
| </li> |
| <li> |
| <p><strong>Out</strong>: The amount of data that the Processor has transferred to its outbound Connections. This does |
| not include FlowFiles that the Processor removes itself, or FlowFiles that are routed to connections |
| that are auto-terminated. Like the "In" metric above, this value is represented as <count> (<size>) |
| where <count> is the number of FlowFiles that have been transferred to outbound Connections and <size> |
| is the total size of those FlowFiles' content. In this example, all of the Relationships are configured to be |
| auto-terminated, so no FlowFiles are reported as having been transferred Out.</p> |
| </li> |
| <li> |
| <p><strong>Tasks/Time</strong>: The number of times that this Processor has been triggered to run in the past 5 minutes, and |
| the amount of time taken to perform those tasks. The format of the time is <hour>:<minute>:<second>. Note |
| that the amount of time taken can exceed five minutes, because many tasks can be executed in parallel. For |
| instance, if the Processor is scheduled to run with 60 Concurrent tasks, and each of those tasks takes one |
| second to complete, it is possible that all 60 tasks will be completed in a single second. However, in this |
| case we will see the Time metric showing that it took 60 seconds, instead of 1 second. This time can be |
| thought of as "System Time," or said another way, this value is 60 seconds because that’s the amount of |
| time it would have taken to perform the action if only a single concurrent task were used.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| </ul> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="process_group_anatomy"><a class="anchor" href="user-guide.html#process_group_anatomy"></a>Anatomy of a Process Group</h3> |
| <div class="paragraph"> |
| <p>The Process Group provides a mechanism for grouping components together into a logical construct in order |
| to organize the DataFlow in a way that makes it more understandable from a higher level. |
| The following image highlights the different elements that make up the anatomy of a Process Group:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/process-group-anatomy.png" alt="Anatomy of a Process Group"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The Process Group consists of the following elements:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Name</strong>: This is the user-defined name of the Process Group. This name is set when the Process Group |
| is added to the canvas. The name can later by changed by right-clicking on the Process Group and clicking |
| the "Configure" menu option. In this example, the name of the Process Group is "Process Group ABC."</p> |
| </li> |
| <li> |
| <p><strong>Bulletin Indicator</strong>: When a child component of a Process Group emits a bulletin, that bulletin is propagated to |
| the component’s parent Process Group, as well. When any component has an active Bulletin, this indicator will appear, |
| allowing the user to hover over the icon with the mouse to see the Bulletin.</p> |
| </li> |
| <li> |
| <p><strong>Active Tasks</strong>: The number of tasks that are currently executing by the components within this |
| Process Group. Here, we can see that the Process Group is currently performing two tasks. If the |
| NiFi instance is clustered, this value represents the number of tasks that are currently executing |
| across all nodes in the cluster.</p> |
| </li> |
| <li> |
| <p><strong>Statistics</strong>: Process Groups provide statistics about the amount of data that has been processed by the Process Group in |
| the past 5 minutes as well as the amount of data currently enqueued within the Process Group. The following elements |
| comprise the "Statistics" portion of a Process Group:</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Queued</strong>: The number of FlowFiles currently enqueued within the Process Group. |
| This field is represented as <count> (<size>) where <count> is the number of FlowFiles that are |
| currently enqueued in the Process Group and <size> is the total size of those FlowFiles' content. In this example, |
| the Process Group currently has 26 FlowFiles enqueued with a total size of 12.7 megabytes (MB).</p> |
| </li> |
| <li> |
| <p><strong>In</strong>: The number of FlowFiles that have been transferred into the Process Group through all of its Input Ports |
| over the past 5 minutes. This field is represented as <count> / <size> → <ports> where <count> is the number of FlowFiles that have entered the Process Group in the past 5 minutes, <size> is the total size of those FlowFiles' content and <ports> is the number of Input Ports. In this example, 8 FlowFiles have entered the Process Group with a total size of 800 KB and two Input Ports exist.</p> |
| </li> |
| <li> |
| <p><strong>Read/Write</strong>: The total size of the FlowFile content that the components within the Process Group have |
| read from disk and written to disk. This provides valuable information about the I/O performance that this |
| Process Group requires. In this example, we see that in the past five minutes, components within this |
| Process Group have read 14.72 MB of the FlowFile content and have written 14.8 MB.</p> |
| </li> |
| <li> |
| <p><strong>Out</strong>: The number of FlowFiles that have been transferred out of the Process Group through its Output Ports |
| over the past 5 minutes. This field is represented as <ports> → <count> (<size>) where <ports> is the number of Output Ports, <count> is the number of FlowFiles that have exited the Process Group in the past 5 minutes and <size> is the total size of those FlowFiles' content. In this example, there are three Output Ports, 16 FlowFiles have exited the Process Group and their total size is 78.57 KB.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p><strong>Component Counts</strong>: The Component Counts element provides information about how many components of each type exist |
| within the Process Group. The following provides information about each of these icons and their meanings:</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><span class="image"><img src="images/iconTransmissionActive.png" alt="Transmission Active"></span> |
| <strong>Transmitting Ports</strong>: The number of Remote Process Group Ports that currently are configured to transmit data to remote |
| instances of NiFi or pull data from remote instances of NiFi.</p> |
| </li> |
| <li> |
| <p><span class="image"><img src="images/iconTransmissionInactive.png" alt="Transmission Inactive"></span> |
| <strong>Non-Transmitting Ports</strong>: The number of Remote Process Group Ports that are currently connected to components within this |
| Process Group but currently have their transmission disabled.</p> |
| </li> |
| <li> |
| <p><span class="image"><img src="images/iconRun.png" alt="Running"></span> |
| <strong>Running Components</strong>: The number of Processors, Input Ports, and Output Ports that are currently running within this |
| Process Group.</p> |
| </li> |
| <li> |
| <p><span class="image"><img src="images/iconStop.png" alt="Stopped Components"></span> |
| <strong>Stopped Components</strong>: The number of Processors, Input Ports, and Output Ports that are currently not running but are |
| valid and enabled. These components are ready to be started.</p> |
| </li> |
| <li> |
| <p><span class="image"><img src="images/iconAlert.png" alt="Invalid Components"></span> |
| <strong>Invalid Components</strong>: The number of Processors, Input Ports, and Output Ports that are enabled but are currently |
| not in a valid state. This may be due to misconfigured properties or missing Relationships.</p> |
| </li> |
| <li> |
| <p><span class="image"><img src="images/iconDisable.png" alt="Disabled Components"></span> |
| <strong>Disabled Components</strong>: The number of Processors, Input Ports, and Output Ports that are currently disabled. These |
| components may or may not be valid. If the Process Group is started, these components will not cause any errors |
| but will not be started.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p><strong>Version State Counts</strong>: The Version State Counts element provides information about how many versioned process groups are within the Process Group. See <a href="user-guide.html#version_states">Version States</a> for more information.</p> |
| </li> |
| <li> |
| <p><strong>Comments</strong>: When the Process Group is added to the canvas, the user is given the option of specifying Comments in order to provide information about the Process Group. The comments can later be changed by right-clicking on the Process Group and clicking the "Configure" menu option.</p> |
| </li> |
| </ul> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="remote_group_anatomy"><a class="anchor" href="user-guide.html#remote_group_anatomy"></a>Anatomy of a Remote Process Group</h3> |
| <div class="paragraph"> |
| <p>When creating a DataFlow, it is often necessary to transfer data from one instance of NiFi to another. |
| In this case, the remote instance of NiFi can be thought of as a Process Group. For this reason, NiFi |
| provides the concept of a Remote Process Group. From the User Interface, the Remote Process Group |
| looks similar to the Process Group. However, rather than showing information about the inner workings |
| and state of a Remote Process Group, such as queue sizes, the information rendered about a Remote |
| Process Group is related to the interaction that occurs between this instance of NiFi and the remote |
| instance.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/remote-group-anatomy.png" alt="Anatomy of a Remote Process Group"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The image above shows the different elements that make up a Remote Process Group. Here, we provide an |
| explanation of the icons and details about the information provided.</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Transmission Status</strong>: The Transmission Status indicates whether or not data Transmission between this |
| instance of NiFi and the remote instance is currently enabled. The icon shown will be the |
| Transmission Enabled icon ( |
| <span class="image"><img src="images/iconTransmissionActive.png" alt="Transmission Active"></span> |
| ) if any of the Input Ports or Output Ports is currently configured to transmit or the Transmission |
| Disabled icon ( |
| <span class="image"><img src="images/iconTransmissionInactive.png" alt="Transmission Inactive"></span> |
| ) if all of the Input Ports and Output Ports that are currently connected are stopped.</p> |
| </li> |
| <li> |
| <p><strong>Remote Instance Name</strong>: This is the name of the NiFi instance that was reported by the remote instance. |
| When the Remote Process Group is first created, before this information has been obtained, the URL(s) |
| of the remote instance will be shown here instead.</p> |
| </li> |
| <li> |
| <p><strong>Remote Instance URL</strong>: This is the URL of the remote instance that the Remote Process Group points to. |
| This URL is entered when the Remote Process Group is added to the canvas and it cannot be changed.</p> |
| </li> |
| <li> |
| <p><strong>Secure Indicator</strong>: This icon indicates whether or not communications with the remote NiFi instance are |
| secure. If communications with the remote instance are secure, this will be indicated by the "Locked" |
| icon ( |
| <span class="image"><img src="images/iconSecure.png" alt="Secure"></span> |
| ). If the communications are not secure, this will be indicated by the "Unlocked" icon ( |
| <span class="image"><img src="images/iconNotSecure.png" alt="Not Secure"></span> |
| ). If the communications are secure, this instance of NiFi will not be able to communicate with the |
| remote instance until an administrator for the remote instance grants access. Whenever the Remote Process |
| Group is added to the canvas, this will automatically initiate a request to have a user for this instance of NiFi created on the |
| remote instance. This instance will be unable to communicate with the remote instance until an administrator |
| on the remote instance adds the user to the system and adds the "NiFi" role to the user. |
| In the event that communications are not secure, the Remote Process Group is able to receive data from anyone, |
| and the data is not encrypted while it is transferred between instances of NiFi.</p> |
| </li> |
| <li> |
| <p><strong>5-Minute Statistics</strong>: Two statistics are shown for Remote Process Groups: <strong>Sent</strong> and <strong>Received</strong>. Both of these are |
| in the format <count> (<size>) where <count> is the number of FlowFiles that have been sent or received in the previous |
| five minutes and <size> is the total size of those FlowFiles' content.</p> |
| </li> |
| <li> |
| <p><strong>Last Refresh Time</strong>: The information that is pulled from a remote instance and rendered on the Remote Process Group |
| in the User Interface is periodically refreshed in the background. This element indicates the time at which that refresh |
| last happened, or if the information has not been refreshed for a significant amount of time, the value will change to |
| indicate <em>Remote flow not current</em>. NiFi can be triggered to initiate a refresh of this information by right-clicking |
| on the Remote Process Group and choosing the "Refresh remote" menu item.</p> |
| </li> |
| </ul> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="Queue_Interaction"><a class="anchor" href="user-guide.html#Queue_Interaction"></a>Queue Interaction</h3> |
| <div class="paragraph"> |
| <p>The FlowFiles enqueued in a Connection can be viewed when necessary. The Queue listing is opened via <code>List queue</code> in |
| a Connection’s context menu. The listing will return the top 100 FlowFiles in the active queue according to the |
| configured priority. The listing can be performed even if the source and destination are actively running.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Additionally, details for a FlowFile in the listing can be viewed by clicking the "Details" button (<span class="image"><img src="images/iconDetails.png" alt="Details"></span>) in the left most column. From here, the FlowFile details and attributes are available as well as buttons for |
| downloading or viewing the content. Viewing the content is only available if the <code>nifi.content.viewer.url</code> has been configured. |
| If the source or destination of the Connection are actively running, there is a chance that the desired FlowFile will |
| no longer be in the active queue.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The FlowFiles enqueued in a Connection can also be deleted when necessary. The removal of the FlowFiles is initiated |
| via <code>Empty queue</code> in the Connection’s context menu. This action can also be performed if the source and destination |
| are actively running.</p> |
| </div> |
| <div class="paragraph"> |
| <p>If the analytics prediction feature is enabled, hovering over the queue will also reveal predicted statistics on when the queue may encounter back pressure, either due to the object count or content size meeting the current threshold |
| settings. Predictions will only be available when NiFi has enough data in its internal repository and if its model is accurate enough to broadcast a prediction. For more information, see the <a href="administration-guide.html#analytics_framework">Analytics Framework</a> section in the System Administrator’s Guide.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="Summary_Page"><a class="anchor" href="user-guide.html#Summary_Page"></a>Summary Page</h3> |
| <div class="paragraph"> |
| <p>While the NiFi canvas is useful for understanding how the configured DataFlow is laid out, this view is not always optimal |
| when trying to discern the status of the system. In order to help the user understand how the DataFlow is functioning |
| at a higher level, NiFi provides a Summary page. This page is available in the Global Menu in the top-right corner |
| of the User Interface. See the <a href="user-guide.html#User_Interface">NiFi User Interface</a> section for more information about the location of this toolbar.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The Summary Page is opened by selecting Summary from the Global Menu. This opens the Summary table dialog:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/summary-table.png" alt="Summary Table"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>This dialog provides a great deal of information about each of the components on the canvas. Below, we have annotated |
| the different elements within the dialog in order to make the discussion of the dialog easier.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/summary-annotated.png" alt="Summary Table Annotated"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The Summary page is primarily comprised of a table that provides information about each of the components on the canvas. Above this |
| table is a set of five tabs that can be used to view the different types of components. The information provided in the table |
| is the same information that is provided for each component on the canvas. Each of the columns in the table may be sorted by |
| clicking on the heading of the column. For more on the types of information displayed, see the sections |
| <a href="user-guide.html#processor_anatomy">Anatomy of a Processor</a>, <a href="user-guide.html#process_group_anatomy">Anatomy of a Process Group</a>, and <a href="user-guide.html#remote_group_anatomy">Anatomy of a Remote Process Group</a> above.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The Summary page also includes the following elements:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Bulletin Indicator</strong>: As in other places throughout the User Interface, when this icon is present, hovering over the icon will |
| provide information about the Bulletin that was generated, including the message, the severity level, the time at which |
| the Bulletin was generated, and (in a clustered environment) the node that generated the Bulletin. Like all the columns in the |
| Summary table, this column where bulletins are shown may be sorted |
| by clicking on the heading so that all the currently existing bulletins are shown at the top of the list.</p> |
| </li> |
| <li> |
| <p><strong>Details</strong>: Clicking the Details icon will provide the user with the details of the component. This dialog is the same as the |
| dialog provided when the user right-clicks on the component and chooses the "View Configuration" menu item.</p> |
| </li> |
| <li> |
| <p><strong>Go To</strong>: Clicking this button will close the Summary page and take the user directly to the component on the NiFi canvas. This |
| may change the Process Group that the user is currently in. This icon is not available if the Summary page has been opened |
| in a new browser tab or window (by clicking the "Pop Out" button, as described below).</p> |
| </li> |
| <li> |
| <p><strong>Status History</strong>: Clicking the Status History icon will open a new dialog that shows a historical view of the statistics that |
| are rendered for this component. See the section <a href="user-guide.html#Status_History">Historical Statistics of a Component</a> for more information.</p> |
| </li> |
| <li> |
| <p><strong>Refresh</strong>: The "Refresh" button allows the user to refresh the information displayed without closing the dialog and opening it |
| again. The time at which the information was last refreshed is shown just to the right of the "Refresh" button. The information |
| on the page is not automatically refreshed.</p> |
| </li> |
| <li> |
| <p><strong>Filter</strong>: The Filter element allows users to filter the contents of the Summary table by typing in all or part of some criteria, |
| such as a Processor Type or Processor Name. The types of filters available differ according to the selected tab. For instance, |
| if viewing the Processor tab, the user is able to filter by name or by type. When viewing the Connections tab, the user |
| is able to filter by source, by name, or by destination name. The filter is automatically applied when the contents |
| of the text box are changed. Below the text box is an indicator of how many entries in the table match the filter and how many |
| entries exist in the table.</p> |
| </li> |
| <li> |
| <p><strong>Pop-Out</strong>: When monitoring a flow, it is helpful to be able to open the Summary table in a separate browser tab or window. The |
| "Pop Out" button, next to the "Close" button, will cause the entire Summary dialog to be opened in a new browser tab or window |
| (depending on the configuration of the browser). Once the page is "popped out", the dialog is closed in the original |
| browser tab/window. In the new tab/window, the "Pop Out" button and the "Go To" button will no longer be available.</p> |
| </li> |
| <li> |
| <p><strong>System Diagnostics</strong>: The System Diagnostics window provides information about how the system is performing with respect to |
| system resource utilization. While this is intended mostly for administrators, it is provided in this view because it |
| does provide a summary of the system. This dialog shows information such as CPU utilization, how full the disks are, |
| and Java-specific metrics, such as memory size and utilization, as well as Garbage Collection information.</p> |
| </li> |
| </ul> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="Status_History"><a class="anchor" href="user-guide.html#Status_History"></a>Historical Statistics of a Component</h3> |
| <div class="paragraph"> |
| <p>While the Summary table and the canvas show numeric statistics pertaining to the performance of a component over the |
| past five minutes, it is often useful to have a view of historical statistics as well. This information is available |
| by right-clicking on a component and choosing the "Status History" menu option or by clicking on the Status History in the Summary page (see <a href="user-guide.html#Summary_Page">Summary Page</a> |
| for more information).</p> |
| </div> |
| <div class="paragraph"> |
| <p>The amount of historical information that is stored is configurable in the NiFi properties but defaults to <code>24 hours</code>. For specific |
| configuration information reference the Component Status Repository of the <a href="administration-guide.html">System Administrator’s Guide</a>. |
| When the Status History dialog is opened, it provides a graph of historical statistics:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/stats-history.png" alt="Status History"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The left-hand side of the dialog provides information about the component that the stats are for, as well as a textual |
| representation of the statistics being graphed. The following information is provided on the left-hand side:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Id</strong>: The ID of the component for which the stats are being shown.</p> |
| </li> |
| <li> |
| <p><strong>Group Id</strong>: The ID of the Process Group in which the component resides.</p> |
| </li> |
| <li> |
| <p><strong>Name</strong>: The Name of the Component for which the stats are being shown.</p> |
| </li> |
| <li> |
| <p><strong>Component-Specific Entries</strong>: Information is shown for each different type of component. For example, for a Processor, |
| the type of Processor is displayed. For a Connection, the source and destination names and IDs are shown.</p> |
| </li> |
| <li> |
| <p><strong>Start</strong>: The earliest time shown on the graph.</p> |
| </li> |
| <li> |
| <p><strong>End</strong>: The latest time shown on the graph.</p> |
| </li> |
| <li> |
| <p><strong>Min/Max/Mean</strong>: The minimum, maximum, and mean (arithmetic mean, or average) values are shown. These values are based |
| only on the range of time selected, if any time range is selected. If this instance of NiFi is clustered, these values |
| are shown for the cluster as a whole, as well as each individual node. In a clustered environment, each node is shown |
| in a different color. This also serves as the graph’s legend, showing the color of each node that is shown in the graph. |
| Hovering the mouse over the Cluster or one of the nodes in the legend will also make the corresponding node bold in the graph.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>The right-hand side of the dialog provides a drop-down list of the different types of metrics to render in the graphs below. |
| The top graph is larger so as to provide an easier-to-read rendering of the information. In the bottom-right corner of |
| this graph is a small handle ( |
| <span class="image"><img src="images/iconResize.png" alt="Resize"></span> |
| ) that can be dragged to resize the graph. The blank areas of the dialog can also be dragged around |
| to move the entire dialog.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The bottom graph is much shorter and provides the ability to select a time range. Selecting a time range here will |
| cause the top graph to show only the time range selected, but in a more detailed manner. Additionally, this will cause the |
| Min/Max/Mean values on the left-hand side to be recalculated. Once a selection has been created by dragging a |
| rectangle over the graph, double-clicking on the selected portion will cause the selection to fully expand in the |
| vertical direction (i.e., it will select all values in this time range). Clicking on the bottom graph without dragging |
| will remove the selection.</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="versioning_dataflow"><a class="anchor" href="user-guide.html#versioning_dataflow"></a>Versioning a DataFlow</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>When NiFi is connected to a NiFi Registry, dataflows can be version controlled on the process group level. For more information about NiFi Registry usage and configuration, see the documentation at <a href="https://nifi.apache.org/docs/nifi-registry-docs/index.html" target="_blank" rel="noopener">https://nifi.apache.org/docs/nifi-registry-docs/index.html</a>.</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="connecting-to-a-nifi-registry"><a class="anchor" href="user-guide.html#connecting-to-a-nifi-registry"></a>Connecting to a NiFi Registry</h3> |
| <div class="paragraph"> |
| <p>To connect NiFi to a Registry, select Controller Settings from the Global Menu.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/controller-settings-selection.png" alt="Global Menu - Controller Settings"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>This displays the NiFi Settings window. Select the Registry Clients tab and click the <code>+</code> button in the upper-right corner to register a new Registry Client.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/registry-clients-tab.png" alt="Registry Clients Tab"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>In the Add Registry Client window, provide a name, select type and add a description (if desired).</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/add-registry-client.png" alt="Add Registry Client Dialog"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Click "Add".</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/registry-client-added.png" alt="Registry Client Added"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Once a Registry Client has been added, the DFM may configure it by clicking the "Edit" button (<span class="image"><img src="images/iconEdit.png" alt="Edit Button"></span>) in the far-right column. When the DFM clicks the "Edit" button, an Edit Registry Client window opens. It has two tabs: Settings and Properties. The Settings tab provides a place for the DFM to edit the Registry Client name. It also lists the UUID, Type, and provides an open-text field to edit or add a Description.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/configure-registry-client-settings.png" alt="Registry Client Settings"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The Properties tab lists the various properties that may be configured for the client. The DFM may hover the mouse over the question mark icons to see more information about each property.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/configure-registry-client-properties.png" alt="Configure Registry Client Properties"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>After configuring a Registry Client, click "Update" to save the configuration and close the window, or click "Cancel" to discard the changes and close the window.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| Versioned flows are stored and organized in registry buckets. Bucket Policies and Special Privileges configured by the registry administrator determine which buckets a user can import versioned flows from and which buckets a user can save versioned flows to. Information on Bucket Policies and Special Privileges can be found in the NiFi Registry User Guide (<a href="https://nifi.apache.org/docs/nifi-registry-docs/html/user-guide.html" target="_blank" rel="noopener">https://nifi.apache.org/docs/nifi-registry-docs/html/user-guide.html</a>). |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="version_states"><a class="anchor" href="user-guide.html#version_states"></a>Version States</h3> |
| <div class="paragraph"> |
| <p>Versioned process groups exist in the following states:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><span class="image"><img src="images/iconUpToDate.png" alt="Up to date"></span> |
| <strong>Up to date</strong>: The flow version is the latest.</p> |
| </li> |
| <li> |
| <p><span class="image"><img src="images/iconLocallyModified.png" alt="Locally Modified"></span> |
| <strong>Locally modified</strong>: Local changes have been made.</p> |
| </li> |
| <li> |
| <p><span class="image"><img src="images/iconStale.png" alt="Stale"></span> |
| <strong>Stale</strong>: A newer version of the flow is available.</p> |
| </li> |
| <li> |
| <p><span class="image"><img src="images/iconLocallyModifiedStale.png" alt="Locally Modified and Stale"></span> |
| <strong>Locally modified and stale</strong>: Local changes have been made and a newer version of the flow is available.</p> |
| </li> |
| <li> |
| <p><span class="image"><img src="images/iconSyncFailure.png" alt="Sync Failure"></span> |
| <strong>Sync failure</strong>: Unable to synchronize the flow with the registry.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>Version state information is displayed:</p> |
| </div> |
| <div class="olist arabic"> |
| <ol class="arabic"> |
| <li> |
| <p>Next to the process group name, for the versioned process group itself. Hovering over the state icon displays additional information about the versioned flow.</p> |
| </li> |
| <li> |
| <p>At the bottom of a process group, for the versioned flows contained in the process group.</p> |
| </li> |
| <li> |
| <p>In the Status Bar at the top of the UI, for the versioned flows contained in the root process group.</p> |
| </li> |
| </ol> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/version-states-display.png" alt="Version States Displayed"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Version state information is also shown in the "Process Groups" tab of the Summary Page.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/version-state-summary-page.png" alt="Version State in Summary Page"> |
| </div> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| To see the most recent version states, it may be necessary to right-click on the NiFi canvas and select 'Refresh' from the context menu. |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="import-a-versioned-flow"><a class="anchor" href="user-guide.html#import-a-versioned-flow"></a>Import a Versioned Flow</h3> |
| <div class="paragraph"> |
| <p>When a NiFi instance is connected to a registry, an "Import" link will appear in the Add Process Group dialog.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/add-process-group-import.png" alt="Import Process Group"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Selecting the link will open the Import Version dialog.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/import-version-dialog.png" alt="Import Version Dialog"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Connected registries will appear as options in the Registry drop-down menu. For the chosen Registry, buckets the user has access to will appear as options in the Bucket drop-down menu. The names of the flows in the chosen bucket will appear as options in the Name drop-down menu. Select the desired version of the flow to import and select "Import" for the dataflow to be placed on the canvas.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/versioned-flow-imported.png" alt="Versioned Flow Imported"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Since the version imported in this example is the latest version (MySQL CDC, Version 3), the state of the versioned process group is "Up to date" (<span class="image"><img src="images/iconUpToDate.png" alt="Up To Date Icon"></span>). If the version imported had been an older version, the state would be "Stale" (<span class="image"><img src="images/iconStale.png" alt="Stale Icon"></span>).</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="start-version-control"><a class="anchor" href="user-guide.html#start-version-control"></a>Start Version Control</h3> |
| <div class="paragraph"> |
| <p>To place a process group under version control, right-click on the process group and in the context menu, select "Version→Start version control".</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/start-version-control.png" alt="Start Version Control"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>In the Save Flow Version window, select a Registry and Bucket and enter a Name for the Flow. If desired, add content for the Description and Comment fields.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/save-flow-version-dialog.png" alt="Save Flow Version Dialog"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Select Save and Version 1 of the flow is saved.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/versioned-process-group.png" alt="Versioned Process Group"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>As the first and latest version of the flow, the state of the versioned process group is "Up to date" (<span class="image"><img src="images/iconUpToDate.png" alt="Up To Date Icon"></span>).</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| The root process group can not be placed under version control. |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="managing_local_changes"><a class="anchor" href="user-guide.html#managing_local_changes"></a>Managing Local Changes</h3> |
| <div class="paragraph"> |
| <p>When changes are made to a versioned process group, the state of the component updates to "Locally modified" (<span class="image"><img src="images/iconLocallyModified.png" alt="Locally Modified Icon"></span>). The DFM can show, revert or commit the local changes. These options are available for selection in the context menu when right-clicking on the process group:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/local-changes-pg-selected.png" alt="Local Changes PG Selected"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>or when right-clicking on the canvas inside the process group:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/local-changes-pg-inside.png" alt="Local Changes Inside PG"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The following actions are not considered local changes:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>stopping/starting processors</p> |
| </li> |
| <li> |
| <p>modifying sensitive property values</p> |
| </li> |
| <li> |
| <p>modifying remote process group URLs</p> |
| </li> |
| <li> |
| <p>updating a processor that was referencing a non-existent controller service to reference an externally available controller service</p> |
| </li> |
| <li> |
| <p>assigning, creating, modifying or deleting parameter contexts</p> |
| </li> |
| <li> |
| <p>creating, modifying or deleting variables</p> |
| </li> |
| </ul> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| Assigning or creating a parameter context does not trigger a local change because assigning or creating a parameter context on its own has not changed anything about what the flow processes. A component will have to be created or modified that uses a parameter in the parameter context, which will trigger a local change. Modifying a parameter context does not trigger a local change because parameters are intended to be different in each environment. When a versioned flow is imported, it is assumed there is a one-time operation required to set those parameters specific for the given environment. Deleting a parameter context does not trigger a local change because any components that reference parameters in that parameter context will need need to be modified, which will trigger a local change. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| Creating a variable does not trigger a local change because creating a variable on its own has not changed anything about what the flow processes. A component will have to be created or modified that uses the new variable, which will trigger a local change. Modifying a variable does not trigger a local change because variable values are intended to be different in each environment. When a versioned flow is imported, it is assumed there is a one-time operation required to set those variables specific for the given environment. Deleting a variable does not trigger a local change because the component that references that variable will need need to be modified, which will trigger a local change. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="admonitionblock warning"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-warning" title="Warning"></i> |
| </td> |
| <td class="content"> |
| Variables do not support sensitive values and will be included when versioning a Process Group. Variables are still supported for compatibility purposes but do not have the same power as Parameters such as support for sensitive properties and more granular control over who can create, modify or use them. Variables will be removed in a future release. As a result, it is highly recommended to switch to Parameters. |
| </td> |
| </tr> |
| </table> |
| </div> |
| <div class="sect3"> |
| <h4 id="show-local-changes"><a class="anchor" href="user-guide.html#show-local-changes"></a>Show Local Changes</h4> |
| <div class="paragraph"> |
| <p>The local changes made to a versioned process group can be viewed in the Show Local Changes dialog by selecting "Version→Show local changes" from the context menu.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/show-local-changes-dialog.png" alt="Show Local Changes Dialog"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>You can navigate to a component by selecting the "Go To" icon (<span class="image"><img src="images/iconGoTo.png" alt="Go To"></span>) in its row.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| As described in the <a href="user-guide.html#managing_local_changes">Managing Local Changes</a> section, there are exceptions to which actions are reviewable local changes. Additionally, multiple changes to the same property will only appear as one change in the list as the changes are determined by diffing the current state of the process group and the saved version of the process group noted in the Show Local Changes dialog. |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="revert-local-changes"><a class="anchor" href="user-guide.html#revert-local-changes"></a>Revert Local Changes</h4> |
| <div class="paragraph"> |
| <p>Revert the local changes made to a versioned process group by selecting "Version→Revert local changes" from the context menu. The Revert Local Changes dialog displays a list of the local changes for the DFM to review and consider prior to initiating the revert. Select "Revert" to remove all changes.</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/revert-local-changes-dialog.png" alt="Revert Local Changes Dialog"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>You can navigate to a component by selecting the "Go To" icon (<span class="image"><img src="images/iconGoTo.png" alt="Go To"></span>) in its row.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| As described in the <a href="user-guide.html#managing_local_changes">Managing Local Changes</a> section, there are exceptions to which actions are revertible local changes. Additionally, multiple changes to the same property will only appear as one change in the list as the changes are determined by diffing the current state of the process group and the saved version of the process group noted in the Revert Local Changes dialog. |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="commit-local-changes"><a class="anchor" href="user-guide.html#commit-local-changes"></a>Commit Local Changes</h4> |
| <div class="paragraph"> |
| <p>To commit and save a flow version, select "Version→Commit local changes" from the context menu. In the Save Flow Version dialog, add comments if desired and select "Save".</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/save-flow-version-commit.png" alt="Save Flow Version Commit"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Local changes can not be committed if the version that has been modified is not the latest version. In this scenario, the version state is "Locally modified and stale" (<span class="image"><img src="images/iconLocallyModifiedStale.png" alt="Locally Modified and Stale"></span>).</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="change-version"><a class="anchor" href="user-guide.html#change-version"></a>Change Version</h3> |
| <div class="paragraph"> |
| <p>To change the version of a flow, right-click on the versioned process group and select "Version→Change version".</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/change-version.png" alt="Change Version"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>In the Change Version dialog, select the desired version and select "Change":</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/change-version-dialog.png" alt="Change Version Dialog"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The version of the flow is changed:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/flow-version-changed.png" alt="Flow Version Changed"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>In the example shown, the versioned flow is upgraded from an older to the newer latest version. However, a versioned flow can also be rollbacked to an older version.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| For "Change version" to be an available selection, local changes to the process group need to be reverted. |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="stop-version-control"><a class="anchor" href="user-guide.html#stop-version-control"></a>Stop Version Control</h3> |
| <div class="paragraph"> |
| <p>To stop version control on a flow, right-click on the versioned process group and select "Version→Stop version control":</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/stop-version-control.png" alt="Stop Version Control"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>In the Stop Version Control dialog, select "Disconnect".</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/stop-version-control-dialog.png" alt="Stop Version Control Dialog"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The removal of the process group from version control is confirmed.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/disconnect-dialog.png" alt="Disconnect Confirmation Dialog"></span></p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/process-group-version-control-stopped.png" alt="Version Control Stopped on Process Group"> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="nested-versioned-flows"><a class="anchor" href="user-guide.html#nested-versioned-flows"></a>Nested Versioned Flows</h3> |
| <div class="paragraph"> |
| <p>A versioned process group can contain other versioned process groups. However, local changes to a parent process group cannot be reverted or saved if it contains a child process group that also has local changes. The child process group must first be reverted or have its changes committed for those actions to be performed on the parent process group.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="parameters-in-versioned-flows"><a class="anchor" href="user-guide.html#parameters-in-versioned-flows"></a>Parameters in Versioned Flows</h3> |
| <div class="paragraph"> |
| <p>When exporting a versioned flow to a Flow Registry, the name of the Parameter Context is sent for each process group that is stored. The Parameters (names, descriptions, values, whether or not sensitive) are also stored with the flow. However, Sensitive Parameter values are not stored.</p> |
| </div> |
| <div class="paragraph"> |
| <p>When a versioned flow is imported, a Parameter Context will be created for each one that doesn’t already exist in the NiFi instance. When importing a versioned flow from Flow Registry, if NiFi has a Parameter Context with the same name, the values are merged, as described in the following example:</p> |
| </div> |
| <div class="paragraph"> |
| <p>A flow has a Parameter Context "PC1" with the following parameters:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/pc1_parameters.png" alt="PC1 Parameters"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The flow is exported and saved to the Flow Registry.</p> |
| </div> |
| <div class="paragraph"> |
| <p>A NiFi instance has a Parameter Context also named "PC1" with the following parameters:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/nifi_pc1_parameters.png" alt="NiFi PC1 Parameters"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The versioned flow is imported into the NiFi instance. The Parameter Context "PC1" now has the following parameters:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/nifi_pc1_parameters_new.png" alt="New NiFi PC1 Parameters"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The "Letters" parameter did not exist in the NiFi instance and was added. The "Numbers" parameter existed in both the versioned flow and NiFi instance with identical values, so no changes were made. "Password" is a sensitive Parameter missing from the NiFi instance, so it was added but with no value. "Port" existed in the NiFi instance with a different value than the versioned flow, so its value remained unchanged.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Parameter Contexts are handled similarly when a flow version is changed. Consider the following two examples:</p> |
| </div> |
| <div class="paragraph"> |
| <p>If the versioned flow referenced earlier is changed to another version (Version 2) and Version 2’s Parameter Context "PC1" has a "Colors" Parameter, "Colors" will be added to "PC1" in the NiFi instance.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Version 1 of a flow does not have a Parameter Context associated with it. A new version (Version 2) does. When the flow is changed from Version 1 to Version 2, one of the following occurs:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>A new Parameter Context is created if it does not already exist</p> |
| </li> |
| <li> |
| <p>An existing Parameter Context is assigned (by name) to the Process Group and the values of the Parameter Contexts are merged</p> |
| </li> |
| </ul> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="Variables_in_Versioned_Flows"><a class="anchor" href="user-guide.html#Variables_in_Versioned_Flows"></a>Variables in Versioned Flows</h3> |
| <div class="paragraph"> |
| <p>Variables are included when a process group is placed under version control. If a versioned flow is imported that references a variable not defined in the versioned process group, the reference is maintained if the variable exists. If the referenced variable does not exist, a copy of the variable will be defined in the process group. To illustrate, assume the variable “RPG_Var" is defined in the root process group:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/rpg-variable.png" alt="Root Process Group Defined Variable"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>A process group PG1 is created:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/PG1_process_group.png" alt="PG1 Process Group"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The GetFile processor in PG1 references the variable "RPG_Var":</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/PG1_variable_ref_1.png" alt="PG1 References RPG Variable"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>PG1 is saved as a versioned flow:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/PG1_versioned_flow.png" alt="PG1 Versioned Flow"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>If PG1 versioned flow is imported into this same NiFi instance:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/PG1_imported_same.png" alt="PG1 Imported to Same NiFi"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>the added GetFile processor will also reference the "RPG_Var" variable that exists in the root process group:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/PG1_variable_ref_2.png" alt="Both PG1 Reference RPG Variable"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>If PG1 versioned flow is imported into a different NiFi instance where "RPG_Var" does not exist:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/PG1_imported_diff.png" alt="PG1 Imported to Different NiFi"> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>A "RPG_Var" variable is created in the PG1 process group:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/PG1_variable_ref_PG.png" alt="PG1 References PG Variable Copy"> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="Restricted_Components_in_Versioned_Flows"><a class="anchor" href="user-guide.html#Restricted_Components_in_Versioned_Flows"></a>Restricted Components in Versioned Flows</h3> |
| <div class="paragraph"> |
| <p>To import a versioned flow or revert local changes in a versioned flow, a user must have access to all the components in the versioned flow. As such, it is recommended that restricted components are created at the root process group level if they are to be utilized in versioned flows. Let’s walk through some examples to illustrate the benefits of this configuration. Assume the following:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>There are two users, "sys_admin" and "test_user" who have access to both view and modify the root process group.</p> |
| </li> |
| <li> |
| <p>"sys_admin" has access to all restricted components.</p> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/sys_admin-restricted-component-access-policy.png" alt="Sys_admin Restricted Component Access Policy"> |
| </div> |
| </div> |
| </li> |
| <li> |
| <p>"test_user" has access to restricted components requiring 'read filesystem' and 'write filesystem'.</p> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/test_user-restricted-component-read-filesystem.png" alt="Test_user Restricted Component Read Filesystem"> |
| </div> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="images/test_user-restricted-component-write-filesystem.png" alt="Test_user Restricted Component Write Filesystem"> |
| </div> |
| </div> |
| </li> |
| </ul> |
| </div> |
| <div class="sect3"> |
| <h4 id="restricted-controller-service-created-in-root-process-group"><a class="anchor" href="user-guide.html#restricted-controller-service-created-in-root-process-group"></a>Restricted Controller Service Created in Root Process Group</h4> |
| <div class="paragraph"> |
| <p>In this first example, sys_admin creates a KeytabCredentialsService controller service at the root process group level.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/keytabCredentialsService-rpg.png" alt="KeytabCredentialsService Controller Service RPG Level"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>KeytabCredentialService controller service is a restricted component that requires 'access keytab' permissions:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/keytabcredentialsservice-permissions.png" alt="KeytabCredentialService Required Permissions"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Sys_admin creates a process group ABC containing a flow with GetFile and PutHDFS processors:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/abc-restricted-component-flow.png" alt="Restricted Component Flow"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>GetFile processor is a restricted component that requires 'write filesystem' and 'read filesystem' permissions:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/getfile-permissions.png" alt="GetFile Required Permissions"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>PutHDFS is a restricted component that requires 'write filesystem' permissions:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/puthdfs-permissions.png" alt="PutHDFS Required Permissions"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The PutHDFS processor is configured to use the root process group level KeytabCredentialsService controller service:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/puthdfs-properties.png" alt="PutHDFS Properties"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Sys_admin saves the process group as a versioned flow:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/abc-versioned-flow.png" alt="ABC Versioned Flow"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Test_user changes the flow by removing the KeytabCredentialsService controller service:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/puthdfs-no-kerberosCS.png" alt="PutHDFS No Kerberos CS"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>If test_user chooses to revert this change:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/test_user-revert-local-changes.png" alt=""Test_user Revert Local Changes"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>the revert is successful:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/revert-success.png" alt="Revert Local Changes Successful"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Additionally, if test_user chooses to import the ABC versioned flow:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/test_user-import-abc-flow.png" alt="Test_user Import Flow"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The import is successful:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/test_user-import-success.png" alt="Test_user Import Successful"></span></p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="restricted-controller-service-created-in-process-group"><a class="anchor" href="user-guide.html#restricted-controller-service-created-in-process-group"></a>Restricted Controller Service Created in Process Group</h4> |
| <div class="paragraph"> |
| <p>Now, consider a second scenario where the controller service is created on the process group level.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Sys_admin creates a process group XYZ:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/xyz-process-group.png" alt="XYZ Process Group"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Sys_admin creates a KeytabCredentialsService controller service at the process group level:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/keytabCredentialsService-pg.png" alt="KeytabCredentialsService Controller Service PG Level"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The same GetFile and PutHDFS flow is created in the process group:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/xyz-flow.png" alt="XYZ Versioned Flow"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>However, PutHDFS now references the process group level controller service:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/puthdfs-properties_2.png" alt="PutHDFS Properties"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Sys_admin saves the process group as a versioned flow.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Test_user changes the flow by removing the KeytabCredentialsService controller service. However, with this configuration, if test_user attempts to revert this change:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/test_user-revert-local-changes-2.png" alt="Test_user Revert Local Changes"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>the revert is unsuccessful because test_user does not have the 'access keytab' permissions required by the KeytabCredentialService controller service:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/revert-failure.png" alt="Revert Local Changes Fails"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Similarly, if test_user tries to import the XYZ versioned flow:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/test_user-import-xyz-flow.png" alt="Test_user Import Flow"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The import fails:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/import-xyz-flow-fails.png" alt="XYZ Import Fails"></span></p> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="templates"><a class="anchor" href="user-guide.html#templates"></a>Templates</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>DFMs have the ability to build very large and complex DataFlows using NiFi. This is achieved |
| by using the basic components: Processor, Funnel, Input/Output Port, Process Group, and Remote Process Group. These |
| can be thought of as the most basic building blocks for constructing a DataFlow. At times, though, using these |
| small building blocks can become tedious if the same logic needs to be repeated several times.</p> |
| </div> |
| <div class="paragraph"> |
| <p>To solve this issue, NiFi provides the concept of a Template. A Template is a way of combining these basic building |
| blocks into larger building blocks. Once a DataFlow has been created, parts of it can be formed into a Template. |
| This Template can then be dragged onto the canvas, or can be exported as an XML file and shared with others. Templates |
| received from others can then be imported into an instance of NiFi and dragged onto the canvas.</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="Create_Template"><a class="anchor" href="user-guide.html#Create_Template"></a>Creating a Template</h3> |
| <div class="paragraph"> |
| <p>To create a Template, select the components that are to be a part of the template, and then click the |
| "Create Template" (<span class="image"><img src="images/iconNewTemplate.png" alt="Create Template"></span>) button in the Operate Palette (See <a href="user-guide.html#User_Interface">NiFi User Interface</a> for more information on the Operate Palette).</p> |
| </div> |
| <div class="paragraph"> |
| <p>Clicking this button without selecting anything will create a Template that contains all of the contents of the |
| current Process Group. This means that creating a Template with nothing selected while on the Root Process Group |
| will create a single Template that contains the entire flow.</p> |
| </div> |
| <div class="paragraph"> |
| <p>After clicking this button, the user is prompted to provide a name and an optional description for the template. |
| Each template must have a unique name. After entering the name and optional description, clicking the "Create" button |
| will generate the template and notify the user that the template was successfully created, or provide an appropriate |
| error message if unable to create the template for some reason.</p> |
| </div> |
| <div class="admonitionblock note"> |
| <table> |
| <tr> |
| <td class="icon"> |
| <i class="fa icon-note" title="Note"></i> |
| </td> |
| <td class="content"> |
| It is important to note that if any Processor that is Templated has a sensitive property (such as a password), the value of that |
| sensitive property is not included in the Template. As a result, when dragging the Template onto the canvas, newly |
| created Processors may not be valid if they are missing values for their sensitive properties. Additionally, any |
| Connection that was selected when making the Template is not included in the Template if either the source or the |
| destination of the Connection is not also included in the Template. |
| </td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="Import_Template"><a class="anchor" href="user-guide.html#Import_Template"></a>Importing a Template</h3> |
| <div class="paragraph"> |
| <p>After receiving a Template that has been exported from another NiFi, the first step needed to use the template is to import |
| the template into this instance of NiFi. You may import templates into any Process Group where you have the appropriate |
| authorization.</p> |
| </div> |
| <div class="paragraph"> |
| <p>From the Operate Palette, click the "Upload Template" (<span class="image"><img src="images/iconUploadTemplate.png" alt="Upload Template"></span>) button (see <a href="user-guide.html#User_Interface">NiFi User Interface</a> for more information on the Operate Palette). This will display the Upload Template |
| dialog. Click the find icon and use the File Selection dialog to choose which template file to upload. |
| Select the file and click Open. |
| Clicking the "Upload" button will attempt to import the Template into this instance of NiFi. |
| The Upload Template dialog will update to show "Success" or an error message if there was a problem importing the template.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="instantiating-a-template"><a class="anchor" href="user-guide.html#instantiating-a-template"></a>Instantiating a Template</h3> |
| <div class="paragraph"> |
| <p>Once a Template has been created (see <a href="user-guide.html#Create_Template">Creating a Template</a>) or imported (see <a href="user-guide.html#Import_Template">Importing a Template</a>), it is ready to be |
| instantiated, or added to the canvas. This is accomplished by dragging the Template icon ( |
| <span class="image"><img src="images/iconTemplate.png" alt="Template"></span> |
| ) from the Components Toolbar (see <a href="user-guide.html#User_Interface">NiFi User Interface</a>) onto the canvas.</p> |
| </div> |
| <div class="paragraph"> |
| <p>This will present a dialog to choose which Template to add to the canvas. After choosing the Template to add, simply |
| click the "Add" button. The Template will be added to the canvas with the upper-left-hand side of the Template |
| being placed wherever the user dropped the Template icon.</p> |
| </div> |
| <div class="paragraph"> |
| <p>This leaves the contents of the newly instantiated Template selected. If there was a mistake, and this Template is no |
| longer wanted, it may be deleted.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="Manage_Templates"><a class="anchor" href="user-guide.html#Manage_Templates"></a>Managing Templates</h3> |
| <div class="paragraph"> |
| <p>One of the most powerful features of NiFi Templates is the ability to easily export a Template to an XML file |
| and to import a Template that has already been exported. This provides a very simple mechanism for sharing parts |
| of a DataFlow with others. You can select Templates from the Global Menu (see <a href="user-guide.html#User_Interface">NiFi User Interface</a>) to open a dialog |
| that displays all of the Templates that are currently available, |
| filter the templates to see only those of interest, export, and delete Templates.</p> |
| </div> |
| <div class="sect3"> |
| <h4 id="Export_Template"><a class="anchor" href="user-guide.html#Export_Template"></a>Exporting a Template</h4> |
| <div class="paragraph"> |
| <p>Once a Template has been created, it can be shared with others in the Template Management page. |
| To export a Template, locate the Template in the table. The Filter in the top-right corner |
| can be used to help find the appropriate Template if several are available. Then click the "Download" button (<span class="image"><img src="images/iconDownloadTemplate.png" alt="Export"></span>). This will download the template as an XML file to your computer. This XML file can then be sent to others and imported into other instances of NiFi (see <a href="user-guide.html#Import_Template">Importing a Template</a>).</p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="removing-a-template"><a class="anchor" href="user-guide.html#removing-a-template"></a>Removing a Template</h4> |
| <div class="paragraph"> |
| <p>Once it is decided that a Template is no longer needed, it can be easily removed from the Template Management page. To delete a Template, locate it in the table (the Filter in the top-right corner |
| may be used to find the appropriate Template if several are available) and click the "Delete" button ( |
| <span class="image"><img src="images/iconDelete.png" alt="Delete"></span> |
| ). This will prompt for confirmation. After confirming the deletion, the Template will be removed from this table |
| and will no longer be available to add to the canvas.</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="data_provenance"><a class="anchor" href="user-guide.html#data_provenance"></a>Data Provenance</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>While monitoring a dataflow, users often need a way to determine what happened to a particular data object (FlowFile). |
| NiFi’s Data Provenance page provides that information. Because NiFi records and indexes data provenance details |
| as objects flow through the system, users may perform searches, conduct troubleshooting and evaluate things |
| like dataflow compliance and optimization in real time. By default, NiFi updates this information every five minutes, but that |
| is configurable.</p> |
| </div> |
| <div class="paragraph"> |
| <p>To access the Data Provenance page, select "Data Provenance" from the Global Menu. This opens a dialog window that allows the user to see the most recent Data Provenance information available, |
| search the information for specific items, and filter the search results. It is also possible to open additional dialog windows to see event details, |
| replay data at any point within the dataflow, and see a graphical representation of the data’s lineage, or path through the flow. |
| (These features are described in depth below.)</p> |
| </div> |
| <div class="paragraph"> |
| <p>When authorization is enabled, accessessing Data Provenance information requires the 'query provenance' Global Policy as well as the 'view provenance' |
| Component Policy for the component which generated the event. In addition, access to event details which include FlowFile attributes and content require |
| the 'view the data' Component Policy for the component which generated the event.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/provenance-annotated.png" alt="Provenance Table"></span></p> |
| </div> |
| <div class="sect2"> |
| <h3 id="provenance_events"><a class="anchor" href="user-guide.html#provenance_events"></a>Provenance Events</h3> |
| <div class="paragraph"> |
| <p>Each point in a dataflow where a FlowFile is processed in some way is considered a 'provenance event'. Various types of provenance |
| events occur, depending on the dataflow design. For example, when data is brought into the flow, a RECEIVE event occurs, and when |
| data is sent out of the flow, a SEND event occurs. Other types of processing events may occur, such as if the data is cloned (CLONE event), routed (ROUTE event), modified (CONTENT_MODIFIED or ATTRIBUTES_MODIFIED event), |
| split (FORK event), combined with other data objects (JOIN event), and ultimately removed from the flow (DROP event).</p> |
| </div> |
| <div class="paragraph"> |
| <p>The provenance event types are:</p> |
| </div> |
| <table class="tableblock frame-all grid-all stretch"> |
| <colgroup> |
| <col style="width: 50%;"> |
| <col style="width: 50%;"> |
| </colgroup> |
| <thead> |
| <tr> |
| <th class="tableblock halign-left valign-top">Provenance Event</th> |
| <th class="tableblock halign-left valign-top">Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">ADDINFO</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates a provenance event when additional information such as a new linkage to a new URI or UUID is added</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">ATTRIBUTES_MODIFIED</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates that a FlowFile’s attributes were modified in some way</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">CLONE</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates that a FlowFile is an exact duplicate of its parent FlowFile</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">CONTENT_MODIFIED</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates that a FlowFile’s content was modified in some way</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">CREATE</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates that a FlowFile was generated from data that was not received from a remote system or external process</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">DOWNLOAD</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates that the contents of a FlowFile were downloaded by a user or external entity</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">DROP</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates a provenance event for the conclusion of an object’s life for some reason other than object expiration</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">EXPIRE</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates a provenance event for the conclusion of an object’s life due to the object not being processed in a timely manner</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">FETCH</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates that the contents of a FlowFile were overwritten using the contents of some external resource</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">FORK</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates that one or more FlowFiles were derived from a parent FlowFile</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">JOIN</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates that a single FlowFile is derived from joining together multiple parent FlowFiles</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">RECEIVE</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates a provenance event for receiving data from an external process</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">REMOTE_INVOCATION</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates that a remote invocation was requested to an external endpoint (e.g., deleting a remote resource)</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">REPLAY</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates a provenance event for replaying a FlowFile</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">ROUTE</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates that a FlowFile was routed to a specified relationship and provides information about why the FlowFile was routed to this relationship</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">SEND</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates a provenance event for sending data to an external process</p></td> |
| </tr> |
| <tr> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">UNKNOWN</p></td> |
| <td class="tableblock halign-left valign-top"><p class="tableblock">Indicates that the type of provenance event is unknown because the user who is attempting to access the event is not authorized to know the type</p></td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| <div class="sect2"> |
| <h3 id="searching-for-events"><a class="anchor" href="user-guide.html#searching-for-events"></a>Searching for Events</h3> |
| <div class="paragraph"> |
| <p>One of the most common tasks performed in the Data Provenance page is a search for a given FlowFile to determine what happened to it. To do this, |
| click the "Search" button in the upper-right corner of the Data Provenance page. This opens a dialog window with parameters that the user can |
| define for the search. The parameters include the processing event of interest, distinguishing characteristics about the FlowFile or the component that produced the event, the timeframe within which to search, and the size of the FlowFile.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/search-events.png" alt="Search Events"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>For example, to determine if a particular FlowFile was received, search for an Event Type of "RECEIVE" and include an |
| identifier for the FlowFile, such as its uuid or filename. The asterisk (*) may be used as a wildcard for any number of characters. |
| So, to determine whether a FlowFile with "ABC" anywhere in its filename was received at any time on July 29, 2016, the search shown in the following |
| image could be performed:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/search-receive-event-abc.png" alt="Search for RECEIVE Event"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>If all filenames that do not have "ABC" anywhere in the filename is desired, then click the checkbox with the label "Exclude from search results" beneath |
| this entry before performing the search.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="event_details"><a class="anchor" href="user-guide.html#event_details"></a>Details of an Event</h3> |
| <div class="paragraph"> |
| <p>In the far-left column of the Data Provenance page, there is a "View Details" icon for each event (<span class="image"><img src="images/iconDetails.png" alt="Details"></span>). |
| Clicking this button opens a dialog window with three tabs: Details, Attributes, and Content.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/event-details.png" alt="Event Details" width="700"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The Details tab shows various details about the event, such as when it occurred, what type of event it was, and the component that produced the event. |
| The information that is displayed will vary according to the event type. This tab also shows information about the FlowFile that was processed. In |
| addition to the FlowFile’s UUID, which is displayed on the left side of the Details tab, the UUIDs of any parent or children FlowFiles that are related |
| to that FlowFile are displayed on the right side of the Details tab.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The Attributes tab shows the attributes that exist on the FlowFile as of that point in the flow. In order to see only the attributes that were modified as |
| a result of the processing event, the user may select the checkbox next to "Only show modified" in the upper-right corner of the Attributes tab.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/event-attributes.png" alt="Event Attributes" width="700"></span></p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="replay_flowfile"><a class="anchor" href="user-guide.html#replay_flowfile"></a>Replaying a FlowFile</h3> |
| <div class="paragraph"> |
| <p>A DFM may need to inspect a FlowFile’s content at some point in the dataflow to ensure that it is being processed as expected. And if it |
| is not being processed properly, the DFM may need to make adjustments to the dataflow and replay the FlowFile again. |
| This can be achieved from the Content tab of the View Details dialog window. The Content tab shows information |
| about the FlowFile’s content, such as its location in the Content Repository |
| and its size. In addition, it is here that the user may click the "Download" button to download a copy of the FlowFile’s content as it existed |
| at this point in the flow. The user may also click the "Submit" button to replay the FlowFile at this point in the flow. Upon clicking "Submit", |
| the FlowFile is sent to the connection feeding the component that produced this processing event.</p> |
| </div> |
| <div class="paragraph"> |
| <p>When a user is developing a dataflow, it can be very beneficial to have easy access to replaying a FlowFile, as well. For example, a user may configure |
| a Processor, run a FlowFile through it, and find that the configuration needs to be modified. The user can then update the configuration, and run the |
| same FlowFile through again to verify the results. In order to ease this process, the user can right-click on a Processor and choose the "Replay last event" |
| item. From here, the user can choose to either replay the last event from just the Primary Node or from all nodes.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/event-content.png" alt="Event Content" width="700"></span></p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="viewing-flowfile-lineage"><a class="anchor" href="user-guide.html#viewing-flowfile-lineage"></a>Viewing FlowFile Lineage</h3> |
| <div class="paragraph"> |
| <p>It is often useful to see a graphical representation of the lineage or path a FlowFile took within the dataflow. To see a FlowFile’s lineage, |
| click on the "Show Lineage" icon ( <span class="image"><img src="images/iconLineage.png" alt="Show Lineage" width="28"></span> ) in the far-right column |
| of the Data Provenance table. This opens a graph displaying the FlowFile ( <span class="image"><img src="images/lineage-flowfile.png" alt="FlowFile" width="32"></span> ) and the |
| various processing events that have occurred. The selected event will be highlighted in red. It is possible to right-click or double-click on any |
| event to see that event’s details (see <a href="user-guide.html#event_details">Details of an Event</a>). |
| To see how the lineage evolved over time, click the slider at the bottom-left of the window and move it to the left to see the state of the lineage at earlier stages in the dataflow.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/lineage-graph-annotated.png" alt="Lineage Graph" width="900"></span></p> |
| </div> |
| <div class="sect3"> |
| <h4 id="find-parents"><a class="anchor" href="user-guide.html#find-parents"></a>Find Parents</h4> |
| <div class="paragraph"> |
| <p>Sometimes, a user may need to track down the original FlowFile that another FlowFile was spawned from. For example, when a FORK or CLONE event occurs, NiFi keeps |
| track of the parent FlowFile that produced other FlowFiles, and it is possible to find that parent FlowFile in the Lineage. Right-click on the event in the |
| lineage graph and select "Find parents" from the context menu.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/find-parents.png" alt="Find Parents"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Once "Find parents" is selected, the graph is re-drawn to show the parent FlowFile and its lineage as well as the child and its lineage.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/parent-found.png" alt="Parent Found"></span></p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="expanding-an-event"><a class="anchor" href="user-guide.html#expanding-an-event"></a>Expanding an Event</h4> |
| <div class="paragraph"> |
| <p>In the same way that it is useful to find a parent FlowFile, the user may also want to determine what children were spawned from a given FlowFile. To do this, right-click on the event in the lineage graph and select "Expand" from the context menu.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/expand-event.png" alt="Expand Event"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>Once "Expand" is selected, the graph is re-drawn to show the children and their lineage.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="images/expanded-events.png" alt="Expanded Events"></span></p> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="writeahead-provenance"><a class="anchor" href="user-guide.html#writeahead-provenance"></a>Write Ahead Provenance Repository</h3> |
| <div class="paragraph"> |
| <p>By default, the Provenance Repository is implemented in a Persistent Provenance configuration. In Apache NiFi 1.2.0, the Write Ahead configuration was introduced to provide the same capabilities as Persistent Provenance, but with far better performance. Migrating to the Write Ahead configuration is easy to accomplish. Simply change the setting for the <code>nifi.provenance.repository.implementation</code> system property in the <em>nifi.properties</em> file from the default value of <code>org.apache.nifi.provenance.PersistentProvenanceRepository</code> to <code>org.apache.nifi.provenance.WriteAheadProvenanceRepository</code> and restart NiFi.</p> |
| </div> |
| <div class="paragraph"> |
| <p>However, to increase the chances of a successful migration consider the following factors and recommended actions.</p> |
| </div> |
| <div class="sect3"> |
| <h4 id="backwards-compatibility"><a class="anchor" href="user-guide.html#backwards-compatibility"></a>Backwards Compatibility</h4> |
| <div class="paragraph"> |
| <p>The <code>WriteAheadProvenanceRepository</code> can use the Provenance data stored by the <code>PersistentProvenanceRepository</code>. However, the <code>PersistentProvenanceRepository</code> may not be able to read the data written by the <code>WriteAheadProvenanceRepository</code>. Therefore, once the Provenance Repository is changed to use the <code>WriteAheadProvenanceRepository</code>, it cannot be changed back to the <code>PersistentProvenanceRepository</code> without first deleting the data in the Provenance Repository. It is therefore recommended that before changing the implementation to Write Ahead, ensure your version of NiFi is stable, in case an issue arises that requires the need to roll back to a previous version of NiFi that did not support the <code>WriteAheadProvenanceRepository</code>.</p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="older-existing-nifi-version"><a class="anchor" href="user-guide.html#older-existing-nifi-version"></a>Older Existing NiFi Version</h4> |
| <div class="paragraph"> |
| <p>If you are upgrading from an older version of NiFi to 1.2.0 or later, it is recommended that you do not change the provenance configuration to Write Ahead until you confirm your flows and environment are stable in 1.2.0 first. This reduces the number of variables in your upgrade and can simplify the debugging process if any issues arise.</p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="bootstrap-conf"><a class="anchor" href="user-guide.html#bootstrap-conf"></a>Bootstrap.conf</h4> |
| <div class="paragraph"> |
| <p>While better performance is achieved with the G1 garbage collector, Java 8 bugs may surface more frequently in the Write Ahead configuration. It is recommended that the following line is commented out in the <em>bootstrap.conf</em> file in the <code>conf</code> directory:</p> |
| </div> |
| <div class="literalblock"> |
| <div class="content"> |
| <pre>java.arg.13=-XX:+UseG1GC</pre> |
| </div> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="system-properties"><a class="anchor" href="user-guide.html#system-properties"></a>System Properties</h4> |
| <div class="paragraph"> |
| <p>Many of the same system properties are supported by both the Persistent and Write Ahead configurations, however the default values have been chosen for a Persistent Provenance configuration. The following exceptions and recommendations should be noted when changing to a Write Ahead configuration:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><code>nifi.provenance.repository.journal.count</code> is not relevant to a Write Ahead configuration</p> |
| </li> |
| <li> |
| <p><code>nifi.provenance.repository.concurrent.merge.threads</code> and <code>nifi.provenance.repository.warm.cache.frequency</code> are new properties. The default values of <code>2</code> for threads and blank for frequency (i.e., disabled) should remain for most installations.</p> |
| </li> |
| <li> |
| <p>Change the settings for <code>nifi.provenance.repository.max.storage.time</code> (default value of <code>24 hours</code>) and <code>nifi.provenance.repository.max.storage.size</code> (default value of <code>1 GB</code>) to values more suitable for your production environment</p> |
| </li> |
| <li> |
| <p>Change <code>nifi.provenance.repository.index.shard.size</code> from the default value of <code>500 MB</code> to <code>4 GB</code></p> |
| </li> |
| <li> |
| <p>Change <code>nifi.provenance.repository.index.threads</code> from the default value of <code>2</code> to either <code>4</code> or <code>8</code> as the Write Ahead repository enables this to scale better</p> |
| </li> |
| <li> |
| <p>If processing a high volume of events, change <code>nifi.provenance.repository.rollover.size</code> from the default of <code>100 MB</code> to <code>1 GB</code></p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>Once these property changes have been made, restart NiFi.</p> |
| </div> |
| <div class="paragraph"> |
| <p><strong>Note:</strong> Detailed descriptions for each of these properties can be found in <a href="administration-guide.html#system_properties">System Properties</a>.</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="experimental_warning"><a class="anchor" href="user-guide.html#experimental_warning"></a>Experimental Warning</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>While all Apache licensed code is provided "on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied" (see <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>), some features of Apache NiFi may be marked <strong>experimental</strong>. Experimental features may:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>have undergone less extensive testing than is normal for standard NiFi features</p> |
| </li> |
| <li> |
| <p>interact with unstable external dependencies</p> |
| </li> |
| <li> |
| <p>be subject to change (any exposed APIs should <strong>not</strong> be considered covered under the minor release backward compatibility guidelines of <a href="https://semver.org">Semantic Versioning</a>)</p> |
| </li> |
| <li> |
| <p>potentially cause data loss</p> |
| </li> |
| <li> |
| <p>not be directly supported by the community in the event issues arise</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>Every attempt is made to provide more detailed and specific information around the nature of the experimental warning on a per-feature basis. Questions around specific experimental features should be directed to the <a href="mailto:dev@nifi.apache.org">Apache NiFi Developer Mailing List</a>.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="other_management_features"><a class="anchor" href="user-guide.html#other_management_features"></a>Other Management Features</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>In addition to the Summary Page, Data Provenance Page, Template Management Page, and Bulletin Board Page, there are |
| other tools in the Global Menu (see <a href="user-guide.html#User_Interface">NiFi User Interface</a>) that are useful to the DFM. Select Flow Configuration History to view |
| all the changes that have been made to the dataflow. The history can aid in troubleshooting, such as if a |
| recent change to the dataflow has caused a problem and needs to be fixed. The DFM can see what changes have been made and |
| adjust the flow as needed to fix the problem. While NiFi does not have an "undo" feature, the DFM can make new changes to the |
| dataflow that will fix the problem.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Select Node Status History to view instance specific metrics from the last 24 hours or if the instance runs for less time, then |
| since it has been started. The status history can help the DFM in troubleshooting performance issues and provides a general |
| view on the health of the node. The status history includes information about the memory usage and disk usage among other things.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Two other tools in the Global Menu are Controller Settings and Users. The Controller Settings page provides the ability to change |
| the name of the NiFi instance, add comments describing the NiFi instance, and set the maximum number of threads that are available |
| to the application. It also provides tabs where DFMs may add and configure <a href="user-guide.html#Controller_Services">Controller Services</a> and <a href="user-guide.html#Reporting_Tasks">Reporting Tasks</a>. The Users page is used to manage user access, which is described in |
| the <a href="administration-guide.html">System Administrator’s Guide</a>.</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div id="footer"> |
| <div id="footer-text"> |
| Last updated 2022-12-05 18:19:00 -0600 |
| </div> |
| </div> |
| </body> |
| </html> |
