| <!DOCTYPE html> |
| <!-- |
| | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/development/fastcapa/index.md at 2019-05-14 |
| | Rendered using Apache Maven Fluido Skin 1.7 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| <head> |
| <meta charset="UTF-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <meta name="Date-Revision-yyyymmdd" content="20190514" /> |
| <meta http-equiv="Content-Language" content="en" /> |
| <title>Metron – Fastcapa Test Environment</title> |
| <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" /> |
| <link rel="stylesheet" href="../../../css/site.css" /> |
| <link rel="stylesheet" href="../../../css/print.css" media="print" /> |
| <script type="text/javascript" src="../../../js/apache-maven-fluido-1.7.min.js"></script> |
| <script type="text/javascript"> |
| $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); |
| </script> |
| </head> |
| <body class="topBarDisabled"> |
| <div class="container-fluid"> |
| <div id="banner"> |
| <div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> |
| <div class="pull-right"></div> |
| <div class="clear"><hr/></div> |
| </div> |
| |
| <div id="breadcrumbs"> |
| <ul class="breadcrumb"> |
| <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> |
| <li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> |
| <li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> |
| <li class="active ">Fastcapa Test Environment</li> |
| <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2019-05-14</li> |
| <li id="projectVersion" class="pull-right">Version: 0.7.1</li> |
| </ul> |
| </div> |
| <div class="row-fluid"> |
| <div id="leftColumn" class="span2"> |
| <div class="well sidebar-nav"> |
| <ul class="nav nav-list"> |
| <li class="nav-header">User Documentation</li> |
| <li><a href="../../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> |
| <li><a href="../../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> |
| <li><a href="../../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> |
| <li><a href="../../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> |
| <li><a href="../../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> |
| <li><a href="../../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-down"></span>Deployment</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup"><span class="none"></span>Kerberos-ambari-setup</a></li> |
| <li><a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup"><span class="none"></span>Kerberos-manual-setup</a></li> |
| <li><a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"><span class="none"></span>Amazon-ec2</a></li> |
| <li><a href="../../../metron-deployment/ansible/index.html" title="Ansible"><span class="icon-chevron-right"></span>Ansible</a></li> |
| <li><a href="../../../metron-deployment/development/index.html" title="Development"><span class="icon-chevron-down"></span>Development</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../../metron-deployment/development/centos6/index.html" title="Centos6"><span class="none"></span>Centos6</a></li> |
| <li class="active"><a href="#"><span class="none"></span>Fastcapa</a></li> |
| <li><a href="../../../metron-deployment/development/ubuntu14/index.html" title="Ubuntu14"><span class="none"></span>Ubuntu14</a></li> |
| </ul> |
| </li> |
| <li><a href="../../../metron-deployment/other-examples/index.html" title="Other-examples"><span class="icon-chevron-right"></span>Other-examples</a></li> |
| <li><a href="../../../metron-deployment/packaging/ambari/index.html" title="Ambari"><span class="icon-chevron-right"></span>Ambari</a></li> |
| <li><a href="../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"><span class="none"></span>Ansible-docker</a></li> |
| <li><a href="../../../metron-deployment/packaging/docker/deb-docker/index.html" title="Deb-docker"><span class="none"></span>Deb-docker</a></li> |
| <li><a href="../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"><span class="none"></span>Rpm-docker</a></li> |
| <li><a href="../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build"><span class="none"></span>Packer-build</a></li> |
| </ul> |
| </li> |
| <li><a href="../../../metron-interface/index.html" title="Interface"><span class="icon-chevron-right"></span>Interface</a></li> |
| <li><a href="../../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> |
| <li><a href="../../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> |
| <li><a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> |
| <li><a href="../../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> |
| <li><a href="../../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> |
| <li><a href="../../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> |
| </ul> |
| </li> |
| </ul> |
| <hr /> |
| <div id="poweredBy"> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" /></a> |
| </div> |
| </div> |
| </div> |
| <div id="bodyColumn" class="span10" > |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1>Fastcapa Test Environment</h1> |
| <p>Provides a test environment for the development and testing of Fastcapa. The environment is automatically validated after it is created to ensure that Fastcapa is behaving correctly.</p> |
| <p>Two virtualized nodes are launched with Vagrant that can communicate with one another over a private network.</p> |
| <ul> |
| |
| <li>The <tt>source</tt> node uses Metron’s <tt>pcap_replay</tt> functionality to transmit raw network packet data over a private network.</li> |
| <li>The <tt>sink</tt> node is running <tt>fastcapa</tt> and is capturing these network packets.</li> |
| <li>Fastcapa then transforms and bundles the packets into a message.</li> |
| <li>The message is sent to a Kafka broker running on the <tt>source</tt> node.</li> |
| </ul> |
| <div class="section"> |
| <h2><a name="Getting_Started"></a>Getting Started</h2> |
| <p>The Fastcapa test environment can be executed on different operating systems. There is a sub-directory for each operating system that Fastcapa can be tested on.</p> |
| <p>To run, simply execute <tt>vagrant up</tt> within the appropriate directory. For example, to run the tests on CentOS 7.1 then execute the following commands.</p> |
| |
| <div> |
| <div> |
| <pre class="source">cd centos-7.1 |
| vagrant up |
| </pre></div></div> |
| |
| <p>Automated tests are executed after provisioning completes to ensure that Fastcapa and the rest of the environment is functioning properly. If you see something like the following, then the tests have passed.</p> |
| |
| <div> |
| <div> |
| <pre class="source">$ vagrant up |
| ==> source: Running provisioner: ansible... |
| source: Running ansible-playbook... |
| ... |
| TASK [debug] ******************************************************************* |
| ok: [source] => { |
| "msg": "Successfully received packets sent from pcap-replay!" |
| } |
| ... |
| TASK [debug] ******************************************************************* |
| ok: [source] => { |
| "msg": "Successfully received a Kafka message from fastcapa!" |
| } |
| </pre></div></div> |
| |
| <p>If the deployment process fails mid-course, running <tt>vagrant provision</tt> will continue the process from where it left off. This can sometimes occur when the VM reboots as part of the deployment process. The error might look like the following.</p> |
| |
| <div> |
| <div> |
| <pre class="source">TASK [fastcapa : Restart for modified kernel params] *************************** |
| fatal: [sink]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Shared connection to 127.0.0.1 closed.\r\n", "unreachable": true} |
| to retry, use: --limit @/Users/nallen/Development/metron/metron-deployment/vagrant/fastcapa-test-platform/playbook.retry |
| |
| PLAY RECAP ********************************************************************* |
| sink : ok=11 changed=9 unreachable=1 failed=0 |
| source : ok=29 changed=25 unreachable=0 failed=0 |
| |
| Ansible failed to complete successfully. Any error output should be |
| visible above. Please fix these errors and try again. |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h2><a name="Going_Deeper"></a>Going Deeper</h2> |
| <p>This section will outline in more detail the environment and how to interact with it.</p> |
| <div class="section"> |
| <h3><a name="source"></a><tt>source</tt></h3> |
| <p>To validate that the <tt>source</tt> node is functioning properly, run the following commands.</p> |
| <p>First, ensure that the <tt>pcap-replay</tt> service is running.</p> |
| |
| <div> |
| <div> |
| <pre class="source">vagrant ssh source |
| sudo service pcap-replay status |
| </pre></div></div> |
| |
| <p>Use <tt>tcpdump</tt> to ensure that the raw packet data is being sent over the private network. Enter ‘CTRL-C’ to kill the <tt>tcpdump</tt> process once you are able to see that packets are being sent.</p> |
| |
| <div> |
| <div> |
| <pre class="source">sudo yum -y install tcpdump |
| sudo tcpdump -i enp0s8 |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="sink"></a><tt>sink</tt></h3> |
| <p>Next validate that the <tt>sink</tt> is functioning properly. Run the following commands starting from the host operating system.</p> |
| <p>First, ensure that the <tt>fastcapa</tt> service is running.</p> |
| |
| <div> |
| <div> |
| <pre class="source">vagrant ssh sink |
| service fastcapa status |
| </pre></div></div> |
| |
| <p>Ensure that the raw network packet data is being received by Kafka.</p> |
| |
| <div> |
| <div> |
| <pre class="source">/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic pcap |
| </pre></div></div> |
| |
| <p>Enter ‘CTRL-C’ to kill the <tt>kafka-console-consumer</tt> process once you are able to see that packets are being sent. These packets will appear to be gibberish in the console. This is the raw binary network packet data after all.</p></div></div> |
| <div class="section"> |
| <h2><a name="FAQ"></a>FAQ</h2> |
| <div class="section"> |
| <h3><a name="Error_Message:_Timed_out_while_waiting_for_the_machine_to_boot"></a>Error Message: Timed out while waiting for the machine to boot</h3> |
| |
| <div> |
| <div> |
| <pre class="source">Timed out while waiting for the machine to boot. This means that |
| Vagrant was unable to communicate with the guest machine within |
| the configured ("config.vm.boot_timeout" value) time period. |
| If you look above, you should be able to see the error(s) that |
| Vagrant had when attempting to connect to the machine. These errors |
| are usually good hints as to what may be wrong. |
| If you're using a custom box, make sure that networking is properly |
| working and you're able to connect to the machine. It is a common |
| problem that networking isn't setup properly in these boxes. |
| Verify that authentication configurations are also setup properly, |
| as well. |
| If the box appears to be booting properly, you may want to increase |
| the timeout ("config.vm.boot_timeout") value. |
| ➜ centos-7.4 git:(master) ✗ vagrant status |
| Current machine states: |
| source running (virtualbox) |
| sink not created (virtualbox) |
| </pre></div></div> |
| |
| <p>If you are unable to launch any of the Fastcapa test environments, which results in a message like the one above, then you may need to upgrade your version of Virtualbox. Success has been reported with versions of VirtualBox 5.1.22+.</p></div></div> |
| </div> |
| </div> |
| </div> |
| <hr/> |
| <footer> |
| <div class="container-fluid"> |
| <div class="row-fluid"> |
| © 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, |
| and the Apache Metron project logo are trademarks of The Apache Software Foundation. |
| </div> |
| </div> |
| </footer> |
| </body> |
| </html> |
