| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| # OpenTAXII |
| |
| Installs [OpenTAXII](https://github.com/EclecticIQ/OpenTAXII) as a deamon that can be launched via a SysV service script. The complementary client implementation, [Cabby](https://github.com/EclecticIQ/cabby) is also installed. |
| |
| OpenTAXII is a robust Python implementation of TAXII Services that delivers a rich feature set and friendly pythonic API. [TAXII](https://stixproject.github.io/) (Trusted Automated eXchange of Indicator Information) is a collection of specifications defining a set of services and message exchanges used for sharing cyber threat intelligence information between parties. |
| |
| ## Getting Started |
| |
| After deployment completes the OpenTAXII service is installed and running. A set of [Hail a TAXII](http://hailataxii.com/) threat intel collections have been defined and configured. Use the `status` option to view the collections that have been defined. |
| |
| ``` |
| $ service opentaxii status |
| Checking opentaxii... Running |
| guest.phishtank_com 0 |
| guest.Abuse_ch 0 |
| guest.CyberCrime_Tracker 0 |
| guest.EmergingThreats_rules 0 |
| guest.Lehigh_edu 0 |
| guest.MalwareDomainList_Hostlist 0 |
| guest.blutmagie_de_torExits 0 |
| guest.dataForLast_7daysOnly 0 |
| guest.dshield_BlockList 0 |
| ``` |
| |
| Notice that each collections contain zero records. None of the data is automatically synced during deployment. To sync the data manually use the `sync` option as defined below. The following example does not provide a begin and end time so the data will be fetched for the current day only. |
| |
| ``` |
| # service opentaxii sync guest.blutmagie_de_torExits |
| 2016-04-21 20:34:42,511 INFO: Starting new HTTP connection (1): localhost |
| 2016-04-21 20:34:42,540 INFO: Response received for Inbox_Message from http://localhost:9000/services/inbox |
| 2016-04-21 20:34:42,542 INFO: Sending Inbox_Message to http://localhost:9000/services/inbox |
| ... |
| 2016-04-21 20:34:42,719 INFO: Response received for Poll_Request from http://localhost:9000/services/poll |
| 2016-04-21 20:34:42,719 INFO: Content blocks count: 1618, is partial: False |
| ``` |
| |
| The OpenTAXII service now contains 1,618 threat intel records indicating Tor Exit nodes. |
| |
| ``` |
| [root@source ~]# service opentaxii status |
| Checking opentaxii... Running |
| guest.phishtank_com 0 |
| guest.Abuse_ch 0 |
| guest.CyberCrime_Tracker 0 |
| guest.EmergingThreats_rules 0 |
| guest.Lehigh_edu 0 |
| guest.MalwareDomainList_Hostlist 0 |
| guest.blutmagie_de_torExits 1618 |
| guest.dataForLast_7daysOnly 0 |
| guest.dshield_BlockList 0 |
| ``` |
| |
| ## Usage |
| |
| A standard SysV script has been installed to manage OpenTAXII. The following functions are available. |
| |
| `start` `stop` `restart` the OpenTAXII service |
| |
| `status` of the OpenTAXII service. The command displays the collections that have been defined and the number of records in each. |
| |
| ``` |
| $ service opentaxii status |
| Checking opentaxii... Running |
| guest.phishtank_com 984 |
| guest.Abuse_ch 45 |
| guest.CyberCrime_Tracker 482 |
| guest.EmergingThreats_rules 0 |
| guest.Lehigh_edu 1030 |
| guest.MalwareDomainList_Hostlist 84 |
| guest.blutmagie_de_torExits 3236 |
| guest.dataForLast_7daysOnly 3377 |
| guest.dshield_BlockList 0 |
| ``` |
| |
| `setup` Initializes the services and collections required to operate the OpenTAXII service. This will destroy all existing data. The user is prompted to continue before any data is destroyed. |
| |
| ``` |
| # service opentaxii setup |
| WARNING: force reset and destroy all opentaxii data? [Ny]: y |
| Stopping opentaxii ..Ok |
| 2016-04-21T19:56:01.886157Z [opentaxii.server] info: api.persistence.loaded {timestamp=2016-04-21T19:56:01.886157Z, logger=opentaxii.server, api_class=SQLDatabaseAPI, event=api.persistence.loaded, level=info} |
| 2016-04-21T19:56:01.896503Z [opentaxii.server] info: api.auth.loaded {timestamp=2016-04-21T19:56:01.896503Z, logger=opentaxii.server, api_class=SQLDatabaseAPI, event=api.auth.loaded, level=info} |
| 2016-04-21T19:56:01.896655Z [opentaxii.server] info: taxiiserver.configured {timestamp=2016-04-21T19:56:01.896655Z, logger=opentaxii.server, event=taxiiserver.configured, level=info} |
| ... |
| Ok |
| ``` |
| |
| `sync [collection] [begin-at] [end-at]` Syncs the threat intel data available at [Hail a TAXII](http://hailataxii.com/). If no begin and end date is provided then data is synced over the current day only. |
| - `collection` Name of the collection to sync. |
| - `begin-at` Exclusive begin of time window; ISO8601 |
| - `end-at` Inclusive end of time window; ISO8601 |
| |
| ``` |
| $ service opentaxii sync guest.phishtank_com |
| + /usr/local/opentaxii/opentaxii-venv/bin/taxii-proxy --poll-path http://hailataxii.com/taxii-data --poll-collection guest.phishtank_com --inbox-path http://localhost:9000/services/guest.phishtank_com-inbox --inbox-collection guest.phishtank_com --binding urn:stix.mitre.org:xml:1.1.1 --begin 2016-04-21 --end 2016-04-22 |
| 2016-04-21 17:36:23,778 INFO: Sending Poll_Request to http://hailataxii.com/taxii-data |
| 2016-04-21 17:36:23,784 INFO: Starting new HTTP connection (1): hailataxii.com |
| 2016-04-21 17:36:24,175 INFO: Response received for Poll_Request from http://hailataxii.com/taxii-data |
| 2016-04-21 17:36:24,274 INFO: Sending Inbox_Message to http://localhost:9000/services/guest.phishtank_com-inbox |
| ... |
| 2016-04-21 17:36:34,867 INFO: Response received for Poll_Request from http://localhost:9000/services/guest.phishtank_com-poll |
| 2016-04-21 17:36:34,868 INFO: Content blocks count: 6993, is partial: False |
| ``` |
| |
| ### Troubleshooting |
| |
| Should you need to explore the installation, here are instructions on doing so. |
| |
| OpenTAXII is installed in a virtual environment. Before exploring the environment run the following commands to perform the necessary setup. The specific paths may change depending on your Ansible settings. |
| |
| ``` |
| export LD_LIBRARY_PATH=/opt/rh/python27/root/usr/lib64 |
| export OPENTAXII_CONFIG=/usr/local/opentaxii/etc/opentaxii-conf.yml |
| cd /usr/local/opentaxii |
| . opentaxii-venv/bin/activate |
| ``` |
| |
| Discover available services. |
| |
| ``` |
| taxii-discovery --discovery http://localhost:9000/services/discovery |
| taxii-discovery --discovery http://hailataxii.com/taxii-data |
| ``` |
| |
| Explore available collections. |
| |
| ``` |
| taxii-collections --discovery http://localhost:9000/services/discovery |
| taxii-collections --discovery http://hailataxii.com/taxii-data |
| ``` |
| |
| Read data from a collection. |
| |
| ``` |
| taxii-poll --discovery http://localhost:9000/services/discovery -c guest.phishtank_com |
| taxii-poll --discovery http://hailataxii.com/taxii-data -c guest.phishtank_com --begin 2016-04-20 |
| ``` |
| |
| Manually load data into a collection. |
| |
| ``` |
| taxii-push \ |
| --discovery http://localhost:9000/services/discovery \ |
| --dest phishtank \ |
| --content-file data.xml \ |
| --username guest \ |
| --password guest |
| ``` |
| |
| Fetch data from a remote service and mirror it locally. |
| |
| ``` |
| taxii-proxy --poll-path http://hailataxii.com/taxii-data \ |
| --poll-collection guest.phishtank_com \ |
| --inbox-path http://localhost:9000/services/guest.phishtank_com-inbox \ |
| --inbox-collection guest.phishtank_com \ |
| --binding urn:stix.mitre.org:xml:1.1.1 \ |
| --inbox-username guest \ |
| --inbox-password guest \ |
| --begin 2016-04-20 |
| ``` |
