| |
| |
| <!DOCTYPE html> |
| <!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <title>Traffic Monitor Golang — Traffic Control 2.1-dev documentation </title> |
| |
| |
| |
| |
| <link rel="shortcut icon" href="../_static/favicon.ico"/> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> |
| |
| |
| |
| <link rel="stylesheet" href="../_static/theme_overrides.css" type="text/css" /> |
| |
| |
| |
| <link rel="top" title="Traffic Control 2.1-dev documentation" href="../index.html"/> |
| <link rel="up" title="Developer’s Guide" href="index.html"/> |
| <link rel="next" title="Traffic Stats" href="traffic_stats.html"/> |
| <link rel="prev" title="Traffic Monitor APIs" href="traffic_monitor/traffic_monitor_api.html"/> |
| |
| |
| <script src="_static/js/modernizr.min.js"></script> |
| |
| </head> |
| |
| <body class="wy-body-for-nav" role="document"> |
| |
| <div class="wy-grid-for-nav"> |
| |
| |
| <nav data-toggle="wy-nav-shift" class="wy-nav-side"> |
| <div class="wy-side-nav-search"> |
| |
| |
| |
| <a href="/" class="icon icon-home"> Traffic Control |
| |
| |
| |
| |
| <img src="../_static/tc_logo_c_only.png" class="logo" /> |
| |
| </a> |
| |
| |
| <div role="search"> |
| <form id="rtd-search-form" class="wy-form" action="../search.html" method="get"> |
| <input type="text" name="q" placeholder="Search docs" /> |
| <input type="hidden" name="check_keywords" value="yes" /> |
| <input type="hidden" name="area" value="default" /> |
| </form> |
| </div> |
| |
| |
| </div> |
| |
| <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> |
| |
| |
| |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../basics/index.html">CDN Basics</a><ul> |
| <li class="toctree-l2"><a class="reference internal" href="../basics/content_delivery_networks.html">Content Delivery Networks</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../basics/http_11.html">HTTP 1.1</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../basics/caching_proxies.html">Caching Proxies</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../basics/cache_revalidation.html">Cache Control Headers and Revalidation</a></li> |
| </ul> |
| </li> |
| </ul> |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../overview/index.html">Traffic Control Overview</a><ul> |
| <li class="toctree-l2"><a class="reference internal" href="../overview/introduction.html">Introduction</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../overview/traffic_ops.html">Traffic Ops</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../overview/traffic_portal.html">Traffic Portal</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../overview/traffic_router.html">Traffic Router</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../overview/traffic_monitor.html">Traffic Monitor</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../overview/traffic_stats.html">Traffic Stats</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../overview/traffic_server.html">Traffic Server</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../overview/traffic_vault.html">Traffic Vault</a></li> |
| </ul> |
| </li> |
| </ul> |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../admin/index.html">Administrator’s Guide</a><ul> |
| <li class="toctree-l2"><a class="reference internal" href="../admin/traffic_ops/installation.html">Traffic Ops - Installing</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../admin/traffic_ops/default_profiles.html">Traffic Ops - Default Profiles</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../admin/traffic_ops/migration_from_10_to_20.html">Traffic Ops - Migrating from 1.x to 2.x</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../admin/traffic_ops/configuration.html">Traffic Ops - Configuring</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../admin/traffic_ops/using.html">Traffic Ops - Using</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../admin/traffic_ops/extensions.html">Managing Traffic Ops Extensions</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../admin/traffic_portal.html">Traffic Portal Administration</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../admin/traffic_monitor.html">Traffic Monitor Administration</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../admin/traffic_monitor_golang.html">Traffic Monitor Administration</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../admin/traffic_router.html">Traffic Router Administration</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../admin/traffic_stats.html">Traffic Stats Administration</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../admin/traffic_server.html">Traffic Server Administration</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../admin/traffic_vault.html">Traffic Vault Administration</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../admin/quick_howto/index.html">Quick How To Guides</a></li> |
| </ul> |
| </li> |
| </ul> |
| <ul class="current"> |
| <li class="toctree-l1 current"><a class="reference internal" href="index.html">Developer’s Guide</a><ul class="current"> |
| <li class="toctree-l2"><a class="reference internal" href="building.html">Building Traffic Control</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="traffic_ops.html">Traffic Ops</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="traffic_portal.html">Traffic Portal</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="traffic_router.html">Traffic Router</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="traffic_monitor.html">Traffic Monitor</a></li> |
| <li class="toctree-l2 current"><a class="current reference internal" href="#">Traffic Monitor Golang</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="traffic_stats.html">Traffic Stats</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="traffic_server.html">Traffic Server</a></li> |
| </ul> |
| </li> |
| </ul> |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../faq/index.html">FAQ</a><ul> |
| <li class="toctree-l2"><a class="reference internal" href="../faq/general.html">General</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../faq/development.html">Development</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../faq/administration.html">Running a Traffic Control CDN</a></li> |
| </ul> |
| </li> |
| </ul> |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../glossary.html">Glossary</a></li> |
| </ul> |
| |
| |
| |
| </div> |
| |
| </nav> |
| |
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> |
| |
| |
| <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> |
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> |
| <a href="../index.html">Traffic Control</a> |
| </nav> |
| |
| |
| |
| <div class="wy-nav-content"> |
| <div class="rst-content"> |
| <div role="navigation" aria-label="breadcrumbs navigation"> |
| <ul class="wy-breadcrumbs"> |
| <li><a href="../index.html">Traffic Control 2.1-dev</a> »</li> |
| |
| <li><a href="index.html">Developer’s Guide</a> »</li> |
| |
| <li>Traffic Monitor Golang</li> |
| <li class="wy-breadcrumbs-aside"> |
| |
| <a href="../_sources/development/traffic_monitor_golang.txt" rel="nofollow"> View page source</a> |
| |
| </li> |
| </ul> |
| <hr/> |
| </div> |
| |
| <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> |
| |
| <a href="traffic_stats.html" class="btn btn-neutral float-right" title="Traffic Stats">Next <span class="fa fa-arrow-circle-right"></span></a> |
| |
| |
| <a href="traffic_monitor/traffic_monitor_api.html" class="btn btn-neutral" title="Traffic Monitor APIs"><span class="fa fa-arrow-circle-left"></span> Previous</a> |
| |
| </div> |
| |
| <div role="main" class="document"> |
| |
| <div class="section" id="traffic-monitor-golang"> |
| <h1>Traffic Monitor Golang<a class="headerlink" href="#traffic-monitor-golang" title="Permalink to this headline">¶</a></h1> |
| <div class="section" id="introduction"> |
| <h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2> |
| <p>The next major version of Traffic Monitor has been completely rewritten in Golang. Currently, this version is functionally equivalent, and should be considered “beta.” It is recommended that new CDN deployments continue to use the existing Java version, until the new version is completely moved over in the source and binary distributions. However, developers and administrators are encouraged to test the Golang version, to prepare for operational differences and look for bugs.</p> |
| <p>Traffic Monitor is an HTTP service application that monitors caches, provides health state information to Traffic Router, and collects statistics for use in tools such as Traffic Ops and Traffic Stats. The health state provided by Traffic Monitor is used by Traffic Router to control which caches are available on the CDN.</p> |
| </div> |
| <div class="section" id="software-requirements"> |
| <h2>Software Requirements<a class="headerlink" href="#software-requirements" title="Permalink to this headline">¶</a></h2> |
| <p>To work on Traffic Monitor you need a *nix (MacOS and Linux are most commonly used) environment that has the following installed:</p> |
| <ul class="simple"> |
| <li>Golang</li> |
| </ul> |
| </div> |
| <div class="section" id="project-tree-overview"> |
| <h2>Project Tree Overview<a class="headerlink" href="#project-tree-overview" title="Permalink to this headline">¶</a></h2> |
| <ul> |
| <li><p class="first"><code class="docutils literal"><span class="pre">traffic_control/traffic_monitor/</span></code> - base directory for Traffic Monitor.</p> |
| </li> |
| <li><p class="first"><code class="docutils literal"><span class="pre">cache/</span></code> - Handler for processing cache results.</p> |
| </li> |
| <li><p class="first"><code class="docutils literal"><span class="pre">config/</span></code> - Application configuration; in-memory objects from <code class="docutils literal"><span class="pre">traffic_monitor.cfg</span></code>.</p> |
| </li> |
| <li><p class="first"><code class="docutils literal"><span class="pre">crconfig/</span></code> - struct for deserlializing the CRConfig from JSON.</p> |
| </li> |
| <li><p class="first"><code class="docutils literal"><span class="pre">deliveryservice/</span></code> - aggregates delivery service data from cache results.</p> |
| </li> |
| <li><p class="first"><code class="docutils literal"><span class="pre">deliveryservicedata/</span></code> - deliveryservice structs. This exists separate from <code class="docutils literal"><span class="pre">deliveryservice</span></code> to avoid circular dependencies.</p> |
| </li> |
| <li><p class="first"><code class="docutils literal"><span class="pre">enum/</span></code> - enumerations and name alias types.</p> |
| </li> |
| <li><p class="first"><code class="docutils literal"><span class="pre">health/</span></code> - functions for calculating cache health, and creating health event objects.</p> |
| </li> |
| <li><dl class="first docutils"> |
| <dt><code class="docutils literal"><span class="pre">manager/</span></code> - manager goroutines (microthreads).</dt> |
| <dd><ul class="first last simple"> |
| <li><code class="docutils literal"><span class="pre">health.go</span></code> - Health request manager. Processes health results, from the health poller -> fetcher -> manager. The health poll is the “heartbeat” containing a small amount of stats, primarily to determine whether a cache is reachable as quickly as possible. Data is aggregated and inserted into shared threadsafe objects.</li> |
| <li><code class="docutils literal"><span class="pre">manager.go</span></code> - Contains <code class="docutils literal"><span class="pre">Start</span></code> function to start all pollers, handlers, and managers.</li> |
| <li><code class="docutils literal"><span class="pre">monitorconfig.go</span></code> - Monitor config manager. Gets data from the monitor config poller, which polls Traffic Ops for changes to which caches are monitored and how.</li> |
| <li><code class="docutils literal"><span class="pre">opsconfig.go</span></code> - Ops config manager. Gets data from the ops config poller, which polls Traffic Ops for changes to monitoring settings.</li> |
| <li><code class="docutils literal"><span class="pre">peer.go</span></code> - Peer manager. Gets data from the peer poller -> fetcher -> handler and aggregates it into the shared threadsafe objects.</li> |
| <li><code class="docutils literal"><span class="pre">stat.go</span></code> - Stat request manager. Processes stat results, from the stat poller -> fetcher -> manager. The stat poll is the large statistics poll, containing all stats (such as HTTP codes, transactions, delivery service statistics, and more). Data is aggregated and inserted into shared threadsafe objects.</li> |
| <li><code class="docutils literal"><span class="pre">statecombiner.go</span></code> - Manager for combining local and peer states, into a single combined states threadsafe object, for serving the CrStates endpoint.</li> |
| </ul> |
| </dd> |
| </dl> |
| </li> |
| <li><p class="first"><code class="docutils literal"><span class="pre">datareq/</span></code> - HTTP routing, which has threadsafe health and stat objects populated by stat and health managers.</p> |
| </li> |
| <li><p class="first"><code class="docutils literal"><span class="pre">peer/</span></code> - Manager for getting and populating peer data from other Traffic Monitors</p> |
| </li> |
| <li><p class="first"><code class="docutils literal"><span class="pre">srvhttp/</span></code> - HTTP service. Given a map of endpoint functions, which are lambda closures containing aggregated data objects.</p> |
| </li> |
| <li><p class="first"><code class="docutils literal"><span class="pre">static/</span></code> - Web GUI HTML and javascript files</p> |
| </li> |
| <li><p class="first"><code class="docutils literal"><span class="pre">threadsafe/</span></code> - Threadsafe objects for storing aggregated data needed by multiple goroutines (typically the aggregator and HTTP server)</p> |
| </li> |
| <li><p class="first"><code class="docutils literal"><span class="pre">trafficopsdata/</span></code> - Struct for fetching and storing Traffic Ops data needed from the CRConfig. This is primarily mappings, such as delivery service servers, and server types.</p> |
| </li> |
| <li><p class="first"><code class="docutils literal"><span class="pre">trafficopswrapper/</span></code> - Threadsafe wrapper around the Traffic Ops client. The client used to not be threadsafe, however, it mostly (possibly entirely) is now. But, the wrapper also serves to overwrite the Traffic Ops <code class="docutils literal"><span class="pre">monitoring.json</span></code> values, which are live, with snapshotted CRConfig values.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="section" id="architecture"> |
| <h2>Architecture<a class="headerlink" href="#architecture" title="Permalink to this headline">¶</a></h2> |
| <p>At the highest level, Traffic Monitor polls caches, aggregates their data and availability, and serves it at HTTP JSON endpoints.</p> |
| <p>In the code, the data flows thru microthread (goroutine) pipelines. All stages of the pipeline are independent running microthreads:sup:<cite>1</cite>. The pipelines are:</p> |
| <ul class="simple"> |
| <li><strong>stat poll</strong> - polls caches for all statistics data. This should be a slower poll, which gets a lot of data.</li> |
| <li><strong>health poll</strong> - polls caches for a tiny amount of data, typically system information. This poll is designed to be a heartbeat, determining quickly whether the cache is reachable. Since it’s a small amount of data, it should poll more frequently.</li> |
| <li><strong>peer poll</strong> - polls Traffic Monitor peers for their availability data, and aggregates it with its own availability results and that of all other peers.</li> |
| <li><strong>monitor config</strong> - polls Traffic Ops for the list of Traffic Monitors and their info.</li> |
| <li><strong>ops config</strong> - polls for changes to the ops config file <code class="docutils literal"><span class="pre">traffic_ops.cfg</span></code>, and sends updates to other pollers when the config file has changed.<ul> |
| <li>The ops config manager also updates the shared Traffic Ops client, since it’s the actor which becomes notified of config changes requiring a new client.</li> |
| <li>The ops config manager also manages, creates, and recreates the HTTP server, since ops config changes necessitate restarting the HTTP server.</li> |
| </ul> |
| </li> |
| </ul> |
| <p>All microthreads in the pipeline are started by <code class="docutils literal"><span class="pre">manager/manager.go:Start()</span></code>.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">--------------------</span> <span class="o">--------------------</span> <span class="o">--------------------</span> |
| <span class="o">|</span> <span class="n">ops</span> <span class="n">config</span> <span class="n">poller</span> <span class="o">|-->|</span> <span class="n">ops</span> <span class="n">config</span> <span class="n">handler</span> <span class="o">|-->|</span> <span class="n">ops</span> <span class="n">config</span> <span class="n">manager</span> <span class="o">|-->-</span><span class="n">restart</span> <span class="n">HTTP</span> <span class="n">server</span><span class="o">-------------------------</span> |
| <span class="o">-------------------</span> <span class="o">--------------------</span> <span class="o">--------------------</span> <span class="o">|</span> <span class="o">|</span> |
| <span class="o">-->-</span><span class="n">ops</span> <span class="n">config</span> <span class="n">change</span> <span class="n">subscriber</span><span class="o">-------------</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> |
| <span class="o">-->-</span><span class="n">Traffic</span> <span class="n">Ops</span> <span class="n">client</span> <span class="n">change</span> <span class="n">subscriber</span><span class="o">--</span> <span class="o">|</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> |
| <span class="o">-------------------------------------------------------------------------------------------------------------</span> <span class="o">|</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">------------------------------------------------------------------------------------------------------------</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> |
| \<span class="o">/</span> \<span class="o">/</span> <span class="o">|</span> |
| <span class="o">-----------------------</span> <span class="o">------------------------</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="n">monitor</span> <span class="n">config</span> <span class="n">poller</span> <span class="o">|-->|</span> <span class="n">monitor</span> <span class="n">config</span> <span class="n">manager</span> <span class="o">|-->-</span><span class="n">stat</span> <span class="n">subscriber</span><span class="o">--------</span> <span class="o">-----------------------</span> |
| <span class="o">-----------------------</span> <span class="o">------------------------</span> <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> |
| <span class="o">|->-</span><span class="n">health</span> <span class="n">subscriber</span><span class="o">---</span> <span class="o">|</span> \<span class="o">/</span> <span class="n">_</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> <span class="o">-------------</span> <span class="n">_</span><span class="p">(</span> <span class="p">)</span><span class="o">.</span><span class="n">_</span> |
| <span class="o">-->-</span><span class="n">peer</span> <span class="n">subscriber</span><span class="o">--</span> <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> <span class="n">HTTP</span> <span class="n">server</span> <span class="o">|->-</span><span class="n">HTTP</span> <span class="n">request</span><span class="o">-></span> <span class="p">(</span><span class="n">____</span><span class="p">)</span><span class="n">_</span><span class="p">)</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> <span class="o">-------------</span> |
| <span class="o">-----------------------------------------------------------------------------</span> <span class="o">|</span> <span class="o">|</span> <span class="o">^</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">-----------------------------------------------------------------------------</span> <span class="o">|</span> <span class="o">------------------------</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">-----------------------------------------------------------------------------</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> <span class="o">^</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> <span class="o">-------------</span> <span class="o">--------------</span> <span class="o">--------------</span> <span class="o">--------------</span> <span class="o">-----------------------</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">-->|</span> <span class="n">stat</span> <span class="n">poller</span> <span class="o">|-->|</span> <span class="n">stat</span> <span class="n">fetcher</span> <span class="o">|-->|</span> <span class="n">stat</span> <span class="n">handler</span> <span class="o">|-->|</span> <span class="n">stat</span> <span class="n">manager</span> <span class="o">|->--------</span><span class="nb">set</span> <span class="n">shared</span> <span class="n">data</span><span class="o">->|</span> <span class="n">shared</span> <span class="n">data</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">-------------</span> <span class="o">|</span> <span class="o">--------------</span> <span class="o">--------------</span> <span class="o">|</span> <span class="o">--------------</span> <span class="o">-----------------------</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> <span class="o">--------------</span> <span class="o">--------------</span> <span class="o">|</span> <span class="o">|</span> <span class="n">events</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">|->|</span> <span class="n">stat</span> <span class="n">fetcher</span> <span class="o">|-->|</span> <span class="n">stat</span> <span class="n">handler</span> <span class="o">|-|</span> <span class="o">|</span> <span class="n">toData</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> <span class="o">--------------</span> <span class="o">--------------</span> <span class="o">|</span> <span class="o">|</span> <span class="n">errorCount</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">...</span> <span class="o">...</span> <span class="o">|</span> <span class="n">healthIteration</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">|</span> <span class="n">fetchCount</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">---------------</span> <span class="o">----------------</span> <span class="o">----------------</span> <span class="o">----------------</span> <span class="o">|</span> <span class="n">localStates</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">---->|</span> <span class="n">health</span> <span class="n">poller</span> <span class="o">|-->|</span> <span class="n">health</span> <span class="n">fetcher</span> <span class="o">|-->|</span> <span class="n">health</span> <span class="n">handler</span> <span class="o">|-->|</span> <span class="n">health</span> <span class="n">manager</span> <span class="o">|->-</span><span class="nb">set</span> <span class="n">shared</span> <span class="n">data</span><span class="o">->|</span> <span class="n">toSession</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">---------------</span> <span class="o">|</span> <span class="o">----------------</span> <span class="o">----------------</span> <span class="o">|</span> <span class="o">----------------</span> <span class="o">|</span> <span class="n">peerStates</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">----------------</span> <span class="o">----------------</span> <span class="o">|</span> <span class="o">|</span> <span class="n">monitorConfig</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">|->|</span> <span class="n">health</span> <span class="n">fetcher</span> <span class="o">|-->|</span> <span class="n">health</span> <span class="n">handler</span> <span class="o">|-|</span> <span class="o">|</span> <span class="n">combinedStates</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="o">----------------</span> <span class="o">----------------</span> <span class="o">|</span> <span class="o">|</span> <span class="n">statInfoHistory</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">...</span> <span class="o">...</span> <span class="o">|</span> <span class="n">statResultHistory</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">|</span> <span class="n">statMaxKbpses</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">-------------</span> <span class="o">--------------</span> <span class="o">--------------</span> <span class="o">--------------</span> <span class="o">|</span> <span class="n">lastKbpsStats</span> <span class="o">|</span> |
| <span class="o">------>|</span> <span class="n">peer</span> <span class="n">poller</span> <span class="o">|-->|</span> <span class="n">peer</span> <span class="n">fetcher</span> <span class="o">|-->|</span> <span class="n">peer</span> <span class="n">handler</span> <span class="o">|-->|</span> <span class="n">peer</span> <span class="n">manager</span> <span class="o">|->----------</span><span class="nb">set</span> <span class="n">shared</span> <span class="n">data</span><span class="o">->|</span> <span class="n">dsStats</span> <span class="o">|</span> |
| <span class="o">-------------</span> <span class="o">|</span> <span class="o">--------------</span> <span class="o">--------------</span> <span class="o">|</span> <span class="o">--------------</span> <span class="o">|</span> <span class="n">localCacheStatus</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">--------------</span> <span class="o">--------------</span> <span class="o">|</span> <span class="o">|</span> <span class="n">lastHealthDurations</span> <span class="o">|</span> |
| <span class="o">|->|</span> <span class="n">peer</span> <span class="n">fetcher</span> <span class="o">|-->|</span> <span class="n">peer</span> <span class="n">handler</span> <span class="o">|-|</span> <span class="o">|</span> <span class="n">healthHistory</span> <span class="o">|</span> |
| <span class="o">|</span> <span class="o">--------------</span> <span class="o">--------------</span> <span class="o">|</span> <span class="o">-----------------------</span> |
| <span class="o">...</span> <span class="o">...</span> |
| </pre></div> |
| </div> |
| <p>:sup:<a href="#id1"><span class="problematic" id="id2">`</span></a>1`Technically, some stages which are one-to-one simply call the next stage as a function. For example, the Fetcher calls the Handler as a function in the same microthread. But this isn’t architecturally significant.</p> |
| <div class="section" id="stat-pipeline"> |
| <h3>Stat Pipeline<a class="headerlink" href="#stat-pipeline" title="Permalink to this headline">¶</a></h3> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">---------</span> <span class="o">---------</span> <span class="o">---------</span> <span class="o">---------</span> |
| <span class="o">|</span> <span class="n">poller</span> <span class="o">|-->|</span> <span class="n">fetcher</span> <span class="o">|-->|</span> <span class="n">handler</span> <span class="o">|-->|</span> <span class="n">manager</span> <span class="o">|</span> |
| <span class="o">--------</span> <span class="o">|</span> <span class="o">---------</span> <span class="o">---------</span> <span class="o">|</span> <span class="o">---------</span> |
| <span class="o">|</span> <span class="o">---------</span> <span class="o">---------</span> <span class="o">|</span> |
| <span class="o">|->|</span> <span class="n">fetcher</span> <span class="o">|-->|</span> <span class="n">handler</span> <span class="o">|-|</span> |
| <span class="o">|</span> <span class="o">---------</span> <span class="o">---------</span> <span class="o">|</span> |
| <span class="o">...</span> <span class="o">...</span> |
| </pre></div> |
| </div> |
| <ul class="simple"> |
| <li><strong>poller</strong> - <code class="docutils literal"><span class="pre">common/poller/poller.go:HttpPoller.Poll()</span></code>. Listens for config changes (from the ops config manager), and starts its own internal microthreads, one for each cache to poll. These internal microthreads call the Fetcher at each cache’s poll interval.</li> |
| <li><strong>fetcher</strong> - <code class="docutils literal"><span class="pre">common/fetcher/fetcher.go:HttpFetcher.Fetch()</span></code>. Fetches the given URL, and passes the returned data to the Handler, along with any errors.</li> |
| <li><strong>handler</strong> - <code class="docutils literal"><span class="pre">traffic_monitor/cache/cache.go:Handler.Handle()</span></code>. Takes the given result and does all data computation possible with the single result. Currently, this computation primarily involves processing the denormalized ATS data into Go structs, and processing System data into OutBytes, Kbps, etc. Precomputed data is then passed to its result channel, which is picked up by the Manager.</li> |
| <li><strong>manager</strong> - <code class="docutils literal"><span class="pre">traffic_monitor/manager/stat.go:StartStatHistoryManager()</span></code>. Takes preprocessed results, and aggregates them. Aggregated results are then placed in shared data structures. The major data aggregated are delivery service statistics, and cache availability data. See <span class="xref std std-ref">Aggregated Stat Data</span> and <span class="xref std std-ref">Aggregated Availability Data</span>.</li> |
| </ul> |
| </div> |
| <div class="section" id="health-pipeline"> |
| <h3>Health Pipeline<a class="headerlink" href="#health-pipeline" title="Permalink to this headline">¶</a></h3> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">---------</span> <span class="o">---------</span> <span class="o">---------</span> <span class="o">---------</span> |
| <span class="o">|</span> <span class="n">poller</span> <span class="o">|-->|</span> <span class="n">fetcher</span> <span class="o">|-->|</span> <span class="n">handler</span> <span class="o">|-->|</span> <span class="n">manager</span> <span class="o">|</span> |
| <span class="o">--------</span> <span class="o">|</span> <span class="o">---------</span> <span class="o">---------</span> <span class="o">|</span> <span class="o">---------</span> |
| <span class="o">|</span> <span class="o">---------</span> <span class="o">---------</span> <span class="o">|</span> |
| <span class="o">|->|</span> <span class="n">fetcher</span> <span class="o">|-->|</span> <span class="n">handler</span> <span class="o">|-|</span> |
| <span class="o">|</span> <span class="o">---------</span> <span class="o">---------</span> <span class="o">|</span> |
| <span class="o">...</span> <span class="o">...</span> |
| </pre></div> |
| </div> |
| <ul class="simple"> |
| <li><strong>poller</strong> - <code class="docutils literal"><span class="pre">common/poller/poller.go:HttpPoller.Poll()</span></code>. Same poller type as the Stat Poller pipeline, with a different handler object.</li> |
| <li><strong>fetcher</strong> - <code class="docutils literal"><span class="pre">common/fetcher/fetcher.go:HttpFetcher.Fetch()</span></code>. Same fetcher type as the Stat Poller pipeline, with a different handler object.</li> |
| <li><strong>handler</strong> - <code class="docutils literal"><span class="pre">traffic_monitor/cache/cache.go:Handler.Handle()</span></code>. Same handler type as the Stat Poller pipeline, but constructed with a flag to not precompute. The health endpoint is of the same form as the stat endpoint, but doesn’t return all stat data. So, it doesn’t precompute like the Stat Handler, but only processes the system data, and passes the processed result to its result channel, which is picked up by the Manager.</li> |
| <li><strong>manager</strong> - <code class="docutils literal"><span class="pre">traffic_monitor/manager/health.go:StartHealthResultManager()</span></code>. Takes preprocessed results, and aggregates them. For the Health pipeline, only health availability data is aggregated. Aggregated results are then placed in shared data structures (lastHealthDurationsThreadsafe, lastHealthEndTimes, etc). See <span class="xref std std-ref">Aggregated Availability Data</span>.</li> |
| </ul> |
| </div> |
| <div class="section" id="peer-pipeline"> |
| <h3>Peer Pipeline<a class="headerlink" href="#peer-pipeline" title="Permalink to this headline">¶</a></h3> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">---------</span> <span class="o">---------</span> <span class="o">---------</span> <span class="o">---------</span> |
| <span class="o">|</span> <span class="n">poller</span> <span class="o">|-->|</span> <span class="n">fetcher</span> <span class="o">|-->|</span> <span class="n">handler</span> <span class="o">|-->|</span> <span class="n">manager</span> <span class="o">|</span> |
| <span class="o">--------</span> <span class="o">|</span> <span class="o">---------</span> <span class="o">---------</span> <span class="o">|</span> <span class="o">---------</span> |
| <span class="o">|</span> <span class="o">---------</span> <span class="o">---------</span> <span class="o">|</span> |
| <span class="o">|->|</span> <span class="n">fetcher</span> <span class="o">|-->|</span> <span class="n">handler</span> <span class="o">|-|</span> |
| <span class="o">|</span> <span class="o">---------</span> <span class="o">---------</span> <span class="o">|</span> |
| <span class="o">...</span> <span class="o">...</span> |
| </pre></div> |
| </div> |
| <ul class="simple"> |
| <li><strong>poller</strong> - <code class="docutils literal"><span class="pre">common/poller/poller.go:HttpPoller.Poll()</span></code>. Same poller type as the Stat and Health Poller pipelines, with a different handler object. Its config changes come from the Monitor Config Manager, and it starts an internal microthread for each peer to poll.</li> |
| <li><strong>fetcher</strong> - <code class="docutils literal"><span class="pre">common/fetcher/fetcher.go:HttpFetcher.Fetch()</span></code>. Same fetcher type as the Stat and Health Poller pipeline, with a different handler object.</li> |
| <li><strong>handler</strong> - <code class="docutils literal"><span class="pre">traffic_monitor/cache/peer.go:Handler.Handle()</span></code>. Decodes the JSON result into an object, and without further processing passes to its result channel, which is picked up by the Manager.</li> |
| <li><strong>manager</strong> - <code class="docutils literal"><span class="pre">traffic_monitor/manager/peer.go:StartPeerManager()</span></code>. Takes JSON peer Traffic Monitor results, and aggregates them. The availability of the Peer Traffic Monitor itself, as well as all cache availability from the given peer result, is stored in the shared <code class="docutils literal"><span class="pre">peerStates</span></code> object. Results are then aggregated via a call to the <code class="docutils literal"><span class="pre">combineState()</span></code> lambda, which signals the State Combiner microthread (which stores the combined availability in the shared object <code class="docutils literal"><span class="pre">combinedStates</span></code>; See <span class="xref std std-ref">State Combiner</span>).</li> |
| </ul> |
| </div> |
| <div class="section" id="monitor-config-pipeline"> |
| <h3>Monitor Config Pipeline<a class="headerlink" href="#monitor-config-pipeline" title="Permalink to this headline">¶</a></h3> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">---------</span> <span class="o">---------</span> |
| <span class="o">|</span> <span class="n">poller</span> <span class="o">|-->|</span> <span class="n">manager</span> <span class="o">|--></span> <span class="n">stat</span> <span class="n">subscriber</span> <span class="p">(</span><span class="n">Stat</span> <span class="n">pipeline</span> <span class="n">Poller</span><span class="p">)</span> |
| <span class="o">--------</span> <span class="o">---------</span> <span class="o">|</span> |
| <span class="o">|-></span> <span class="n">health</span> <span class="n">subscriber</span> <span class="p">(</span><span class="n">Health</span> <span class="n">pipeline</span> <span class="n">Poller</span><span class="p">)</span> |
| <span class="o">|</span> |
| <span class="o">--></span> <span class="n">peer</span> <span class="n">subscriber</span> <span class="p">(</span><span class="n">Peer</span> <span class="n">pipeline</span> <span class="n">Poller</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <ul class="simple"> |
| <li><strong>poller</strong> - <code class="docutils literal"><span class="pre">common/poller/poller.go:MonitorConfigPoller.Poll()</span></code>. The Monitor Config poller, on its interval, polls Traffic Ops for the Monitor configuration, and writes the polled value to its result channel, which is read by the Manager.</li> |
| <li><strong>manager</strong> - <code class="docutils literal"><span class="pre">traffic_monitor/manager/monitorconfig.go:StartMonitorConfigManager()</span></code>. Listens for results from the poller, and processes them. Cache changes are written to channels read by the Health, Stat, and Peer pollers. In the Shared Data objects, this also sets the list of new delivery services and removes ones which no longer exist, and sets the list of peer Traffic Monitors.</li> |
| </ul> |
| </div> |
| <div class="section" id="ops-config-pipeline"> |
| <h3>Ops Config Pipeline<a class="headerlink" href="#ops-config-pipeline" title="Permalink to this headline">¶</a></h3> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">---------</span> <span class="o">---------</span> <span class="o">---------</span> |
| <span class="o">|</span> <span class="n">poller</span> <span class="o">|-->|</span> <span class="n">handler</span> <span class="o">|-->|</span> <span class="n">manager</span> <span class="o">|--></span> <span class="n">ops</span> <span class="n">config</span> <span class="n">change</span> <span class="n">subscriber</span> <span class="p">(</span><span class="n">Monitor</span> <span class="n">Config</span> <span class="n">Poller</span><span class="p">)</span> |
| <span class="o">--------</span> <span class="o">---------</span> <span class="o">---------</span> <span class="o">|</span> |
| <span class="o">--></span> <span class="n">Traffic</span> <span class="n">ops</span> <span class="n">client</span> <span class="n">change</span> <span class="n">subscriber</span> <span class="p">(</span><span class="n">Monitor</span> <span class="n">Config</span> <span class="n">Poller</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <ul class="simple"> |
| <li><strong>poller</strong> - <code class="docutils literal"><span class="pre">common/poller/poller.go:FilePoller.Poll()</span></code>. Polls for changes to the Traffic Ops config file <code class="docutils literal"><span class="pre">traffic_ops.cfg</span></code>, and writes the changed config to its result channel, which is read by the Handler.</li> |
| <li><strong>handler</strong> - <code class="docutils literal"><span class="pre">common/handler/handler.go:OpsConfigFileHandler.Listen()</span></code>. Takes the given raw config, unmarshalls the JSON into an object, and writes the object to its channel, which is read by the Manager, along with any error.</li> |
| <li><strong>manager</strong> - <code class="docutils literal"><span class="pre">traffic_monitor/manager/monitorconfig.go:StartMonitorConfigManager()</span></code>. Listens for new configs, and processes them. When a new config is received, a new HTTP dispatch map is created via <code class="docutils literal"><span class="pre">traffic_monitor/datareq/datareq.go:MakeDispatchMap()</span></code>, and the HTTP server is restarted with the new dispatch map. The Traffic Ops client is also recreated, and stored in its shared data object. The Ops Config change subscribers and Traffic Ops Client change subscribers (the Monitor Config poller) are also passed the new ops config and new Traffic Ops client.</li> |
| </ul> |
| </div> |
| <div class="section" id="events"> |
| <h3>Events<a class="headerlink" href="#events" title="Permalink to this headline">¶</a></h3> |
| <p>The <code class="docutils literal"><span class="pre">events</span></code> shared data object is passed to each pipeline microthread which needs to signal events. Most of them do. Events are then logged, and visible in the UI as well as an HTTP JSON endpoint. Most events are caches becoming available or unavailable, but include other things such as peer availability changes.</p> |
| </div> |
| <div class="section" id="state-combiner"> |
| <h3>State Combiner<a class="headerlink" href="#state-combiner" title="Permalink to this headline">¶</a></h3> |
| <p>The State Combiner is a microthread started in <code class="docutils literal"><span class="pre">traffic_monitor/manager/manager.go:Start()</span></code> via <code class="docutils literal"><span class="pre">traffic_monitor/manager/statecombiner.go:StartStateCombiner()</span></code>, which listens for signals to combine states. It should be signaled by any pipeline which updates the local or peer availability shared data objects, <code class="docutils literal"><span class="pre">localStates</span></code> and <code class="docutils literal"><span class="pre">peerStates</span></code>. It holds the threadsafe shared data objects for local states and peer states, so no data is passed or returned, only a signal.</p> |
| <p>When a signal is received, it combines the local and peer states optimistically. That is, if a cache is marked available locally or by any peer, that cache is marked available in the combined states. There exists a variable to combine pessimistically, which may be set at compile time (it’s unusual for a CDN to operate well with pessimistic cache availability). Combined data is stored in the threadsafe shared data object <code class="docutils literal"><span class="pre">combinedStates</span></code>.</p> |
| </div> |
| <div class="section" id="aggregated-stat-data"> |
| <h3>Aggregated Stat Data<a class="headerlink" href="#aggregated-stat-data" title="Permalink to this headline">¶</a></h3> |
| <p>The Stat pipeline Manager is responsible for aggregating stats from all caches, into delivery services statistics. This is done via a call to <code class="docutils literal"><span class="pre">traffic_monitor/deliveryservice/stat.go:CreateStats()</span></code>.</p> |
| </div> |
| <div class="section" id="aggregated-availability-data"> |
| <h3>Aggregated Availability Data<a class="headerlink" href="#aggregated-availability-data" title="Permalink to this headline">¶</a></h3> |
| <p>Both the Stat and Health pipelines aggregate availability data received from caches. This is done via a call to <code class="docutils literal"><span class="pre">traffic_monitor/deliveryservice/health.go:CalcAvailability()</span></code> followed by a call to <code class="docutils literal"><span class="pre">combineState()</span></code>. The <code class="docutils literal"><span class="pre">CalcAvailability</span></code> function calculates the availability of each cache from the result of polling it, that is, local availability. The <code class="docutils literal"><span class="pre">combineState()</span></code> function is a lambda passed to the Manager, which signals the State Combiner microthread, which will combine the local and peer Traffic Monitor availability data, and insert it into the shared data <code class="docutils literal"><span class="pre">combinedStates</span></code> object.</p> |
| </div> |
| <div class="section" id="http-data-requests"> |
| <h3>HTTP Data Requests<a class="headerlink" href="#http-data-requests" title="Permalink to this headline">¶</a></h3> |
| <p>Data is provided to HTTP requests via the threadsafe shared data objects (see <span class="xref std std-ref">Shared Data</span>). These objects are closed in lambdas created via <code class="docutils literal"><span class="pre">traffic_monitor/datareq/datareq.go:MakeDispatchMap()</span></code>. This is called by the Ops Config Manager when it recreates the HTTP server.</p> |
| <p>Each HTTP endpoint is mapped to a function which closes around the shared data objects it needs, and takes the request data it needs (such as query parameters). Each endpoint function resides in its own file in <code class="docutils literal"><span class="pre">traffic_monitor/datareq/</span></code>. Because each Go HTTP routing function must be a <code class="docutils literal"><span class="pre">http.HandlerFunc</span></code>, wrapper functions take the endpoint functions and return <code class="docutils literal"><span class="pre">http.HandlerFunc</span></code> functions which call them, and which are stored in the dispatch map, to be registered with the HTTP server.</p> |
| </div> |
| <div class="section" id="shared-data"> |
| <h3>Shared Data<a class="headerlink" href="#shared-data" title="Permalink to this headline">¶</a></h3> |
| <p>Processed and aggregated data must be shared between the end of the stat and health processing pipelines, and HTTP requests. The CSP paradigm of idiomatic Go does not work efficiently with storing and sharing state. While not idiomatic Go, shared mutexed data structures are faster and simpler than CSP manager microthreads for each data object.</p> |
| <p>Traffic Monitor has many threadsafe shared data types and objects. All shared data objects can be seen in <code class="docutils literal"><span class="pre">manager/manager.go:Start()</span></code>, where they are created and passed to the various pipeline stage microthreads that need them. Their respective types all include the word <code class="docutils literal"><span class="pre">Threadsafe</span></code>, and can be found in <code class="docutils literal"><span class="pre">traffic_monitor/threadsafe/</span></code> as well as, for dependency reasons, various appropriate directories.</p> |
| <p>Currently, all Threadsafe shared data types use mutexes. In the future, these could be changed to lock-free or wait-free structures, if the performance needs outweighed the readability and correctness costs. They could also easily be changed to internally be manager microthreads and channels, if being idiomatic were deemed more important than readability or performance.</p> |
| </div> |
| </div> |
| <div class="section" id="formatting-conventions"> |
| <h2>Formatting Conventions<a class="headerlink" href="#formatting-conventions" title="Permalink to this headline">¶</a></h2> |
| <p>Go code should be formatted with <code class="docutils literal"><span class="pre">gofmt</span></code>. See also <code class="docutils literal"><span class="pre">CONTRIBUTING.md</span></code>.</p> |
| </div> |
| <div class="section" id="installing-the-developer-environment"> |
| <h2>Installing The Developer Environment<a class="headerlink" href="#installing-the-developer-environment" title="Permalink to this headline">¶</a></h2> |
| <p>To install the Traffic Monitor Developer environment:</p> |
| <ol class="arabic simple"> |
| <li>Install <cite>go</cite> version 1.7 or greater, from <a class="reference external" href="https://golang.org/doc/install">https://golang.org/doc/install</a> and <a class="reference external" href="https://golang.org/doc/code.html">https://golang.org/doc/code.html</a></li> |
| <li>Clone the traffic_control repository using Git, into <code class="docutils literal"><span class="pre">$GOPATH/src/github.com/apache/incubator-trafficcontrol</span></code></li> |
| <li>Change directories into <code class="docutils literal"><span class="pre">$GOPATH/src/github.com/apache/incubator-trafficcontrol/traffic_monitor_golang/traffic_monitor</span></code></li> |
| <li>Run <code class="docutils literal"><span class="pre">./build.sh</span></code></li> |
| </ol> |
| </div> |
| <div class="section" id="test-cases"> |
| <h2>Test Cases<a class="headerlink" href="#test-cases" title="Permalink to this headline">¶</a></h2> |
| <p>Tests can be executed by running <code class="docutils literal"><span class="pre">go</span> <span class="pre">test</span> <span class="pre">./...</span></code> at the root of the <code class="docutils literal"><span class="pre">traffic_monitor_golang</span></code> project.</p> |
| </div> |
| <div class="section" id="api"> |
| <h2>API<a class="headerlink" href="#api" title="Permalink to this headline">¶</a></h2> |
| <p><a class="reference internal" href="traffic_monitor/traffic_monitor_api.html#reference-tm-api"><span class="std std-ref">Traffic Monitor APIs</span></a></p> |
| <div class="toctree-wrapper compound"> |
| </div> |
| </div> |
| </div> |
| |
| |
| </div> |
| <footer> |
| |
| <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> |
| |
| <a href="traffic_stats.html" class="btn btn-neutral float-right" title="Traffic Stats">Next <span class="fa fa-arrow-circle-right"></span></a> |
| |
| |
| <a href="traffic_monitor/traffic_monitor_api.html" class="btn btn-neutral" title="Traffic Monitor APIs"><span class="fa fa-arrow-circle-left"></span> Previous</a> |
| |
| </div> |
| |
| |
| <hr/> |
| |
| <div role="contentinfo"> |
| <p> |
| </p> |
| </div> |
| Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. |
| |
| </footer> |
| |
| </div> |
| </div> |
| |
| </section> |
| |
| </div> |
| |
| |
| |
| |
| |
| <script type="text/javascript"> |
| var DOCUMENTATION_OPTIONS = { |
| URL_ROOT:'../', |
| VERSION:'2.1-dev', |
| COLLAPSE_INDEX:false, |
| FILE_SUFFIX:'.html', |
| HAS_SOURCE: true, |
| SOURCELINK_SUFFIX: '' |
| }; |
| </script> |
| <script type="text/javascript" src="../_static/jquery.js"></script> |
| <script type="text/javascript" src="../_static/underscore.js"></script> |
| <script type="text/javascript" src="../_static/doctools.js"></script> |
| |
| |
| |
| |
| |
| <script type="text/javascript" src="../_static/js/theme.js"></script> |
| |
| |
| |
| |
| <script type="text/javascript"> |
| jQuery(function () { |
| SphinxRtdTheme.StickyNav.enable(); |
| }); |
| </script> |
| |
| |
| </body> |
| </html> |
