| <!DOCTYPE html>
|
| <html lang="en">
|
| <head>
|
| <meta charset="utf-8">
|
| <meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
|
|
|
|
|
| <meta name="author" content="The Apache Software Foundation">
|
|
|
|
|
| <meta name="keywords" content="python,libcloud,cloud,cloud computing,rackspace,amazon ec2,cloudfiles,openstack,cloudstack" />
|
|
|
|
|
|
|
| <title>Apache Libcloud | Apache Libcloud</title>
|
|
|
|
|
| <!-- fav icons -->
|
| <link rel="shortcut icon" href="/images/favicon.png" />
|
| <link rel="apple-touch-icon" href="/images/apple-touch-icon.png" />
|
| <link rel="apple-touch-icon-precomposed" href="/images/apple-touch-icon.png" />
|
|
|
| <link href="/blog/atom.xml" type="application/atom+xml" rel="alternate" title="Apache Libcloud Blog Feed" />
|
|
|
| <!-- Facebook OpenGraph tags -->
|
| <meta content="Apache Libcloud" property="og:site_name">
|
|
|
| <meta content="Apache Libcloud" property="og:title">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| <meta content="website" property="og:type">
|
|
|
|
|
|
|
| <meta content="https://libcloud.apache.org/blog/tags/tutorial" property="og:url">
|
|
|
|
|
|
|
|
|
|
|
| <link href='/assets/global-49923225b15584efebd7a337daac5102.css' rel='stylesheet' type='text/css' />
|
| </head>
|
|
|
| <body data-spy="scroll" data-target=".sidebar-nav" data-offset="80">
|
| <nav class="navbar navbar-fixed-top navbar-inverse" role="navigation">
|
| <div class="container">
|
| <div class="navbar-header">
|
| <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-ex1-collapse">
|
| <span class="sr-only">Toggle navigation</span>
|
| <span class="icon-bar"></span>
|
| <span class="icon-bar"></span>
|
| <span class="icon-bar"></span>
|
| </button>
|
| <a class="navbar-brand" href="/index.html"><img src="/images/libcloud_logo.png" class="navbar-logo" /> Apache Libcloud</a>
|
| </div>
|
| <div class="collapse navbar-collapse navbar-ex1-collapse">
|
| <ul class="nav navbar-nav">
|
|
|
|
|
|
|
| <li ><a href="/" >Home</a></li>
|
|
|
|
|
| <li ><a href="/about.html" >About</a></li>
|
|
|
|
|
| <li ><a href="/getting-started.html" >Quick Start</a></li>
|
|
|
|
|
| <li ><a href="https://libcloud.readthedocs.org/en/stable/" target="_blank">Documentation</a></li>
|
|
|
|
|
| <li ><a href="/downloads.html" >Downloads</a></li>
|
|
|
|
|
| <li ><a href="/community.html" >Community</a></li>
|
|
|
|
|
| <li ><a href="/blog/" >Blog</a></li>
|
|
|
| </ul>
|
|
|
| <div class="material-switch pull-right">
|
| <input id="theme-switch" name="theme-switch" type="checkbox" onclick="modeSwitcher()"/>
|
| <label for="theme-switch" class="label-default"></label>
|
| <span id="theme-toggle" class="theme-switch">Dark mode</span>
|
| </div>
|
|
|
| </div><!-- /.navbar-collapse -->
|
| </div><!-- /.container -->
|
| </nav>
|
|
|
| <div class="container main-container">
|
| <div class="row section page-content"> |
| <div class="col-lg-8 col-lg-offset-2"> |
| <h1>Tag: tutorial</h1> |
| |
| |
| |
| |
| <div class="post"> |
| |
| <h2><a href="/blog/2017/04/09/async-rfc.html">Have your say - async support in Apache Libcloud</a></h2> |
| |
| <span class="post-date-author">By Anthony Shaw on Apr 09, 2017</span> |
| |
| <div class="post-content"> |
| <p>One of the big requests whilst we were replacing <code class="language-plaintext highlighter-rouge">httplib</code> with the <code class="language-plaintext highlighter-rouge">requests</code> package in 2.0 was why didn’t |
| we use a HTTP library that supports <em>asynchronous</em> API calls.</p> |
| |
| <p>The intention for 2.0 and replacing the HTTP backend classes was to improve the usability of the project, by making SSL |
| certificates easier to manage, improving the maintainability of our source code by using an active 3rd party package and |
| also improving performance and stability.</p> |
| |
| <p>Apache Libcloud already has documentation on threaded libraries like gevent and callback-based libraries like Twisted, see |
| <a href="https://libcloud.readthedocs.io/en/latest/other/using-libcloud-in-multithreaded-and-async-environments.html">using libcloud in multithreaded environments</a> |
| for examples.</p> |
| |
| <p><a href="https://www.python.org/dev/peps/pep-0492/#">PEP 492</a>, implemented in Python 3.5 provides a new coroutine protocol using methods, |
| <code class="language-plaintext highlighter-rouge">__await__</code> for classes, a coroutine method wrapper, or a method that returns a coroutine object. |
| Also async <a href="https://www.python.org/dev/peps/pep-0492/#asynchronous-iterators-and-async-for">iterators</a> and <a href="https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with">context managers</a> |
| have been introduced.</p> |
| |
| <p>We would like to take advantage of the new language features by offering APIs in Apache Libcloud without breaking backward compatibility and |
| compatibility for users of <Python 3.5.</p> |
| |
| <p>Use cases for this would be:</p> |
| |
| <ul> |
| <li>Being able to fetch <code class="language-plaintext highlighter-rouge">Node</code> or <code class="language-plaintext highlighter-rouge">StorageObject</code>s from multiple geographies or drivers simultaneously.</li> |
| <li>Being able to quickly upload or download storage objects by parallelizing operations on the <code class="language-plaintext highlighter-rouge">StorageDriver</code>.</li> |
| <li>Being able to call a long-running API method (e.g. generate report), whilst running other code.</li> |
| </ul> |
| |
| <h2 id="design-1---async-context-managers-pr-1016">Design 1 - async context managers <a href="https://github.com/apache/libcloud/pull/1016">PR 1016</a></h2> |
| |
| <p>This design would allow drivers to operate in 2 modes, the first is for synchronous method calls, they return list or object |
| data as per usual. The second mode, API methods like <code class="language-plaintext highlighter-rouge">NodeDriver.list_nodes</code> would return a <a href="https://www.python.org/dev/peps/pep-0492/#coroutine-objects">coroutine object</a> |
| and could be awaited or gathered using an event loop.</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code><span class="kn">import</span> <span class="nn">asyncio</span> |
| |
| <span class="kn">from</span> <span class="nn">integration.driver.test</span> <span class="kn">import</span> <span class="n">TestNodeDriver</span> |
| <span class="kn">from</span> <span class="nn">libcloud.async_util</span> <span class="kn">import</span> <span class="n">AsyncSession</span> |
| |
| <span class="n">driver</span> <span class="o">=</span> <span class="n">TestNodeDriver</span><span class="p">(</span><span class="s">'apache'</span><span class="p">,</span> <span class="s">'libcloud'</span><span class="p">)</span> |
| |
| <span class="k">async</span> <span class="k">def</span> <span class="nf">run</span><span class="p">():</span> |
| <span class="c1"># regular API call |
| </span> <span class="n">nodes</span> <span class="o">=</span> <span class="n">driver</span><span class="p">.</span><span class="n">list_nodes</span><span class="p">()</span> |
| |
| <span class="k">async</span> <span class="k">with</span> <span class="n">AsyncSession</span><span class="p">(</span><span class="n">driver</span><span class="p">)</span> <span class="k">as</span> <span class="n">async_instance</span><span class="p">:</span> |
| <span class="n">nodes</span> <span class="o">=</span> <span class="k">await</span> <span class="n">async_instance</span><span class="p">.</span><span class="n">list_nodes</span><span class="p">()</span> |
| |
| <span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">nodes</span><span class="p">)</span> <span class="o">==</span> <span class="mi">2</span> |
| |
| <span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="p">.</span><span class="n">get_event_loop</span><span class="p">()</span> |
| <span class="n">loop</span><span class="p">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">run</span><span class="p">())</span> |
| <span class="n">loop</span><span class="p">.</span><span class="n">close</span><span class="p">()</span> |
| </code></pre> |
| </div> |
| </div> |
| |
| <h2 id="design-2---additional-methods-in-each-driver-for-coroutines-pr-1027">Design 2 - Additional methods in each driver for coroutines <a href="https://github.com/apache/libcloud/pull/1027">PR 1027</a></h2> |
| |
| <p>This is the second design concept for async support in Libcloud.</p> |
| |
| <p>The concept here is to have Asynchronous Mixins, <code class="language-plaintext highlighter-rouge">LibcloudConnection</code> uses requests and <code class="language-plaintext highlighter-rouge">LibcloudAsyncConnection</code> uses aiohttp for async transport <a href="https://github.com/tonybaloney/libcloud/blob/d4fe097476d2f02941e17d5e1b1d405fcf44c0f7/libcloud/connection_async.py#L22-L42">see</a></p> |
| |
| <p>The LibcloudAsyncConnection is an implementation detail of AsyncConnection, which is the API for the drivers to consume <a href="https://github.com/tonybaloney/libcloud/blob/d4fe097476d2f02941e17d5e1b1d405fcf44c0f7/libcloud/common/base.py#L742-L778">see</a></p> |
| |
| <p>The drivers then use this mixin for their custom connection classes, e.g.</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code> |
| <span class="k">class</span> <span class="nc">GoogleStorageConnection</span><span class="p">(</span><span class="n">ConnectionUserAndKey</span><span class="p">,</span> <span class="n">AsyncConnection</span><span class="p">):</span> |
| <span class="p">...</span> |
| |
| </code></pre> |
| </div> |
| </div> |
| |
| <p>They then inherit from <code class="language-plaintext highlighter-rouge">libcloud.storage.base.StorageAsyncDriver</code>, which uses a new set of base methods, e.g. <code class="language-plaintext highlighter-rouge">iterate_containers_async</code> and can be implemented like this:</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code> <span class="k">async</span> <span class="k">def</span> <span class="nf">iterate_containers_async</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| <span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="p">.</span><span class="n">connection</span><span class="p">.</span><span class="n">request_async</span><span class="p">(</span><span class="s">'/'</span><span class="p">)</span> |
| <span class="k">if</span> <span class="n">response</span><span class="p">.</span><span class="n">status</span> <span class="o">==</span> <span class="n">httplib</span><span class="p">.</span><span class="n">OK</span><span class="p">:</span> |
| <span class="n">containers</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">_to_containers</span><span class="p">(</span><span class="n">obj</span><span class="o">=</span><span class="n">response</span><span class="p">.</span><span class="nb">object</span><span class="p">,</span> |
| <span class="n">xpath</span><span class="o">=</span><span class="s">'Buckets/Bucket'</span><span class="p">)</span> |
| <span class="k">return</span> <span class="n">containers</span> |
| |
| <span class="k">raise</span> <span class="n">LibcloudError</span><span class="p">(</span><span class="s">'Unexpected status code: %s'</span> <span class="o">%</span> <span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="n">status</span><span class="p">),</span> |
| <span class="n">driver</span><span class="o">=</span><span class="bp">self</span><span class="p">)</span> |
| </code></pre> |
| </div> |
| </div> |
| |
| <p>Now the consumer can more or less do this:</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code><span class="kn">from</span> <span class="nn">libcloud.storage.providers</span> <span class="kn">import</span> <span class="n">get_driver</span> |
| <span class="kn">from</span> <span class="nn">libcloud.storage.types</span> <span class="kn">import</span> <span class="n">Provider</span> |
| |
| <span class="kn">import</span> <span class="nn">asyncio</span> |
| |
| <span class="n">GoogleStorageDriver</span> <span class="o">=</span> <span class="n">get_driver</span><span class="p">(</span><span class="n">Provider</span><span class="p">.</span><span class="n">GOOGLE_STORAGE</span><span class="p">)</span> |
| <span class="n">driver</span> <span class="o">=</span> <span class="n">GoogleStorageDriver</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="n">KEY</span><span class="p">,</span> <span class="n">secret</span><span class="o">=</span><span class="n">SECRET</span><span class="p">)</span> |
| |
| <span class="k">def</span> <span class="nf">do_stuff_with_object</span><span class="p">(</span><span class="n">obj</span><span class="p">):</span> |
| <span class="k">print</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span> |
| |
| <span class="k">async</span> <span class="k">def</span> <span class="nf">run</span><span class="p">():</span> |
| <span class="n">tasks</span> <span class="o">=</span> <span class="p">[]</span> |
| <span class="k">async</span> <span class="k">for</span> <span class="n">container</span> <span class="ow">in</span> <span class="n">driver</span><span class="p">.</span><span class="n">iterate_containers_async</span><span class="p">():</span> |
| <span class="k">async</span> <span class="k">for</span> <span class="n">obj</span> <span class="ow">in</span> <span class="n">driver</span><span class="p">.</span><span class="n">iterate_container_objects_async</span><span class="p">(</span><span class="n">container</span><span class="p">):</span> |
| <span class="n">tasks</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">asyncio</span><span class="p">.</span><span class="n">ensure_future</span><span class="p">(</span><span class="n">do_stuff_with_object</span><span class="p">(</span><span class="n">obj</span><span class="p">)))</span> |
| <span class="k">await</span> <span class="n">asyncio</span><span class="p">.</span><span class="n">gather</span><span class="p">(</span><span class="o">*</span><span class="n">tasks</span><span class="p">)</span> |
| |
| <span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="p">.</span><span class="n">get_event_loop</span><span class="p">()</span> |
| <span class="n">loop</span><span class="p">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">run</span><span class="p">())</span> |
| <span class="n">loop</span><span class="p">.</span><span class="n">close</span><span class="p">()</span> |
| </code></pre> |
| </div> |
| </div> |
| |
| <h2 id="design-3---initializer-with-async-mode">Design 3 - Initializer with “<em>async</em>” mode</h2> |
| |
| <p>This option is similar to 2, except that if a driver is instantiated with “<code class="language-plaintext highlighter-rouge">async=True</code>”, |
| then all driver class methods would return coroutine objects. Internally, it would |
| patch the Connection class with the AsyncConnection class.</p> |
| |
| <p>The downside of this is that all method calls to a driver would need to be awaited or used |
| by an event loop.</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code><span class="kn">from</span> <span class="nn">libcloud.storage.providers</span> <span class="kn">import</span> <span class="n">get_driver</span> |
| <span class="kn">from</span> <span class="nn">libcloud.storage.types</span> <span class="kn">import</span> <span class="n">Provider</span> |
| |
| <span class="kn">import</span> <span class="nn">asyncio</span> |
| |
| <span class="n">GoogleStorageDriver</span> <span class="o">=</span> <span class="n">get_driver</span><span class="p">(</span><span class="n">Provider</span><span class="p">.</span><span class="n">GOOGLE_STORAGE</span><span class="p">)</span> |
| <span class="n">driver</span> <span class="o">=</span> <span class="n">GoogleStorageDriver</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="n">KEY</span><span class="p">,</span> <span class="n">secret</span><span class="o">=</span><span class="n">SECRET</span><span class="p">,</span> <span class="k">async</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> |
| |
| <span class="k">def</span> <span class="nf">do_stuff_with_object</span><span class="p">(</span><span class="n">obj</span><span class="p">):</span> |
| <span class="k">print</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span> |
| |
| <span class="k">async</span> <span class="k">def</span> <span class="nf">run</span><span class="p">():</span> |
| <span class="n">tasks</span> <span class="o">=</span> <span class="p">[]</span> |
| <span class="k">async</span> <span class="k">for</span> <span class="n">container</span> <span class="ow">in</span> <span class="n">driver</span><span class="p">.</span><span class="n">iterate_containers</span><span class="p">():</span> |
| <span class="k">async</span> <span class="k">for</span> <span class="n">obj</span> <span class="ow">in</span> <span class="n">driver</span><span class="p">.</span><span class="n">iterate_container_objects</span><span class="p">(</span><span class="n">container</span><span class="p">):</span> |
| <span class="n">tasks</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">asyncio</span><span class="p">.</span><span class="n">ensure_future</span><span class="p">(</span><span class="n">do_stuff_with_object</span><span class="p">(</span><span class="n">obj</span><span class="p">)))</span> |
| <span class="k">await</span> <span class="n">asyncio</span><span class="p">.</span><span class="n">gather</span><span class="p">(</span><span class="o">*</span><span class="n">tasks</span><span class="p">)</span> |
| |
| <span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="p">.</span><span class="n">get_event_loop</span><span class="p">()</span> |
| <span class="n">loop</span><span class="p">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">run</span><span class="p">())</span> |
| <span class="n">loop</span><span class="p">.</span><span class="n">close</span><span class="p">()</span> |
| </code></pre> |
| </div> |
| </div> |
| |
| <h1 id="give-us-feedback">Give us feedback</h1> |
| |
| <p>Got a better idea? Have an API or design, the question we’re asking is |
| “if you wanted to use Libcloud for an async application, what would the code look like?” This helps us design |
| the API and the implementation details can follow.</p> |
| |
| <p>Feel free to comment on the mailing list or on the pull requests, or raise your own pull-request with an API design.</p> |
| |
| </div> |
| |
| <div class="row section post-meta"> |
| <div class="col-md-12 post-tags"> |
| <p>Tags: <a href="/blog/tags/news.html" rel="tag">news</a>, <a href="/blog/tags/tutorial.html" rel="tag">tutorial</a></p> |
| </div> |
| </div> |
| </div> |
| |
| |
| |
| |
| <div class="post"> |
| |
| <h2><a href="/blog/2016/04/06/requests-support.html">Experimental support for the requests package</a></h2> |
| |
| <span class="post-date-author">By Anthony Shaw on Apr 06, 2016</span> |
| |
| <div class="post-content"> |
| <h2 id="background">Background</h2> |
| |
| <p>I’ve just pushed a branch of the latest version of libcloud using the popular <code class="language-plaintext highlighter-rouge">requests</code> package by Kenneth Reitz instead of our home-rolled HTTP client library.</p> |
| |
| <p>This article is for both users and developers of libcloud. If you want to give feedback, please join the developer mailing list.</p> |
| |
| <h2 id="why">Why?</h2> |
| |
| <ul> |
| <li>requests is the defacto standard - it would be in the standard library but agreed against to allow it to develop faster https://github.com/kennethreitz/requests/issues/2424</li> |
| <li>it works with python 2.6->3.5</li> |
| <li>Our SSL experience has a lot to be desired for Windows users, having to download the CA cert package and setting environment variables just to get SSL working</li> |
| <li>Developers can use requests_mock for deeper integration testing</li> |
| <li>less code to maintain</li> |
| <li>the role of libcloud is for cloud abstraction, we provide no value in writing and maintaining our own HTTP client library</li> |
| </ul> |
| |
| <h2 id="benefits-of-requests">Benefits of requests</h2> |
| |
| <p>There are a number of benefits to having a requests package</p> |
| |
| <ul> |
| <li>The client library code is smaller, leaner and simpler.</li> |
| <li>Requests has built in decompression support, we no longer need to support this</li> |
| <li>Requests has built in RAW download, upload support, helping with our storage drivers</li> |
| </ul> |
| |
| <h2 id="implications-of-the-change">Implications of the change</h2> |
| |
| <ul> |
| <li>There are no longer 2 classes (<code class="language-plaintext highlighter-rouge">LibcloudHTTPSConnection</code> and <code class="language-plaintext highlighter-rouge">LibcloudHTTPConnection</code>) to be provided to each driver, they are now 1 class - <code class="language-plaintext highlighter-rouge">LibcloudConnection</code>. You probably won’t notice this because it is a property of the <code class="language-plaintext highlighter-rouge">Connection</code> class, but |
| if you are developing or extending functionality then it is implicated.</li> |
| <li>Unit tests will look slightly different (see below)</li> |
| <li>This change broke 4200 unit tests (out of 6340)! I’ve since fixed them all since they were coupled to the original implementation, but now I don’t know if all of tests are valid.</li> |
| </ul> |
| |
| <h2 id="testing-with-requests">Testing with requests</h2> |
| |
| <p>Unit tests that were written like this:</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code><span class="k">class</span> <span class="nc">DigitalOceanTests</span><span class="p">(</span><span class="n">LibcloudTestCase</span><span class="p">):</span> |
| |
| <span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| <span class="n">DigitalOceanBaseDriver</span><span class="p">.</span><span class="n">connectionCls</span><span class="p">.</span><span class="n">conn_classes</span> <span class="o">=</span> \ |
| <span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="n">DigitalOceanMockHttp</span><span class="p">)</span> |
| <span class="n">DigitalOceanMockHttp</span><span class="p">.</span><span class="nb">type</span> <span class="o">=</span> <span class="bp">None</span> |
| <span class="bp">self</span><span class="p">.</span><span class="n">driver</span> <span class="o">=</span> <span class="n">DigitalOceanBaseDriver</span><span class="p">(</span><span class="o">*</span><span class="n">DIGITALOCEAN_v1_PARAMS</span><span class="p">)</span> |
| </code></pre> |
| </div> |
| </div> |
| |
| <p>Because of the change have been modified to (I updated all of them - so this is just for future reference)</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code><span class="k">class</span> <span class="nc">DigitalOceanTests</span><span class="p">(</span><span class="n">LibcloudTestCase</span><span class="p">):</span> |
| |
| <span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| <span class="n">DigitalOceanBaseDriver</span><span class="p">.</span><span class="n">connectionCls</span><span class="p">.</span><span class="n">conn_class</span> <span class="o">=</span> <span class="n">DigitalOceanMockHttp</span> |
| <span class="n">DigitalOceanMockHttp</span><span class="p">.</span><span class="nb">type</span> <span class="o">=</span> <span class="bp">None</span> |
| <span class="bp">self</span><span class="p">.</span><span class="n">driver</span> <span class="o">=</span> <span class="n">DigitalOceanBaseDriver</span><span class="p">(</span><span class="o">*</span><span class="n">DIGITALOCEAN_v1_PARAMS</span><span class="p">)</span> |
| </code></pre> |
| </div> |
| </div> |
| |
| <h1 id="check-it-out">Check it out!</h1> |
| |
| <p>The package is on my personal apache site, you can download it and install it in a virtualenv for testing.</p> |
| |
| <p><code class="language-plaintext highlighter-rouge">pip install -e http://people.apache.org/~anthonyshaw/libcloud/1.0.0-rc2-requests/apache-libcloud-1.0.0-rc2-requests.zip@feature#egg=apache-libcloud</code></p> |
| |
| <p>The hashes are my <a href="http://people.apache.org/~anthonyshaw/libcloud/1.0.0-rc2-requests/">apache space</a></p> |
| |
| <p>Have a look at the <a href="https://github.com/apache/libcloud/pull/728/files">PR and the change set</a> for a list of changes</p> |
| |
| <h1 id="what-might-break">What might break?</h1> |
| |
| <p>What I’m really looking for is for users of Libcloud to take 15 minutes, an existing (working) libcloud script, install this package in a virtualenv and just validate |
| that there are no regression bugs with this change.</p> |
| |
| <p>I’m particularly sceptical about the storage drivers.</p> |
| |
| <p>Once we have enough community feedback, we will propose a vote to merge this into trunk for future release.</p> |
| |
| <h2 id="credit">Credit</h2> |
| |
| <p>Credit to dz0ny on IRC for contributing some of the requests patch.</p> |
| |
| </div> |
| |
| <div class="row section post-meta"> |
| <div class="col-md-12 post-tags"> |
| <p>Tags: <a href="/blog/tags/news.html" rel="tag">news</a>, <a href="/blog/tags/api.html" rel="tag">API</a>, <a href="/blog/tags/tutorial.html" rel="tag">tutorial</a></p> |
| </div> |
| </div> |
| </div> |
| |
| |
| |
| |
| <div class="post"> |
| |
| <h2><a href="/blog/2016/02/05/libcloud-containers-example.html">Using the container abstraction API in 1.0.0-pre1</a></h2> |
| |
| <span class="post-date-author">By Anthony Shaw on Feb 05, 2016</span> |
| |
| <div class="post-content"> |
| <h2 id="background">Background</h2> |
| |
| <p>Containers are the talk of the town, you can’t escape an event or meetup without someone talking about containers. The lessons we |
| learnt with compute abstraction are applying widely with containers in 2016. APIs are not consistent between clouds, designs are not |
| standardised and yet, users are trying to consume multiple services.</p> |
| |
| <p>We introduced Container-as-a-Service support in <a href="http://libcloud.apache.org/blog/2016/01/26/libcloud-1-0-0-pre1-released.html">1.0.0-pre1</a>, a community pre-release with the intention of sparking feedback from |
| the open-source community about the design and the implementation of 4 example drivers :</p> |
| |
| <ul> |
| <li>Docker</li> |
| <li>Joyent Triton</li> |
| <li>Amazon EC2 Container Service</li> |
| <li>Google Kubernetes</li> |
| </ul> |
| |
| <p>In this tutorial we’re going to explore how to do this:</p> |
| |
| <div class="imginline"> |
| <p><img src="/images/posts/2016-02-05-containers/container_cloud_example.png" class="img-responsive inline" /></p> |
| <p class="img-caption">Deploying containers across platforms.</p> |
| </div> |
| |
| <p>Pulling images from the Docker hub, deploying to Docker, Kubernetes and Amazon ECS then auditing them with a single query.</p> |
| |
| <h2 id="getting-started-with-100-pre1">Getting Started with 1.0.0-pre1</h2> |
| |
| <p>First off, let’s install the new packages, you probably want to do this within a virtualenv if you’re using Apache Libcloud for other projects.</p> |
| |
| <p>So run these commands at a Linux Shell to create a virtualenv called ‘containers’ and install the pre-release packages into that environment.</p> |
| |
| <div class="language-bash highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code> virtualenv containers |
| <span class="nb">cd </span>containers |
| <span class="nb">source </span>bin/activate |
| pip <span class="nb">install </span>apache-libcloud<span class="o">==</span>1.0.0-pre1 |
| </code></pre> |
| </div> |
| </div> |
| |
| <p>Now you can start using this package with a test script, let’s create one called containers.py</p> |
| |
| <div class="language-bash highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code> <span class="nb">touch </span>containers.py |
| </code></pre> |
| </div> |
| </div> |
| |
| <p>Using your favourite text editor, update that file to import the 1.0.0-pre1 libraries and the factory methods for instantiating containers.</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code> <span class="kn">from</span> <span class="nn">libcloud.container.providers</span> <span class="kn">import</span> <span class="n">get_driver</span> |
| <span class="kn">from</span> <span class="nn">libcloud.container.types</span> <span class="kn">import</span> <span class="n">Provider</span> |
| </code></pre> |
| </div> |
| </div> |
| |
| <p><code class="language-plaintext highlighter-rouge">get_driver</code> is a factory method as with all libcloud APIs, you call this method with the <code class="language-plaintext highlighter-rouge">Provider</code> that you want to instantiate. Our options are:</p> |
| |
| <ul> |
| <li><code class="language-plaintext highlighter-rouge">Provider.DOCKER</code> - Standalone Docker API</li> |
| <li><code class="language-plaintext highlighter-rouge">Provider.KUBERNETES</code> - Kubernetes Cluster endpoint</li> |
| <li><code class="language-plaintext highlighter-rouge">Provider.JOYENT</code> - Joyent Triton Public API</li> |
| <li><code class="language-plaintext highlighter-rouge">Provider.ECS</code> - Amazon EC2 Container Service</li> |
| </ul> |
| |
| <p>Calling <code class="language-plaintext highlighter-rouge">get_driver</code> will return a reference to the driver class that you requested. You can then instantiate that class into an object using the |
| contructor. This is always a set of parameters for setting the host or region, the authentication and any other options.</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code> <span class="n">driver</span> <span class="o">=</span> <span class="n">get_driver</span><span class="p">(</span><span class="n">Provider</span><span class="p">.</span><span class="n">DOCKER</span><span class="p">)</span> |
| </code></pre> |
| </div> |
| </div> |
| |
| <p>Now we can call our driver and get an instance of it called <code class="language-plaintext highlighter-rouge">docker_driver</code> and use that to deploy a container. For Docker you need the pem files on the server, |
| the host (IP or FQDN) and the port.</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code> <span class="n">docker_driver</span> <span class="o">=</span> <span class="n">driver</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="s">'https://198.61.239.128'</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="mi">4243</span><span class="p">,</span> |
| <span class="n">key_file</span><span class="o">=</span><span class="s">'key.pem'</span><span class="p">,</span> <span class="n">cert_file</span><span class="o">=</span><span class="s">'cert.pem'</span><span class="p">)</span> |
| </code></pre> |
| </div> |
| </div> |
| |
| <p>Docker requires that images are available in the image database before they can be deployed as containers. With Kubernetes and Amazon ECS this step is not required |
| as when you deploy a container it carries out that download for you.</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code> <span class="n">image</span> <span class="o">=</span> <span class="n">driver</span><span class="p">.</span><span class="n">install_image</span><span class="p">(</span><span class="s">'tomcat:8.0'</span><span class="p">)</span> |
| </code></pre> |
| </div> |
| </div> |
| |
| <p>Now that Docker has the version 8.0 image of Apache Tomcat, you can deploy this as a container called <code class="language-plaintext highlighter-rouge">my_tomcat_container</code>. Tomcat runs on TCP/8080 by default so we |
| want to bind that port for our container using an optional parameter <code class="language-plaintext highlighter-rouge">port_bindings</code></p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code> <span class="n">bindings</span> <span class="o">=</span> <span class="p">{</span> <span class="s">"22/tcp"</span><span class="p">:</span> <span class="p">[{</span> <span class="s">"HostPort"</span><span class="p">:</span> <span class="s">"11022"</span> <span class="p">}]</span> <span class="p">}</span> |
| <span class="n">container</span> <span class="o">=</span> <span class="n">driver</span><span class="p">.</span><span class="n">deploy_container</span><span class="p">(</span><span class="s">'my_tomcat_container'</span><span class="p">,</span> <span class="n">image</span><span class="p">,</span> <span class="n">port_bindings</span><span class="o">=</span><span class="n">bindings</span><span class="p">)</span> |
| </code></pre> |
| </div> |
| </div> |
| |
| <p>This will have deployed the container and started it up for you, you can disable the automatic startup by using <code class="language-plaintext highlighter-rouge">start=False</code> as a keyword argument. You can now call upon this container and |
| run methods, <code class="language-plaintext highlighter-rouge">restart</code>, <code class="language-plaintext highlighter-rouge">start</code>, <code class="language-plaintext highlighter-rouge">stop</code> and <code class="language-plaintext highlighter-rouge">destroy</code>.</p> |
| |
| <p>For example, to blow away that test container:</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code> <span class="n">container</span><span class="p">.</span><span class="n">destroy</span><span class="p">()</span> |
| </code></pre> |
| </div> |
| </div> |
| |
| <h2 id="crossing-the-streams-calling-kubernetes-and-amazon-ec2-container-service">Crossing the streams; calling Kubernetes and Amazon EC2 Container Service</h2> |
| |
| <p>With Docker we saw that we needed to “pull” the image before we deployed it. Kubernetes and Amazon ECS don’t have that requirement, but as a safeguard you can query the Docker Hub API using a |
| utility class provided</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code> <span class="kn">from</span> <span class="nn">libcloud.container.utils.docker</span> <span class="kn">import</span> <span class="n">HubClient</span> |
| <span class="n">hub</span> <span class="o">=</span> <span class="n">HubClient</span><span class="p">()</span> |
| <span class="n">image</span> <span class="o">=</span> <span class="n">hub</span><span class="p">.</span><span class="n">get_image</span><span class="p">(</span><span class="s">'tomcat'</span><span class="p">,</span> <span class="s">'8.0'</span><span class="p">)</span> |
| </code></pre> |
| </div> |
| </div> |
| |
| <p>Now <code class="language-plaintext highlighter-rouge">image</code> can be used to deploy to any driver instance that you create. Let’s try that against Kubernetes and ECS.</p> |
| |
| <h3 id="amazon-ecs">Amazon ECS</h3> |
| |
| <p>Before you run this example, you will need an API key and the permissions for that key to have the <code class="language-plaintext highlighter-rouge">AmazonEC2ContainerServiceFullAccess</code> role. <code class="language-plaintext highlighter-rouge">ap-southeast-2</code> is my nearest region, but you can |
| swap this out for any of the Amazon public regions that have the ECS service available.</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code> <span class="n">e_cls</span> <span class="o">=</span> <span class="n">get_driver</span><span class="p">(</span><span class="n">Provider</span><span class="p">.</span><span class="n">ECS</span><span class="p">)</span> |
| <span class="n">ecs</span> <span class="o">=</span> <span class="n">e_cls</span><span class="p">(</span><span class="n">access_id</span><span class="o">=</span><span class="s">'SDHFISJDIFJSIDFJ'</span><span class="p">,</span> |
| <span class="n">secret</span><span class="o">=</span><span class="s">'THIS_IS)+_MY_SECRET_KEY+I6TVkv68o4H'</span><span class="p">,</span> |
| <span class="n">region</span><span class="o">=</span><span class="s">'ap-southeast-2'</span><span class="p">)</span> |
| </code></pre> |
| </div> |
| </div> |
| |
| <p>ECS and Kubernetes both support some form of grouping or clustering for your containers. This is available as <code class="language-plaintext highlighter-rouge">create_cluster</code>, <code class="language-plaintext highlighter-rouge">list_cluster</code>.</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code> <span class="n">cluster</span> <span class="o">=</span> <span class="n">ecs</span><span class="p">.</span><span class="n">create_cluster</span><span class="p">(</span><span class="s">'default'</span><span class="p">)</span> |
| <span class="n">container</span> <span class="o">=</span> <span class="n">ecs</span><span class="p">.</span><span class="n">deploy_container</span><span class="p">(</span> |
| <span class="n">cluster</span><span class="o">=</span><span class="n">cluster</span><span class="p">,</span> |
| <span class="n">name</span><span class="o">=</span><span class="s">'hello-world'</span><span class="p">,</span> |
| <span class="n">image</span><span class="o">=</span><span class="n">image</span><span class="p">,</span> |
| <span class="n">start</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> |
| <span class="n">ex_container_port</span><span class="o">=</span><span class="mi">8080</span><span class="p">,</span> <span class="n">ex_host_port</span><span class="o">=</span><span class="mi">8080</span><span class="p">)</span> |
| </code></pre> |
| </div> |
| </div> |
| |
| <p>This will have deployed a task definition in Amazon ECS with a single container inside, with a cluster called ‘main’ and deployed the tomcat:8.0 image from the Docker hub to that region.</p> |
| |
| <p>Check out the <a href="http://libcloud.readthedocs.org/en/latest/container/drivers/ecs.html">ECS Documentation</a> for more details.</p> |
| |
| <h3 id="kubernetes">Kubernetes</h3> |
| |
| <p>Kubernetes authentication is currently only implemented for None (off) and Basic HTTP authentication. Let’s use the <a href="http://kubernetes.io/v1.1/docs/admin/authentication.html">basic HTTP authentication method</a> to connect.</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code> |
| <span class="n">k_cls</span> <span class="o">=</span> <span class="n">get_driver</span><span class="p">(</span><span class="n">Provider</span><span class="p">.</span><span class="n">KUBERNETES</span><span class="p">)</span> |
| |
| <span class="n">kubernetes</span> <span class="o">=</span> <span class="n">k_cls</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="s">'my_username'</span><span class="p">,</span> |
| <span class="n">secret</span><span class="o">=</span><span class="s">'THIS_IS)+_MY_SECRET_KEY+I6TVkv68o4H'</span><span class="p">,</span> |
| <span class="n">host</span><span class="o">=</span><span class="s">'126.32.21.4'</span><span class="p">)</span> |
| <span class="n">cluster2</span> <span class="o">=</span> <span class="n">kubernetes</span><span class="p">.</span><span class="n">create_cluster</span><span class="p">(</span><span class="s">'default'</span><span class="p">)</span> |
| <span class="n">container2</span> <span class="o">=</span> <span class="n">kubernetes</span><span class="p">.</span><span class="n">deploy_container</span><span class="p">(</span> |
| <span class="n">cluster</span><span class="o">=</span><span class="n">cluster</span><span class="p">,</span> |
| <span class="n">name</span><span class="o">=</span><span class="s">'hello-world'</span><span class="p">,</span> |
| <span class="n">image</span><span class="o">=</span><span class="n">image</span><span class="p">,</span> |
| <span class="n">start</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span> |
| </code></pre> |
| </div> |
| </div> |
| |
| <h2 id="wrapping-it-up">Wrapping it up</h2> |
| |
| <p>Now, let’s wrap that all up by doing a list comprehension across the 3 drivers to get a list of all containers and print their ID’s and Names. Then delete them.</p> |
| |
| <div class="language-python highlighter-rouge"> |
| <div class="highlight"> |
| <pre class="highlight"><code><span class="n">containers</span> <span class="o">=</span> <span class="p">[</span><span class="n">conn</span><span class="p">.</span><span class="n">list_containers</span><span class="p">()</span> <span class="k">for</span> <span class="n">conn</span> <span class="ow">in</span> <span class="p">[</span><span class="n">docker</span><span class="p">,</span> <span class="n">ecs</span><span class="p">,</span> <span class="n">kubernetes</span><span class="p">]]</span> |
| <span class="k">for</span> <span class="n">container</span> <span class="ow">in</span> <span class="n">containers</span><span class="p">:</span> |
| <span class="k">print</span><span class="p">(</span><span class="s">"%s : %s"</span> <span class="o">%</span> <span class="p">(</span><span class="n">container</span><span class="p">.</span><span class="nb">id</span><span class="p">,</span> <span class="n">container</span><span class="p">.</span><span class="n">name</span><span class="p">))</span> |
| <span class="n">container</span><span class="p">.</span><span class="n">destroy</span><span class="p">()</span> |
| |
| </code></pre> |
| </div> |
| </div> |
| |
| <h3 id="about-the-author">About the Author</h3> |
| |
| <p>Anthony Shaw is on the PMC for Apache Libcloud, you can follow Anthony on Twitter at <a href="https://twitter.com/anthonypjshaw">@anthonypjshaw</a>.</p> |
| |
| |
| </div> |
| |
| <div class="row section post-meta"> |
| <div class="col-md-12 post-tags"> |
| <p>Tags: <a href="/blog/tags/news.html" rel="tag">news</a>, <a href="/blog/tags/containers.html" rel="tag">containers</a>, <a href="/blog/tags/tutorial.html" rel="tag">tutorial</a></p> |
| </div> |
| </div> |
| </div> |
| |
| |
| |
| |
| </div> |
| </div> |
|
|
|
|
| <hr />
|
|
|
| <footer>
|
| <div class="row">
|
| <div class="col-lg-12 text-center">
|
| <div class="footer-links"> |
| <p><a href="http://www.apache.org/licenses/">License</a> | <a |
| href="/security.html">Security</a> | <a |
| href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a> | |
| <a href="http://www.apache.org/foundation/thanks.html">Thanks</a> | |
| <a href="/credits.html">Credits</a> | <a href="/media.html">Media</a> |
| </div> |
| |
| <div class="footer-text"> |
| <p class="">Copyright © 2009-2022 <a href="https://www.apache.org/" target="_blank">The Apache Software Foundation</a></p> |
| <p class="">Apache Libcloud, Libcloud, Apache, the Apache feather, and the Apache Libcloud project logo are trademarks of the Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> |
| <p class="">Site last updated on 2022-03-29 22:17:41 +0000</p> |
| </div> |
|
|
| </div>
|
| </div>
|
| </footer>
|
|
|
| </div><!-- /.container -->
|
|
|
| <!-- JavaScript -->
|
| <script src='/assets/global-23549af89e29254587a07d91b7eb9749.js' type='text/javascript'></script>
|
|
|
|
|
|
|
| <script type="text/javascript"> |
| var _gaq = _gaq || []; |
| _gaq.push(['_setAccount', 'UA-23580482-1']); |
| _gaq.push(['_trackPageview']); |
| |
| (function() { |
| var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; |
| ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; |
| var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); |
| })(); |
| </script> |
|
|
| </body>
|
| </html>
|
