This guide describes common problems encountered when deploying applications.
The error Invalid YAML: Plan not in acceptable format: Cannot convert ...
means that the text is not valid YAML. Common reasons include that the indentation is incorrect, or that there are non-matching brackets.
The error Unrecognized application blueprint format: no services defined
means that the services:
section is missing.
An error like the one shown below means that the given entity type (in this case com.acme.Foo) is not in the catalog or on the classpath:
{% highlight bash %} Deployment plan item Service[name=,description=,serviceType=com.acme.Foo,characteristics=[],customAttributes={}] cannot be matched {% endhighlight %}
An error like the one shown below means that the given location (in this case aws-ec3) was unknown:
{% highlight bash %} Illegal parameter for ‘location’ (aws-ec3); not resolvable: java.util.NoSuchElementException: Unknown location ‘aws-ec3’: either this location is not recognised or there is a problem with location resolver configuration {% endhighlight %}
This means it does not match any of the named locations in brooklyn.properties, nor any of the clouds enabled in the jclouds support, nor any of the locations added dynamically through the catalog API.
There are many stages at which VM provisioning can fail! An error Failure running task provisioning
means there was some problem obtaining or connecting to the machine.
An error like ... Not authorized to access cloud ...
usually means the wrong identity/credential was used.
AWS requires a X-Amz-Date header which contains the date of the Apache Brooklyn AWS client. If the date on the server is wrong, for example several minutes behind you will get an Authorization Exception. This is to prevent replay attacks. Please be sure to set the clock correctly on the machine running Apache Brooklyn. To set the time on Linux you can use the ntp client (e.g. sudo ntpdate pool.ntp.org
). We advise running the ntp daemon so that the clock is kept continually in sync.
An error like Unable to match required VM template constraints
means that a matching image (e.g. AMI in AWS terminology) could not be found. This could be because an incorrect explicit image id was supplied, or because the match-criteria could not be satisfied using the given images available in the given cloud. The first time this error is encountered, a listing of all images in that cloud/region will be written to the debug log.
Failure to form an ssh connection to the newly provisioned VM can be reported in several different ways, depending on the nature of the error. This breaks down into failures at different points:
... could not connect to any ip address port 22 on node ...
).... Exhausted available authentication methods ...
).There are many possible reasons for this ssh failure, which include:
machineCreateAttempts
configuration option, to automatically retry with a new VM.loginUser
configuration option. An example of this is with some Ubuntu VMs, where the “ubuntu” user should be used. However, on some clouds it defaults to trying to ssh as “root”.A very useful debug configuration is to set destroyOnFailure
to false. This will allow ssh failures to be more easily investigated.
The exception java.security.KeyException
can be thrown when jclouds is attempting the SSL handshake, to make cloud API calls. This can happen if the version of nss is older than 3.16 - the nss package includes the SSL library.
To fix this on CentOS, run:
{% highlight bash %} sudo yum upgrade nss {% endhighlight %}
For a discussion of investigating this kind of issue, see this Backslasher blog.
The full stacktrace is shown below:
{% highlight java %} Caused by: javax.net.ssl.SSLException: java.security.ProviderException: java.security.KeyException at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1906) at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1889) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1410) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250) at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:294) at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170) at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64) at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:95) ... 64 more Caused by: java.security.ProviderException: java.security.KeyException at sun.security.ec.ECKeyPairGenerator.generateKeyPair(ECKeyPairGenerator.java:147) at java.security.KeyPairGenerator$Delegate.generateKeyPair(KeyPairGenerator.java:703) at sun.security.ssl.ECDHCrypt.(ECDHCrypt.java:77) at sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:721) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:281) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) ... 74 more Caused by: java.security.KeyException at sun.security.ec.ECKeyPairGenerator.generateECKeyPair(Native Method) at sun.security.ec.ECKeyPairGenerator.generateKeyPair(ECKeyPairGenerator.java:128) ... 83 more {% endhighlight %}
A common generic error message is that there was a timeout waiting for service-up.
This just means that the entity did not get to service-up in the pre-defined time period (the default is two minutes, and can be configured using the start.timeout
config key; the timer begins after the start tasks are completed).
See the overview for where to find additional information, especially the section on “Entity's Error Status”.
If you receive an error message similar to the one below when provisioning a VM, it means that the wrong username is being used for ssh'ing to the machine. The “invalid packet” is because a response such as “Please login as the ubuntu user rather than root user.” is being sent back.
You can workaround the issue by explicitly setting the user that Brooklyn should use to login to the VM (typically the OS default user).
{% highlight bash %} error acquiring SFTPClient() (out of retries - max 50) Invalid packet: indicated length too large java.lang.IllegalStateException Invalid packet: indicated length too large {% endhighlight %}
An example of how to explicitly set the user is shown below (when defining a Location) by using ‘loginUser’:
{% highlight yaml %} brooklyn.locations:
The following error, when deploying a blueprint, has been shown to be caused by issues with DNS provided by your ISP or traffic filtering such as child-safe type filtering:
Caused by: javax.net.ssl.SSLException: Received fatal alert: close_notify
To resolve this try disabling traffic filtering and setting your DNS to a public server such as 8.8.8.8 to use google DNS. See here for details on how to configure this.
When downloading an install artifact with Curl, using CentOS 7.0, one can get the failure shown below:
curl: (35) Peer reports incompatible or unsupported protocol version.
This can be caused by incompatible TLS negotiation with the web server (e.g. with github). For more details, see Red Hat bug 1170339, “use the default min/max TLS version provided by NSS [RHEL-7]”.
To confirm this is the issue, try running the failing curl command on the same machine with curl -v
for verbose output. You should see a more detailed error such as:
NSS error -12286 (SSL_ERROR_NO_CYPHER_OVERLAP) Cannot communicate securely with peer: no common encryption algorithm(s). Closing connection 1
Possible workarounds include:
Use a more recent version of CentOS. On AWS, a good choice is the most recent centos.org image from the AWS marketplace. However, this involves first subscribing to it in the marketplace. The Amazon Linux AMI is another good choice, but this is not a normal CentOS image so it depends what distro(s) the entity was developed/tested against.
Change your blueprint to first do sudo yum update -y curl nss
, before the curl command is executed.