| <!DOCTYPE html> |
| <!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| |
| |
| <title>Packages - Apache Apex Documentation</title> |
| |
| |
| <link rel="shortcut icon" href="../favicon.ico"> |
| |
| |
| |
| <link href='https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700' rel='stylesheet' type='text/css'> |
| |
| <link rel="stylesheet" href="../css/theme.css" type="text/css" /> |
| <link rel="stylesheet" href="../css/theme_extra.css" type="text/css" /> |
| <link rel="stylesheet" href="../css/highlight.css"> |
| |
| |
| <script> |
| // Current page data |
| var mkdocs_page_name = "Packages"; |
| var mkdocs_page_input_path = "application_packages.md"; |
| var mkdocs_page_url = "/application_packages/"; |
| </script> |
| |
| <script src="../js/jquery-2.1.1.min.js"></script> |
| <script src="../js/modernizr-2.8.3.min.js"></script> |
| <script type="text/javascript" src="../js/highlight.pack.js"></script> |
| <script src="../js/theme.js"></script> |
| |
| |
| </head> |
| |
| <body class="wy-body-for-nav" role="document"> |
| |
| <div class="wy-grid-for-nav"> |
| |
| |
| <nav data-toggle="wy-nav-shift" class="wy-nav-side stickynav"> |
| <div class="wy-side-nav-search"> |
| <a href=".." class="icon icon-home"> Apache Apex Documentation</a> |
| <div role="search"> |
| <form id ="rtd-search-form" class="wy-form" action="../search.html" method="get"> |
| <input type="text" name="q" placeholder="Search docs" /> |
| </form> |
| </div> |
| </div> |
| |
| <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> |
| <ul class="current"> |
| |
| <li> |
| <li class="toctree-l1 "> |
| <a class="" href="..">Apache Apex</a> |
| |
| </li> |
| <li> |
| |
| <li> |
| <ul class="subnav"> |
| <li><span>Development</span></li> |
| |
| |
| |
| <li class="toctree-l1 "> |
| <a class="" href="../apex_development_setup/">Development Setup</a> |
| |
| </li> |
| |
| |
| |
| <li class="toctree-l1 "> |
| <a class="" href="../application_development/">Applications</a> |
| |
| </li> |
| |
| |
| |
| <li class="toctree-l1 current"> |
| <a class="current" href="./">Packages</a> |
| |
| <ul> |
| |
| <li class="toctree-l3"><a href="#apache-apex-packages">Apache Apex Packages</a></li> |
| |
| |
| <li class="toctree-l3"><a href="#application-packages">Application Packages</a></li> |
| |
| <li><a class="toctree-l4" href="#requirements">Requirements</a></li> |
| |
| <li><a class="toctree-l4" href="#creating-your-first-apex-app-package">Creating Your First Apex App Package</a></li> |
| |
| <li><a class="toctree-l4" href="#writing-your-own-app-package">Writing Your Own App Package</a></li> |
| |
| <li><a class="toctree-l4" href="#zip-structure-of-application-package">Zip Structure of Application Package</a></li> |
| |
| <li><a class="toctree-l4" href="#examining-and-launching-application-packages-through-cli">Examining and Launching Application Packages Through CLI</a></li> |
| |
| |
| <li class="toctree-l3"><a href="#configuration-packages">Configuration Packages</a></li> |
| |
| <li><a class="toctree-l4" href="#creating-configuration-packages">Creating Configuration Packages</a></li> |
| |
| <li><a class="toctree-l4" href="#assembling-your-own-configuration-package">Assembling your own configuration package</a></li> |
| |
| |
| </ul> |
| |
| </li> |
| |
| |
| |
| <li class="toctree-l1 "> |
| <a class="" href="../operator_development/">Operators</a> |
| |
| </li> |
| |
| |
| |
| <li class="toctree-l1 "> |
| <a class="" href="../autometrics/">AutoMetric API</a> |
| |
| </li> |
| |
| |
| |
| <li class="toctree-l1 "> |
| <a class="" href="../development_best_practices/">Best Practices</a> |
| |
| </li> |
| |
| |
| </ul> |
| <li> |
| |
| <li> |
| <ul class="subnav"> |
| <li><span>Operations</span></li> |
| |
| |
| |
| <li class="toctree-l1 "> |
| <a class="" href="../apex_cli/">Apex CLI</a> |
| |
| </li> |
| |
| |
| |
| <li class="toctree-l1 "> |
| <a class="" href="../security/">Security</a> |
| |
| </li> |
| |
| |
| </ul> |
| <li> |
| |
| <li> |
| <li class="toctree-l1 "> |
| <a class="" href="../compatibility/">Compatibility</a> |
| |
| </li> |
| <li> |
| |
| </ul> |
| </div> |
| |
| </nav> |
| |
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> |
| |
| |
| <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> |
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> |
| <a href="..">Apache Apex Documentation</a> |
| </nav> |
| |
| |
| <div class="wy-nav-content"> |
| <div class="rst-content"> |
| <div role="navigation" aria-label="breadcrumbs navigation"> |
| <ul class="wy-breadcrumbs"> |
| <li><a href="..">Docs</a> »</li> |
| |
| |
| |
| <li>Development »</li> |
| |
| |
| |
| <li>Packages</li> |
| <li class="wy-breadcrumbs-aside"> |
| |
| </li> |
| </ul> |
| <hr/> |
| </div> |
| <div role="main"> |
| <div class="section"> |
| |
| <h1 id="apache-apex-packages">Apache Apex Packages</h1> |
| <h1 id="application-packages">Application Packages</h1> |
| <p>An Apache Apex Application Package is a zip file that contains all the |
| necessary files to launch an application in Apache Apex. It is the |
| standard way for assembling and sharing an Apache Apex application.</p> |
| <h2 id="requirements">Requirements</h2> |
| <p>You will need have the following installed:</p> |
| <ol> |
| <li>Apache Maven 3.0 or later (for assembling the App Package)</li> |
| <li>Apache Apex 3.2.0 or later (for launching the App Package in your cluster)</li> |
| </ol> |
| <h2 id="creating-your-first-apex-app-package">Creating Your First Apex App Package</h2> |
| <p>You can create an Apex Application Package using your Linux command |
| line, or using your favorite IDE.</p> |
| <h3 id="using-command-line">Using Command Line</h3> |
| <p>First, change to the directory where you put your projects, and create |
| an Apex application project using Maven by running the following |
| command. Replace "com.example", "mydtapp" and "1.0-SNAPSHOT" with the |
| appropriate values (make sure this is all on one line):</p> |
| <pre><code>$ mvn archetype:generate \ |
| -DarchetypeGroupId=org.apache.apex \ |
| -DarchetypeArtifactId=apex-app-archetype -DarchetypeVersion=3.4.0 \ |
| -DgroupId=com.example -Dpackage=com.example.mydtapp -DartifactId=mydtapp \ |
| -Dversion=1.0-SNAPSHOT |
| </code></pre> |
| <p>This creates a Maven project named "mydtapp". Open it with your favorite |
| IDE (e.g. NetBeans, Eclipse, IntelliJ IDEA). In the project, there is a |
| sample DAG that generates a number of tuples with a random number and |
| prints out "hello world" and the random number in the tuples. The code |
| that builds the DAG is in |
| src/main/java/com/example/mydtapp/Application.java, and the code that |
| runs the unit test for the DAG is in |
| src/test/java/com/example/mydtapp/ApplicationTest.java. Try it out by |
| running the following command:</p> |
| <pre><code>$cd mydtapp; mvn package |
| </code></pre> |
| <p>This builds the App Package runs the unit test of the DAG. You should |
| be getting test output similar to this:</p> |
| <pre><code> ------------------------------------------------------- |
| TESTS |
| ------------------------------------------------------- |
| |
| Running com.example.mydtapp.ApplicationTest |
| hello world: 0.8015370953286478 |
| hello world: 0.9785359225545481 |
| hello world: 0.6322611586644047 |
| hello world: 0.8460953663451775 |
| hello world: 0.5719372906929072 |
| hello world: 0.6361174312337172 |
| hello world: 0.14873007534816318 |
| hello world: 0.8866986277418261 |
| hello world: 0.6346526809866057 |
| hello world: 0.48587295703904465 |
| hello world: 0.6436832429676687 |
| |
| ... |
| |
| Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.863 |
| sec |
| |
| Results : |
| |
| Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 |
| </code></pre> |
| |
| <p>The "mvn package" command creates the App Package file in target |
| directory as target/mydtapp-1.0-SNAPSHOT.apa. You will be able to use |
| that App Package file to launch this sample application in your actual |
| Apex installation.</p> |
| <p>Alternatively you can perform the same steps within your IDE (IDEA IntelliJ, Eclipse, NetBeans all support it). Please check the IDE documentation for details.</p> |
| <p>Group ID: org.apache.apex |
| Artifact ID: apex-app-archetype |
| Version: 3.4.0 (or any later version)</p> |
| <h2 id="writing-your-own-app-package">Writing Your Own App Package</h2> |
| <p>Please refer to the [Application Developer Guide][application_development.md] on the basics on how to write an Apache Apex application. In your AppPackage project, you can add custom operators (refer to <a href="../operator_development/">Operator Development Guide</a>, project dependencies, default and required configuration properties, pre-set configurations and other metadata. Note that you can also specify the DAG using Java, JSON or properties files. </p> |
| <h3 id="adding-and-removing-project-dependencies">Adding (and removing) project dependencies</h3> |
| <p>Under the project, you can add project dependencies in pom.xml, or do it |
| through your IDE. Here’s the section that describes the dependencies in |
| the default pom.xml:</p> |
| <pre><code> <dependencies> |
| <!-- add your dependencies here --> |
| <dependency> |
| <groupId>org.apache.apex</groupId> |
| <artifactId>malhar-library</artifactId> |
| <version>${apex.version}</version> |
| <!-- |
| If you know your application do not need the transitive dependencies that are pulled in by malhar-library, |
| Uncomment the following to reduce the size of your app package. |
| --> |
| <!-- |
| <exclusions> |
| <exclusion> |
| <groupId>*</groupId> |
| <artifactId>*</artifactId> |
| </exclusion> |
| </exclusions> |
| --> |
| </dependency> |
| <dependency> |
| <groupId>org.apache.apex</groupId> |
| <artifactId>apex-engine</artifactId> |
| <version>${apex.version}</version> |
| <scope>provided</scope> |
| </dependency> |
| <dependency> |
| <groupId>junit</groupId> |
| <artifactId>junit</artifactId> |
| <version>4.10</version> |
| <scope>test</scope> |
| </dependency> |
| </dependencies> |
| </code></pre> |
| |
| <p>By default, as shown above, the default dependencies include |
| malhar-library in compile scope, dt-engine in provided scope, and junit |
| in test scope. Do not remove these three dependencies since they are |
| necessary for any Apex application. You can, however, exclude |
| transitive dependencies from malhar-library to reduce the size of your |
| App Package, provided that none of the operators in malhar-library that |
| need the transitive dependencies will be used in your application.</p> |
| <p>In the sample application, it is safe to remove the transitive |
| dependencies from malhar-library, by uncommenting the "exclusions" |
| section. It will reduce the size of the sample App Package from 8MB to |
| 700KB.</p> |
| <p>Note that if we exclude *, in some versions of Maven, you may get |
| warnings similar to the following:</p> |
| <pre><code> |
| [WARNING] 'dependencies.dependency.exclusions.exclusion.groupId' for |
| org.apache.apex:malhar-library:jar with value '*' does not match a |
| valid id pattern. |
| |
| [WARNING] |
| [WARNING] It is highly recommended to fix these problems because they |
| threaten the stability of your build. |
| [WARNING] |
| [WARNING] For this reason, future Maven versions might no longer support |
| building such malformed projects. |
| [WARNING] |
| |
| </code></pre> |
| |
| <p>This is a bug in early versions of Maven 3. The dependency exclusion is |
| still valid and it is safe to ignore these warnings.</p> |
| <h3 id="application-configuration">Application Configuration</h3> |
| <p>A configuration file can be used to configure an application. Different |
| kinds of configuration parameters can be specified. They are application |
| attributes, operator attributes and properties, port attributes, stream |
| properties and application specific properties. They are all specified |
| as name value pairs, in XML format, like the following.</p> |
| <pre><code><?xml version="1.0"?> |
| <configuration> |
| <property> |
| <name>some_name_1</name> |
| <value>some_default_value</value> |
| </property> |
| <property> |
| <name>some_name_2</name> |
| <value>some_default_value</value> |
| </property> |
| </configuration> |
| </code></pre> |
| |
| <h3 id="application-attributes">Application attributes</h3> |
| <p>Application attributes are used to specify the platform behavior for the |
| application. They can be specified using the parameter |
| <code>dt.attr.<attribute></code>. The prefix “dt” is a constant, “attr” is a |
| constant denoting an attribute is being specified and <code><attribute></code> |
| specifies the name of the attribute. Below is an example snippet setting |
| the streaming windows size of the application to be 1000 milliseconds.</p> |
| <pre><code> <property> |
| <name>dt.attr.STREAMING_WINDOW_SIZE_MILLIS</name> |
| <value>1000</value> |
| </property> |
| </code></pre> |
| |
| <p>The name tag specifies the attribute and value tag specifies the |
| attribute value. The name of the attribute is a JAVA constant name |
| identifying the attribute. The constants are defined in |
| com.datatorrent.api.Context.DAGContext and the different attributes can |
| be specified in the format described above.</p> |
| <h3 id="operator-attributes">Operator attributes</h3> |
| <p>Operator attributes are used to specify the platform behavior for the |
| operator. They can be specified using the parameter |
| <code>dt.operator.<operator-name>.attr.<attribute></code>. The prefix “dt” is a |
| constant, “operator” is a constant denoting that an operator is being |
| specified, <code><operator-name></code> denotes the name of the operator, “attr” is |
| the constant denoting that an attribute is being specified and |
| <code><attribute></code> is the name of the attribute. The operator name is the |
| same name that is specified when the operator is added to the DAG using |
| the addOperator method. An example illustrating the specification is |
| shown below. It specifies the number of streaming windows for one |
| application window of an operator named “input” to be 10</p> |
| <pre><code><property> |
| <name>dt.operator.input.attr.APPLICATION_WINDOW_COUNT</name> |
| <value>10</value> |
| </property> |
| </code></pre> |
| |
| <p>The name tag specifies the attribute and value tag specifies the |
| attribute value. The name of the attribute is a JAVA constant name |
| identifying the attribute. The constants are defined in |
| com.datatorrent.api.Context.OperatorContext and the different attributes |
| can be specified in the format described above.</p> |
| <h3 id="operator-properties">Operator properties</h3> |
| <p>Operators can be configured using operator specific properties. The |
| properties can be specified using the parameter |
| <code>dt.operator.<operator-name>.prop.<property-name></code>. The difference |
| between this and the operator attribute specification described above is |
| that the keyword “prop” is used to denote that it is a property and |
| <code><property-name></code> specifies the property name. An example illustrating |
| this is specified below. It specifies the property “hostname” of the |
| redis server for a “redis” output operator.</p> |
| <pre><code> <property> |
| <name>dt.operator.redis.prop.host</name> |
| <value>127.0.0.1</value> |
| </property> |
| </code></pre> |
| |
| <p>The name tag specifies the property and the value specifies the property |
| value. The property name is converted to a setter method which is called |
| on the actual operator. The method name is composed by appending the |
| word “set” and the property name with the first character of the name |
| capitalized. In the above example the setter method would become |
| setHost. The method is called using JAVA reflection and the property |
| value is passed as an argument. In the above example the method setHost |
| will be called on the “redis” operator with “127.0.0.1” as the argument.</p> |
| <h3 id="port-attributes">Port attributes</h3> |
| <p>Port attributes are used to specify the platform behavior for input and |
| output ports. They can be specified using the parameter <code>dt.operator.<operator-name>.inputport.<port-name>.attr.<attribute></code> |
| for input port and <code>dt.operator.<operator-name>.outputport.<port-name>.attr.<attribute></code> |
| for output port. The keyword “inputport” is used to denote an input port |
| and “outputport” to denote an output port. The rest of the specification |
| follows the conventions described in other specifications above. An |
| example illustrating this is specified below. It specifies the queue |
| capacity for an input port named “input” of an operator named “range” to |
| be 4k.</p> |
| <pre><code><property> |
| <name>dt.operator.range.inputport.input.attr.QUEUE_CAPACITY</name> |
| <value>4000</value> |
| </property> |
| </code></pre> |
| |
| <p>The name tag specifies the attribute and value tag specifies the |
| attribute value. The name of the attribute is a JAVA constant name |
| identifying the attribute. The constants are defined in |
| com.datatorrent.api.Context.PortContext and the different attributes can |
| be specified in the format described above.</p> |
| <p>The attributes for an output port can also be specified in a similar way |
| as described above with a change that keyword “outputport” is used |
| instead of “intputport”. A generic keyword “port” can be used to specify |
| either an input or an output port. It is useful in the wildcard |
| specification described below.</p> |
| <h3 id="stream-properties">Stream properties</h3> |
| <p>Streams can be configured using stream properties. The properties can be |
| specified using the parameter |
| <code>dt.stream.<stream-name>.prop.<property-name></code> The constant “stream” |
| specifies that it is a stream, <code><stream-name></code> specifies the name of the |
| stream and <code><property-name></code> the name of the property. The name of the |
| stream is the same name that is passed when the stream is added to the |
| DAG using the addStream method. An example illustrating the |
| specification is shown below. It sets the locality of the stream named |
| “stream1” to container local indicating that the operators the stream is |
| connecting be run in the same container.</p> |
| <pre><code> <property> |
| <name>dt.stream.stream1.prop.locality</name> |
| <value>CONTAINER_LOCAL</value> |
| </property> |
| </code></pre> |
| |
| <p>The property name is converted into a set method on the stream in the |
| same way as described in operator properties section above. In this case |
| the method would be setLocality and it will be called in the stream |
| “stream1” with the value as the argument.</p> |
| <p>Along with the above system defined parameters, the applications can |
| define their own specific parameters they can be specified in the |
| configuration file. The only condition is that the names of these |
| parameters don’t conflict with the system defined parameters or similar |
| application parameters defined by other applications. To this end, it is |
| recommended that the application parameters have the format |
| <code><full-application-class-name>.<param-name>.</code> The |
| full-application-class-name is the full JAVA class name of the |
| application including the package path and param-name is the name of the |
| parameter within the application. The application will still have to |
| still read the parameter in using the configuration API of the |
| configuration object that is passed in populateDAG.</p> |
| <h3 id="wildcards">Wildcards</h3> |
| <p>Wildcards and regular expressions can be used in place of names to |
| specify a group for applications, operators, ports or streams. For |
| example, to specify an attribute for all ports of an operator it can be |
| done as follows</p> |
| <pre><code><property> |
| <name>dt.operator.range.port.*.attr.QUEUE_CAPACITY</name> |
| <value>4000</value> |
| </property> |
| </code></pre> |
| |
| <p>The wildcard “*” was used instead of the name of the port. Wildcard can |
| also be used for operator name, stream name or application name. Regular |
| expressions can also be used for names to specify attributes or |
| properties for a specific set.</p> |
| <h3 id="adding-configuration-properties">Adding configuration properties</h3> |
| <p>It is common for applications to require configuration parameters to |
| run. For example, the address and port of the database, the location of |
| a file for ingestion, etc. You can specify them in |
| src/main/resources/META-INF/properties.xml under the App Package |
| project. The properties.xml may look like:</p> |
| <pre><code><?xml version="1.0"?> |
| <configuration> |
| <property> |
| <name>some_name_1</name> |
| </property> |
| <property> |
| <name>some_name_2</name> |
| <value>some_default_value</value> |
| </property> |
| </configuration> |
| </code></pre> |
| |
| <p>The name of an application-specific property takes the form of:</p> |
| <p><code>dt.operator.{opName}.prop.{propName}</code></p> |
| <p>The first represents the property with name propName of operator opName. |
| Or you can set the application name at run time by setting this |
| property:</p> |
| <pre><code> dt.attr.APPLICATION_NAME |
| </code></pre> |
| <p>In this example, property some_name_1 is a required property which |
| must be set at launch time, or it must be set by a pre-set configuration |
| (see next section). Property some_name_2 is a property that is |
| assigned with value some_default_value unless it is overridden at |
| launch time.</p> |
| <h3 id="adding-pre-set-configurations">Adding pre-set configurations</h3> |
| <p>At build time, you can add pre-set configurations to the App Package by |
| adding configuration XML files under <code>src/site/conf/<conf>.xml</code>in your |
| project. You can then specify which configuration to use at launch |
| time. The configuration XML is of the same format of the properties.xml |
| file.</p> |
| <h3 id="application-specific-properties-file">Application-specific properties file</h3> |
| <p>You can also specify properties.xml per application in the application |
| package. Just create a file with the name properties-{appName}.xml and |
| it will be picked up when you launch the application with the specified |
| name within the application package. In short:</p> |
| <p>properties.xml: Properties that are global to the Configuration |
| Package</p> |
| <p>properties-{appName}.xml: Properties that are specific when launching |
| an application with the specified appName.</p> |
| <h3 id="properties-source-precedence">Properties source precedence</h3> |
| <p>If properties with the same key appear in multiple sources (e.g. from |
| app package default configuration as META-INF/properties.xml, from app |
| package configuration in the conf directory, from launch time defines, |
| etc), the precedence of sources, from highest to lowest, is as follows:</p> |
| <ol> |
| <li>Launch time defines (using -D option in CLI)</li> |
| <li>Launch time specified configuration file in file system (using -conf |
| option in CLI)</li> |
| <li>Launch time specified package configuration (using -apconf option in |
| CLI)</li> |
| <li>Configuration from \$HOME/.dt/dt-site.xml</li> |
| <li>Application defaults within the package as |
| META-INF/properties-{appname}.xml</li> |
| <li>Package defaults as META-INF/properties.xml</li> |
| <li>dt-site.xml in local DT installation</li> |
| <li>dt-site.xml stored in HDFS</li> |
| </ol> |
| <h3 id="other-meta-data">Other meta-data</h3> |
| <p>In a Apex App Package project, the pom.xml file contains a |
| section that looks like:</p> |
| <pre><code><properties> |
| <apex.version>3.4.0</apex.version> |
| <apex.apppackage.classpath\>lib*.jar</apex.apppackage.classpath> |
| </properties> |
| </code></pre> |
| |
| <p>apex.version is the Apache Apex version that are to be used |
| with this Application Package.</p> |
| <p>apex.apppackage.classpath is the classpath that is used when |
| launching the application in the Application Package. The default is |
| lib/*.jar, where lib is where all the dependency jars are kept within |
| the Application Package. One reason to change this field is when your |
| Application Package needs the classpath in a specific order.</p> |
| <h3 id="logging-configuration">Logging configuration</h3> |
| <p>Just like other Java projects, you can change the logging configuration |
| by having your log4j.properties under src/main/resources. For example, |
| if you have the following in src/main/resources/log4j.properties:</p> |
| <pre><code> log4j.rootLogger=WARN,CONSOLE |
| log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender |
| log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout |
| log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} [%t] %-5p |
| %c{2} %M - %m%n |
| </code></pre> |
| |
| <p>The root logger’s level is set to WARN and the output is set to the console (stdout).</p> |
| <p>Note that by default from project created from the maven archetype, |
| there is already a log4j.properties file under src/test/resources and |
| that file is only used for the unit test.</p> |
| <h2 id="zip-structure-of-application-package">Zip Structure of Application Package</h2> |
| <p>Apache Apex Application Package files are zip files. You can examine the content of any Application Package by using unzip -t on your Linux command line.</p> |
| <p>There are four top level directories in an Application Package:</p> |
| <ol> |
| <li>"app" contains the jar files of the DAG code and any custom operators, and any JSON or properties files that specify a DAG.</li> |
| <li>"lib" contains all dependency jars</li> |
| <li>"conf" contains all the pre-set configuration XML files.</li> |
| <li>"META-INF" contains the MANIFEST.MF file and the properties.xml file.</li> |
| <li>“resources” contains any other files</li> |
| </ol> |
| <h2 id="examining-and-launching-application-packages-through-cli">Examining and Launching Application Packages Through CLI</h2> |
| <p>If you are working with Application Packages in the local filesystem, you can use the Apex Command Line Interface (apex).</p> |
| <h3 id="getting-application-package-meta-information">Getting Application Package Meta Information</h3> |
| <p>You can get the meta information about the Application Package using |
| this Apex CLI command.</p> |
| <pre><code> apex> get-app-package-info <app-package-file> |
| </code></pre> |
| |
| <h3 id="getting-available-operators-in-application-package">Getting Available Operators In Application Package</h3> |
| <p>You can get the list of available operators in the Application Package |
| using this command.</p> |
| <pre><code> apex> get-app-package-operators <app-package-file> <package-prefix> |
| [parent-class] |
| </code></pre> |
| |
| <h3 id="getting-properties-of-operators-in-application-package">Getting Properties of Operators in Application Package</h3> |
| <p>You can get the list of properties of any operator in the Application |
| Package using this command.</p> |
| <p>apex> get-app-package-operator-properties <app-package-file> <operator-class></p> |
| <h3 id="launching-an-application-package">Launching an Application Package</h3> |
| <p>You can launch an application within an Application Package.</p> |
| <pre><code>apex> launch [-D property-name=property-value, ...] [-conf config-name] |
| [-apconf config-file-within-app-package] <app-package-file> |
| [matching-app-name] |
| </code></pre> |
| |
| <p>Note that -conf expects a configuration file in the file system, while -apconf expects a configuration file within the app package.</p> |
| <h1 id="configuration-packages">Configuration Packages</h1> |
| <p>Sometimes just a configuration file is not enough for launching an application package. If a configuration requires |
| additional files to be packaged, you can use an Apex Configuration Package.</p> |
| <h2 id="creating-configuration-packages">Creating Configuration Packages</h2> |
| <p>Creating Configuration Packages is similar to creating Application Packages. You can create a configuration |
| package project using Maven by running the following command. Replace "com.example", "mydtconfig" and "1.0-SNAPSHOT" with the appropriate values:</p> |
| <pre><code>$ mvn archetype:generate -DarchetypeGroupId=org.apache.apex \ |
| -DarchetypeArtifactId=apex-conf-archetype -DarchetypeVersion=3.4.0 \ |
| -DgroupId=com.example -Dpackage=com.example.mydtconfig -DartifactId=mydtconfig \ |
| -Dversion=1.0-SNAPSHOT |
| </code></pre> |
| |
| <p>And create the configuration package file by running:</p> |
| <pre><code>$ mvn package |
| </code></pre> |
| |
| <p>The "mvn package" command creates the Config Package file in target |
| directory as target/mydtconfig.apc. You will be able to use that |
| Configuration Package file to launch an Apache Apex application.</p> |
| <h2 id="assembling-your-own-configuration-package">Assembling your own configuration package</h2> |
| <p>Inside the project created by the archetype, these are the files that |
| you should know about when assembling your own configuration package:</p> |
| <pre><code>./pom.xml |
| ./src/main/resources/classpath |
| ./src/main/resources/files |
| ./src/main/resources/META-INF/properties.xml |
| ./src/main/resources/META-INF/properties-{appname}.xml |
| </code></pre> |
| <h3 id="pomxml">pom.xml</h3> |
| <p>Example:</p> |
| <pre><code class="xml"> <groupId>com.example</groupId> |
| <version>1.0.0</version> |
| <artifactId>mydtconf</artifactId> |
| <packaging>jar</packaging> |
| <!-- change these to the appropriate values --> |
| <name>My Apex Application Configuration</name> |
| <description>My Custom Application Configuration Description</description> |
| <properties> |
| <apex.apppackage.name>myapexapp</apex.apppackage.name> |
| <apex.apppackage.minversion>1.0.0</apex.apppackage.minversion> |
| <apex.apppackage.maxversion>1.9999.9999</apex.apppackage.maxversion> |
| <apex.appconf.classpath>classpath/*</apex.appconf.classpath> |
| <apex.appconf.files>files/*</apex.appconf.files> |
| </properties> |
| |
| </code></pre> |
| |
| <p>In pom.xml, you can change the following keys to your desired values</p> |
| <ul> |
| <li><code><groupId></code></li> |
| <li><code><version></code></li> |
| <li><code><artifactId></code></li> |
| <li><code><name></code></li> |
| <li><code><description></code></li> |
| </ul> |
| <p>You can also change the values of</p> |
| <ul> |
| <li><code><apex.apppackage.name></code></li> |
| <li><code><apex.apppackage.minversion></code></li> |
| <li><code><apex.apppackage.maxversion></code></li> |
| </ul> |
| <p>to reflect what Application Packages can be used with this configuration package. Apex will use this information to check whether a |
| configuration package is compatible with the Application Package when you issue a launch command.</p> |
| <h3 id="srcmainresourcesclasspath">./src/main/resources/classpath</h3> |
| <p>Place any file in this directory that you’d like to be copied to the |
| compute machines when launching an application and included in the |
| classpath of the application. Example of such files are Java properties |
| files and jar files.</p> |
| <h3 id="srcmainresourcesfiles">./src/main/resources/files</h3> |
| <p>Place any file in this directory that you’d like to be copied to the |
| compute machines when launching an application but not included in the |
| classpath of the application.</p> |
| <h3 id="properties-xml-file">Properties XML file</h3> |
| <p>A properties xml file consists of a set of key-value pairs. The set of |
| key-value pairs specifies the configuration options the application |
| should be launched with.</p> |
| <p>Example:</p> |
| <pre><code class="xml"><configuration> |
| <property> |
| <name>some-property-name</name> |
| <value>some-property-value</value> |
| </property> |
| ... |
| </configuration> |
| </code></pre> |
| |
| <p>Names of properties XML file:</p> |
| <ul> |
| <li><strong>properties.xml:</strong> Properties that are global to the Configuration |
| Package</li> |
| <li><strong>properties-{appName}.xml:</strong> Properties that are specific when launching |
| an application with the specified appName within the Application |
| Package.</li> |
| </ul> |
| <p>After you are done with the above, remember to do mvn package to |
| generate a new configuration package, which will be located in the |
| target directory in your project.</p> |
| <h3 id="zip-structure-of-configuration-package">Zip structure of configuration package</h3> |
| <p>Apex Application Configuration Package files are zip files. You |
| can examine the content of any Application Configuration Package by |
| using unzip -t on your Linux command line. The structure of the zip |
| file is as follow:</p> |
| <pre><code>META-INF |
| MANIFEST.MF |
| properties.xml |
| properties-{appname}.xml |
| classpath |
| {classpath files} |
| files |
| {files} |
| </code></pre> |
| |
| <h3 id="launching-with-cli">Launching with CLI</h3> |
| <p><code>-conf</code> option of the launch command in CLI supports specifying configuration package in the local filesystem. Example:</p> |
| <pre><code>dt\> launch mydtapp-1.0.0.apa -conf mydtconfig.apc |
| </code></pre> |
| <p>This command expects both the application package and the configuration package to be in the local file system.</p> |
| |
| </div> |
| </div> |
| <footer> |
| |
| <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> |
| |
| <a href="../operator_development/" class="btn btn-neutral float-right" title="Operators">Next <span class="icon icon-circle-arrow-right"></span></a> |
| |
| |
| <a href="../application_development/" class="btn btn-neutral" title="Applications"><span class="icon icon-circle-arrow-left"></span> Previous</a> |
| |
| </div> |
| |
| |
| <hr/> |
| |
| <div role="contentinfo"> |
| <!-- Copyright etc --> |
| |
| </div> |
| |
| Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. |
| </footer> |
| |
| </div> |
| </div> |
| |
| </section> |
| |
| </div> |
| |
| <div class="rst-versions" role="note" style="cursor: pointer"> |
| <span class="rst-current-version" data-toggle="rst-current-version"> |
| |
| |
| <span><a href="../application_development/" style="color: #fcfcfc;">« Previous</a></span> |
| |
| |
| <span style="margin-left: 15px"><a href="../operator_development/" style="color: #fcfcfc">Next »</a></span> |
| |
| </span> |
| </div> |
| |
| </body> |
| </html> |
