Google Git
Sign in
apache / forrest / copyless / . / src / documentation / content / xdocs / forrestbot-old.xml
blob: 9f119e456b6e507846e19573780b61fc76358c6e [file] [log] [blame]
<?xml version="1.0"?>
<!--
Copyright 2002-2004 The Apache Software Foundation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V1.2//EN" "http://apache.org/forrest/dtd/document-v12.dtd">
<document>
<header>
<title>The ForrestBot</title>
<abstract>This document explains how the Forrest project can be used to
centrally build the documentation for different projects. It also explains
which pieces in the Forrest distribution form this
functionality, and how they could be extended in the future.</abstract>
</header>
<body>
<warning>This documentation applies to the forrestbot included with Forrest 0.5.1 and previous releases.
The next version of forrest will include a new forrestbot, which has some documentation <link href="site:forrestbot">here</link>.
</warning>
<section>
<title>Foreword</title>
<p>This document explains how the Forrest project can be used to
centrally build the documentation for different projects.
It also explains which pieces in the Forrest distribution form this
functionality, and how they could be extended in the future.</p>
<p>See some related documents:</p>
<ul>
<li><link href="site:forrest-contract">Our Contract</link>
- explains the rules with which your project should comply
</li>
<li><link href="site:forrestbot-intro-old">Forrestbot introduction</link>
- concise overview of Forrestbot
</li>
</ul>
</section>
<section>
<title>Using the Forrestbot</title>
<section>
<title>Setting up the centralized service</title>
<p>The goal is to publish a static website that can be updated
in near-real-time to accomodate newly available content. Forrestbot
can run on one machine, get content from another machine, and publish
to yet another machine.</p>
<p>Running the bot is as easy as calling the following in your shell.
(executed in the root dir of the distribution.)</p>
<source>forrestbot -Dbot.config=myOwn.conf.xml -Dtemplate.echo=[true|false]</source>
<p>It is quite typical to perform this from a shell script that gets
scheduled for execution on a regular basis.</p>
</section>
<section>
<title>Describing your project to the bot</title>
<fixme author="mpo">There is not yet a DTD for this configuration
file.</fixme>
<p>The actions the bot is going to take are defined in the XML
file pointed at with the <code>bot.config</code> System
property. (if not specified with <code>-Dbot.config</code>
then the default config from the
distribution is taken: <code>./forrestbot.conf.xml</code>) </p>
<p>This configuration file has one (required, but optionally empty) child-element for
setting cross-project defaults, followed by a list of
project-elements that hold the configuration for the various projects
that the bot is going to work on.</p>
<source>
<![CDATA[<?xml version="1.0"?>
<forrest-config>
<defaults>
...<!-- different fallback values for various workstages -->
</defaults>
<project name="project-name-1">
...<!-- different subelements configure the various workstages -->
</project>
<project name="project-name-2">
...<!-- different subelements configure the various workstages -->
</project>
</forrest-config>
]]>
</source>
<p>During the bot-run, each project entry will start a parallel and
independent process consisting of a number of so called 'workstages'.</p>
<p>These workstages are listed in the
<code>./src/resources/forrestbot/stages.conf.xml</code> file and their actual
execution is handled by the various template tasks in the
<code>./resources/forrestbot/ant/templates.build.xml</code> file. </p>
<p>Currently the various workstages are:</p>
<ul>
<dl>
<dt>prepare</dt>
<dd> Make a clean Forrest-defined Cocoon context environment.</dd>
</dl>
<dl>
<dt>get-src</dt>
<dd>Slide in the project specific document sources in that Cocoon
context environment.</dd>
</dl>
<dl>
<dt>generate</dt>
<dd> Actually generate the site using Cocoon</dd>
</dl>
<dl>
<dt>deploy</dt>
<dd>Publish the newly created site</dd>
</dl>
<dl>
<dt>cleanup</dt>
<dd>&lt;No real actions yet&gt;</dd>
</dl>
</ul>
<p>Inside the project element the configuration allows to set some
parameters that get consumed in the template targets. This workstage dependant
configuration always takes the following layout: an element with the name of
one of the tasks, and an optional type-attribute with nested elements that hold
parameter names, with required name attribute holding the actual parameter
value. Example:</p>
<source>&lt;project name=&quot;xml-forrest&quot;&gt;
&lt;prepare&gt;
&lt;skin name=&quot;forrest-site&quot;/&gt;
&lt;/prepare&gt;
&lt;get-src type=&quot;cvs&quot;&gt;
&lt;host name=&quot;cvs.apache.org&quot;/&gt;
&lt;root name=&quot;/home/cvspublic&quot;/&gt;
&lt;user name=&quot;anoncvs&quot;/&gt;
&lt;passwd name=&quot;anoncvs&quot;/&gt;
&lt;module name=&quot;xml-forrest&quot;/&gt;
&lt;content-dir name=&quot;src/documentation&quot;/&gt;
&lt;project-dir name=&quot;&quot;/&gt;
&lt;/get-src&gt;
&lt;generate&gt;
&lt;debuglevel name=&quot;ERROR&quot; /&gt;
&lt;/generate&gt;
&lt;deploy type=&quot;scp&quot;&gt;
&lt;host name=&quot;krysalis.sourceforge.net&quot;/&gt;
&lt;root name=&quot;/home/groups/k/kr/krysalis/htdocs&quot;/&gt;
&lt;user name=&quot;forrestbot&quot;/&gt;
&lt;dir name=&quot;forrest&quot;/&gt;
&lt;/deploy&gt;
&lt;/project&gt;</source>
<note>Only the workstages that need configuration settings have an
entry here. At all times there will be a workstage template called for each
stage of the stages.conf.xml (even if it is not explicitly mentioned inside the
&lt;project&gt; element.) </note>
<p>Explained in detail, this will</p>
<ul>
<li>SET names properties (workstage(.@type)?.*) to appropriate values
(*/@name)</li>
<ul>
<dl>
<dt>prepare.skin</dt>
<dd>forrest-site</dd>
</dl>
<dl>
<dt>get-src.cvs.host</dt>
<dd>cvs.apache.org</dd>
</dl>
<dl>
<dt>get-src.cvs.root</dt>
<dd>/home/cvspublic</dd>
</dl>
<dl>
<dt>get-src.cvs.user</dt>
<dd>anoncvs</dd>
</dl>
<dl>
<dt>get-src.cvs.passwd</dt>
<dd>anoncvs</dd>
</dl>
<dl>
<dt>get-src.cvs.module</dt>
<dd>xml-forrest</dd>
</dl>
<dl>
<dt>get-src.cvs.content-dir</dt>
<dd>src/documentation</dd>
</dl><dl>
<dt>get-src.cvs.project-dir</dt>
<dd></dd>
</dl>
<dl>
<dt>generate.debuglevel</dt>
<dd>ERROR</dd>
</dl>
<dl>
<dt>deploy.scp.host</dt>
<dd>krysalis.sourceforge.net</dd>
</dl>
<dl>
<dt>deploy.scp.root</dt>
<dd>/home/groups/k/kr/krysalis/htdocs</dd>
</dl>
<dl>
<dt>deploy.scp.user</dt>
<dd>forrestbot</dd>
</dl>
<dl>
<dt>deploy.scp.dir</dt>
<dd>forrest</dd>
</dl>
</ul>
<li>and CALL the following templates
(template.workstage(.@type)?):</li>
<ul>
<li>template.prepare</li>
<li>template.get-src.cvs</li>
<li>template.generate</li>
<li>template.deploy.scp</li>
<li>template.cleanup</li>
</ul>
</ul>
<p>What each of these templates is going to do with the parameters can
be found in the templates.build.xml itself (at this stage) and is put out to
the console if the bot is run with the <code>-Dtemplate.echo=true</code>
flag</p>
<section>
<title>Setting bot-wide defaults</title>
<p>The <code>&lt;defaults&gt;</code> section in the
<code>&lt;forrest-config&gt;</code> allows for one-time setting of parameters
you want to reuse for different projects. The syntax (and semantic value) of
the nested elements is identical to the use inside the
<code>&lt;project&gt;</code> tags.</p>
</section>
</section>
<note>Once you understand all of this you might decide NOT to run your
own centralized service but rather ask your project to be taken up in the
centralized service of the project team itself: just send a mail to
forrest-dev <link href="site:mail-lists">mail list</link>
defining the required parameters to be taken up in the
process.</note>
<section>
<title>Checking logs and getting them by mail</title>
<p>For each distinct project the Forrest bot is maintaining a separate
working log of the different workstages being executed. These logs can be
opened and viewed in the ./build/bot directory</p>
<p>In addition to the @name attribute the &lt;project&gt; tag also can
contain a @sendlogto attribute that must be holding a (comma separated list
off) email address(es) where the project specific log needs to be sent to after
all workstages have been completed (or as soon as building failed)</p>
</section>
</section>
<section>
<title>Current supported workstages and types.</title>
<section>
<title>
<code>&lt;prepare&gt;</code>
</title>
<p>No specific type versions. (don't use type-attribute)</p>
<p>Template arguments:</p>
<ul>
<li><code>&lt;skin&gt;</code> @name holds the name of the skin to use.</li>
</ul>
</section>
<section>
<title>
<code>&lt;get-src ...&gt;</code>
</title>
<section>
<title>
<code>&lt;get-src type=&quot;cvs&quot;&gt;</code>
</title>
<p>Template arguments:</p>
<ul>
<li><code>&lt;host&gt;</code> @name holds the ipaddress or dns name of the cvs serving host</li>
<li><code>&lt;root&gt;</code> @name holds the cvs-root directory on that host</li>
<li><code>&lt;user&gt;</code> @name holds the username to use on the cvs repository </li>
<li><code>&lt;passwd&gt;</code> @name holds the password to use on the cvs repository</li>
<li><code>&lt;module&gt;</code> @name holds the module name that holds the </li>
<li><code>&lt;dir&gt;</code> @name holds the relative path to the {docroot} directory. (This is the dir that is holding the ./content/xdocs as specified in t. </li>
</ul>
</section>
<section>
<title>
<code>&lt;get-src type=&quot;local-copy&quot;&gt;</code>
</title>
<p>Template arguments:</p>
<ul>
<li><code>&lt;content-dir/&gt;</code> @name holds the path pointing to the {docroot} directory. (See the <link href="site:forrest-contract">contract</link> to understand what should be there.)</li>
<li><code>&lt;project-dir/&gt;</code> @name holds the path pointing to the {projecthome} directory where the xml project descriptors reside.</li>
</ul>
</section>
</section>
<section>
<title>
<code>&lt;generate&gt;</code>
</title>
<p>No specific type versions. (don't use type-attribute)</p>
<p>Template arguments:</p>
<ul>
<li><code>&lt;debuglevel&gt;</code> @name holds the threshold-level for showing debug statements in the cocoon generation process.</li>
</ul>
</section>
<section>
<title>
<code>&lt;deploy ...&gt;</code>
</title>
<section>
<title>
<code>&lt;deploy type=&quot;scp&quot;&gt;</code>
</title>
<p>Template arguments:</p>
<ul>
<li><code>&lt;host&gt;</code> @name holds the remote host where the file will be copied to.</li>
<li><code>&lt;user&gt;</code> @name holds the username to be used when logging onto the remote host.</li>
<li><code>&lt;root&gt;</code> @name holds the prefix part of where the content needs to be published.</li>
<li><code>&lt;dir&gt;</code> @name holds the suffix part of where the content needs to be published.</li>
</ul>
<warning>Using this approach requires that the public key of the user executing the bot target is present in the authorized key file (<code>~/.ssh/authorized_keys2</code>) of the remote user (on the remote host).</warning>
<note>Might be interesting to understand that the full process actually is bundling all files to copy in a *.tar.gz that is un-tar.gzipped on the remote host using ssh. </note>
</section>
<section>
<title>
<code>&lt;deploy type=&quot;local-copy&quot;&gt;</code>
</title>
<p>Template arguments:</p>
<ul>
<li><code>&lt;destination&gt;</code> @name holds the path where the generated content needs to be copied to.</li>
</ul>
</section>
<section>
<title>
<code>&lt;deploy type=&quot;ftp&quot;&gt;</code>
</title>
<p>Template arguments:</p>
<ul>
<li><code>&lt;host&gt;</code> @name holds the hostname of the ftp server to publish to.</li>
<li><code>&lt;user&gt;</code> @name holds the username to use for loging onto the ftp server.</li>
<li><code>&lt;passwd&gt;</code> @name holds the password to use.</li>
<li><code>&lt;destination&gt;</code> @name holds the path to the directory on the remote host where the generated content needs to be published. The process is using the <code>ftp cd</code> command to get there. This means this directory has to exist.</li>
</ul>
</section>
<warning>Currently the ant distribution that is included in forrest cvs is not offering the required NetComponents.jar to actually support this type of deployment. If you want to use it anyway you should get the required jar from <link href="http://www.savarese.org/oro/downloads/index.html#NetComponents">http://www.savarese.org/oro/downloads</link> and drop that into your <code>{forrest-sandbox}/tools/antipede/lib</code> directory. </warning>
</section>
<section>
<title>
<code>&lt;cleanup&gt;</code>
</title>
<p>No specific type versions. (don't use type-attribute)</p>
<p>Template arguments: None.</p>
</section>
</section>
<section>
<title>ForrestBot design</title>
<p>Most of us will just like things to work, and will be happy enough
just using it, maybe even letting the forrest-dev
<link href="site:mail-lists">mail list</link> know if we could improve
this or the other. Some however might be drawn to the
<link href="http://www.howstuffworks.com/">how stuff works</link> of things.
For them is this section as a start in the rest of their own pursuit.</p>
<section>
<title>The Ant play</title>
<p>All actions and targets related to the actual execution of the bot's
work is separated over 3 distinct build files:</p>
<ul>
<dl>
<dt>[main] <code>./build.xml</code></dt>
<dd>the starting point where generation and preparation is initiated.
</dd>
</dl>
<dl>
<dt>[generated] <code>./build/bot/work.build.xml</code></dt>
<dd>the one that reflects the configured tasks in the
forrestbot.conf.xml</dd>
</dl>
<dl>
<dt>[templates]
<code>./src/resources/forrestbot/ant/templates.build.xml</code></dt>
<dd>listing the different targets that actually depict the supported
workstages (and types of)</dd>
</dl>
</ul>
<p>The complete process runs as follows:</p>
<ol>
<li>'bot' target inside main build.xml depends on a number of targets
that</li>
<ol>
<li>'bot.init': initialize the environment</li>
<li>'bot.conf2build': use XSLT stylesheets to convert the
forrestbot.conf.xml into the work.build.xml (and a properties file for the
default-section)</li>
<li>'bot.run': delegates to the 'work' target of the generated
work.build.xml</li>
</ol>
<li>'work' target inside generated work.build.xml will then start in
parallel (so failure of the one would not influence the other) the different
project specific work-processes. </li>
<li>each of these project runs (generated 'work.[projectname]' targets)
depends on the sequential execution of the different workstage actions
(generated '[workstage].[projectname(.[type])?]' targets) </li>
<li>In those specific targets the pattern is as follows:</li>
<ol>
<li>set the project specific parameters for this project and
workstage</li>
<li>load the default parameter values for the ones that aren't set
yet</li>
<li>call the actual template.[workstage(.[type]?)]</li>
</ol>
</ol>
</section>
<section>
<title>Adding features to the bot - The meta template</title>
<p>Given the mechanism, the actual things the bot <strong>can</strong>
do boil down to the template targets it can call. So extending the bot means:
adding templates-targets.</p>
<p>If you want to add your own tempales to
<code>template.build.xml</code>, take in account the following guidelines:</p>
<ul>
<li> in the future there will be a DTD for forrestbot.conf.xml that
should be updated probably after you added stuff to the
<code>templates.build.xml</code>. Backwards-incompatible changes need a formal
vote.</li>
<li>build an echo target for each template target you make: </li>
</ul>
<source>
<![CDATA[
<target name="echo.workstage-name.optional-type" if="template.echo">
... here you put echo ant-tasks that somewhat tell what you will
... do with all the params you expect
</target>
<target name="template.workstage-name.optional-type"
depends="shared.set-props, echo.workstage-name.optional-type">
... here you put whatever ant-tasks that actually do it
</target>
]]>
</source>
</section>
</section>
</body>
</document>