<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements.  See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License.  You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<HTML>
<BODY>

<P>Contains the implementation of the external JMX APIs from
<a href="{@docRoot}/org/apache/geode/admin/jmx/package-summary.html#package_description">org.apache.geode.admin.jmx</a>.</P>

<H2>JMX Support in GemFire</H2>

Our goal was to provide JMX administrative support in GemFire.  The design was influenced by these important factors:
<p>
<ul>
<li/>1) Requirement to not impact performance of the product
<li/>2) Desire to introduce JMX without altering the existing product or the management console
<li/>3) Similar functionality already existed in the console and the internal.admin pkg which the console uses
<li/>4) Requirement to also introduce a simple and usable Admin API which or may not be related to JMX
</ul>

From a functional stand point, the JMX support was supposed to provide most of the same administrative and operational monitoring that the console already provides.  In some cases we limited the functionality due to security concerns and in others because it was hard to express the features as JMX beans.  The JMX Agent also provides some functionality (such as Health monitoring) that is not currently in the console.
<p>
The Agent communicates with the distributed system using the same distribution manager {@link org.apache.geode.distributed.internal.DistributionManager} as the console uses and thus has the same requirements that determine what distributed system it can manage.  Although the Console currently supports managing multiple distributed systems, we decided that a given Agent should only be able to manage a single system.  We have not tested the use of more than one Agent for the same system, however nothing currently prohibits this.
<p>
We decided to develop a simple public Admin API which in essence wraps the internal.admin API that the Console currently uses extensively.  The Admin API also contains implementations of new functionality not in internal.admin.  Our JMX support is an extension to this Admin API.  In an overly simplified view, the GemFire JMX MBeans are ModelMBeans that manage instances of the Admin API objects housed in the Agent's MBeanServer.
<p>
The selected architecture consists of a Daemon Agent, which exists in a separate VM that GemFire does not depend on.  This Agent hosts an MBeanServer, instances of any and all MBeans registered for managing a GemFire distributed system, and server connectors/adaptors that various types of clients can connect to.
<p>
The two server connectors we selected are the HttpAdaptor and the RMI Connector.  The HttpAdaptor provides an HTML user interface of all MBeans in the MBeanServer.  Although this generic UI is not as rich as an RMI client (or the GemFire Console) could be, it provides a functional and easy to use UI with no development required.  The JMX Remote specification details the standard connectors.  Although the HttpAdaptor is not required by this Sun spec. it is included in some form with all JMX implementations that I investigated.  It should be noted that our JMX Agent currently starts up the HttpAdaptor, but not the RMI Connector.  The latter is deferred as later work since some form of client must be developed for testing.  Further research may also uncover a generic, configurable open-source RMI client for JMX.  The GemFire Console could in theory be reworked as an RMI Connector client, but this is not currently planned.
<p>
Two open-source JMX implementations made it to our final review for consideration: <a href="http://www.xmojo.org">XMOJO</a> and <a href="http://www.mx4j.org">MX4J</a>.  The decision to go with MX4J was based mainly on our perceptions of MX4J being more active and widespread in use.  Additionally, XMOJO is associated with <a href="http://www.adventnet.com/">AdventNet</a> which produces commercial products.  This made MX4J seem more true to open-source and safer from corporate tampering.
<p>
ModelMBeans are very dynamic and capable of managing aggregate resources.  Use of a ModelMBean entails specifying meta-data to an instance of javax.management.modelmbean.RequiredModelMBean.  This meta-data identifies the manageble resource(s) which can consist of a single object, or many objects, including those in one VM or in any number of distributed VMs.  We decided to subclass classes in the Admin API in order to massage them a little and make them easier to use as a managed resource by the ModelMBean.  For example, org.apache.geode.admin.GemFireManager represents a type of member in a GemFire system which manages shared memory.  When an MBean is registered for managing the GemFireManager, the JMX Agent instantiates a "JMX friendlier" subclass: org.apache.geode.admin.jmx.internal.GemFireMangerJmxImpl.  Comparison of this class with the non-JMX super class org.apache.geode.admin.internal.GemFireManagerImpl will illustrate what "JMX friendly" means better than trying to explain it here...
<p>
One standard approach to defining a ModelMBean is to programmatically
build the necessary meta-data.  The folks on the Tomcat approach
developed a better solution... an XML definition file which Jakarta
Commons-Modeler parses to create the meta-data objects required to
definie the ModelMBean.  We currently have our XML descriptor file at
org.apache.geode.admin.jmx.mbeans-descriptors.xml.
Commons-Modeler can be found at <A href="http://jakarta.apache.org/commons/modeler">http://jakarta.apache.org/commons/modeler/</A>
<p>
Here's a quick run-down of the Admin and JMX pkgs in GemFire...
<p>
<b>org.apache.geode.admin</b>
<ul>
<li/>interfaces describing GemFire entities and resources for managing or monitoring
</ul>
<p>
<b>org.apache.geode.admin.internal</b>
<ul>
<li/>implementations of the admin pkg which could be used to managing GemFire
<li/>recommendation is to create one set on non-JMX unit tests for these and then wrap that unit test within another test that is specific to its use in JMX
</ul>
<p>
<b>org.apache.geode.admin.jmx</b>
<ul>
<li/>Commons-Modeler descriptor file mbeans-descriptors.xml
<li/>AgentMBean - non-required interface that simply represents a public documentation of what the Agent does
<li/>VersionMBean and Version - unused (but functional) examples of a standard MBean (other MBeans as defined in mbeans-descriptors.xml also provide version info)
</ul>
<p>
<b>org.apache.geode.admin.jmx.internal</b>
<ul>
<li/>subclasses of org.apache.geode.admin.internal classes that use JMX services, implement ManagedResource, register ModelMBeans to manage themselves, and other JMX related details
<li/>ManagedResource - simple interface we use to create a more uniform approach in using JMX
<li/>Agent - application with main, registers HttpAdaptor, acts as a factory to DistributedSystemJmxImpl (which is the entry point to using JMX with a GemFire system)
<li/>AgentConfig - configuration for Agent that reads/writes to a properties file
<li/>MBeanUtil - utility class with static methods for defining MBeans, registering for timer-based refresh notifications, and lots of other JMX stuff
<li/>StatisticAttributeInfo and ConfigAttributeInfo - these are the results of refactoring common code from some of the JmxImpl classes that required runtime dynamic (ie, not known at design time) MBean attributes
<li/>SpecManagedBean - subclass of part of Commons-Modeler, this class acts as a workaround for a bug in Commons-Modeler (I hope to remove this after working with Jakarta-Commons to correct the bug)
<li/>AgentPrintStream - this is a hack workaround for suppressing warnings from Xalan that derive from a purported incompatibility between the XSLT sheets in MX4J and the version of Xalan in JDK1.4.2 - experts with Xalan recommend upgrading to the very latest version of Xalan which is not currently desired for GemFire
</ul>
<p>
<h3>Some caveats, workarounds, and GemFire bugs to be aware of:</h3>
<p>
1) MX4J uses XSLT style sheets that are intended to work with the latest version of Xalan.  JDK1.4.2 bundles an older version of Xalan which generates warnings.
<p>
2) MX4J's implementation of javax.management.modelmbean.RequiredModelMBean contains a bug such that any return type defined as an array of some object results in it attempting to Class.forName the class name with the "[]" still in the name of the class.  Our current workaround is to return java.lang.Object where we'd like to return an array of some non-primitive type.  I hope to look at this closer and work with MX4J to correct it (unless that latest code base at MX4J already has a fix).
<p>
3) Our MBeans currently have some return types of Admin interfaces and GemFire MBean types.  This is not recommended.  The correct approach is to return javax.management.ObjectName or an array of ObjectName so that remotability is not broken.  We have a bug filed to correct this in GemFire.
<p>
    4) Commons-Modeler provides a simple, incomplete implementation of ModelMBean called
    org.apache.commons.modeler.BaseModelMBean. We decided to use the standard RequiredModelMBean
    which all JMX implementations are required to supply. The JMX spec details several "managed
    resource types" that a ModelMBean can support. ObjectReference is the type that both MX4J's
    RequiredModelMBean and Modeler's BaseModelMBean support. However, MX4J is more strict in it's
    interpretation of the spec which spells it out as "ObjectReference". Modeler's BaseModelMBean
    performs no enforcement and simply assumes it is managing type ObjectReference. Modeler has a
    bug in org.apache.commons.modeler.ManagedBean because it specifies "objectReference" which is
    the incorrect case in comparison to the specification. I intend to work with the Jakarta-Commons
    folks to change this to comply with the spec. The Modeler will use BaseModelMBean by default
    even though it is actually depending on a JMX implementation such as MX4J or XMOJO.
    org.apache.geode.admin.jmx.internal.MBeanUtils tells Modeler to use RequiredModelMBean instead
    and also uses org.apache.geode.admin.jmx.internal.SpecManagedBean as a workaround to the whole
    "objectReference" issue. You could feasibly use org.apache.commons.modeler.BaseModelMBean or
    RequiredModelMBean.
<p>

<h3>Capabilities currently supported in GemFire via JMX:</h3>
<ul>
<li/>1) View overall system and its settings and start/stop all members
<li/>2) View all managers and applications
<li/>3) View and modify config (includes optional use of JMX Timer service to auto-refresh)
<li/>4) View any statistics (includes optional use of JMX Timer service to auto-refresh)
<li/>5) View member's cache and its statistics
<li/>6) View a cache's region and its attributes, statistics, but not the data
<li/>7) Health monitoring (ask David W. for more info... I haven't had a chance to look at this yet)
<li/>8) Create, start, and stop managers
</ul>
<p>
<h3>TODO:</h3>
<ul>
<li/>1) Creation and control of Locators
<li/>2) Enable the RMI Connector and build tests
<li/>3) Bind Address support (bug 30898)
<li/>4) SSL Configuration support
<li/>5) JMX Connector security
<li/>6) more thorough automated tests!!
<li/>7) Documentation of use w/ built-in Connectors (RMI and HTTP)
<li/>8) Documentation of use w/ AdventNet SNMP Adaptor
<li/>9) Documentation of use w/ selected commercial management products
<li/>10) Determine high-availability requirements if any (mentioned by Jags)
</ul>
See also <a href="#deferred_ops">deferred backlog from Hardening/Operations sprint</a>
<p>
It's very easy to use the JMX Monitor service to monitor an MBean's attribute and fire JMX Notifications.  Statistics and Config Parameters are "MBean attributes".

<p>
<h3>Using Third Party Management Consoles/Tools</h3>
There is information on <a href="http://www.adventnet.com/">AdventNet's website</a> detailing the use of their <a href="http://www.adventnet.com/products/snmpadaptor/index.html">SNMP Adaptor</a> to enable connectivity to <a href="http://www.openview.hp.com/">HP OpenView</a>, <a href="http://www.ibm.com/software/tivoli/">IBM Tivoli</a>, <a href="http://www3.ca.com/solutions/solution.asp?id=315">CA Unicenter</a>, and other SNMP managers.
<p>
Aside from using AdventNet's SNMP Adaptor, HPOpenView is the first I've seen of the big players offering what appears to be a way to plug our JMX support into their product for management of GemFire.  I haven't had a chance to look at it yet.  Most likely their <a href="http://www.openview.hp.com/products/spi/">SPI</a> supports writing a small amount of glue code to have their console become an RMI client to our JMX VM (which is essentially an MBeanServer "server" that hosts our MBeans).  I think it's unlikely that such vendors would want to support hosting of third party MBeans and the classpath/versioning headaches involved in that.  The JMX Remote specification really is meant to cover how such console's would perform the remote connections to custom MBeans in external VMs/products.  It is likely that the other management tool vendors such as Tivoli have a similar SPI available or in the works.
<p>
See HP Dev Resource Central's page on <a href="http://devresource.hp.com/jmx.htm">JMX</a> for info and links on using JMX to integrate with HP OpenView.  You can also sign up for <a href="http://www.hpdev.com/.docs/pg/5">HP Developer News</a>
<p>
<h3><a name="deferred_ops">Deferred Backlog from Hardening/Operations Sprint</a></h3>
Note: some of these tasks may no longer be worded properly or even needed...
<ul>
<li/>Provide a consolidated view of all cache Regions via JMX
<li/>Send an alert upon system failure, communication failure, etc.
<li/>Monitor cache system failures, comm failures, loader failures, etc via JMX
<li/>Detect/handle stuck shared memory locks via JMX
<li/>Provide a "combination view" (distributed system-wide) of certain statistics via JMX
<li/>Integrate with Tivoli and produce a document
<li/>Test managing caches via JMX with a large number of Regions, Entries, etc.
<li/>Create XDoclet module to auto-generate mbeans-descriptors.xml (Note: we're using JavaDoc 1.4.2 which is incompatible with XDoclet; Sun intends to restore compatibility in the next version of JavaDoc)
<li/>Investigate AdventNet's JMX administration console
<li/>View percentage of VM memory used by a Cache via JMX
<li/>Investigate modifying the GemFire Console to use JMX
<li/>Investigate providing secure management via JMX
<li/>Document security usage for JMX management
<li/>Investigate managing GemFire via BMC Patrol
<li/>Investigate managing GemFire via HP OpenView
<li/>Investigate managing GemFire via UniCenter
<li/>Submit fix to Jakrta Commons-Modeler for ObjectReference fix in ManagedBean
<li/>Submit fix to MX4J for classload error in RequiredModelMBean
<li/>Enable the RMI Connector (currently commented out in Agent)
<li/>Create tests for RMI Connector
<li/>Document and JavaDoc usage of the RMI Connector
<li/>Add support to MBeanUtil to determine which MBeanServer the Agent is in
<li/>Correlate information from GemFire MBeans with MBeans from other products
<li/>Test deploying GemFire MBeans into other products' MBean containers
</ul>

</BODY>
</HTML>