==============================
Next actions
==============================

- Membership coordinator logic and membership view Id.
 
- Implement primary/secondary replication logic (fhanik)
  Idea for implementation: Cookie stores info primary/backup node and hit counter.
  Steps to implement: (coming soon)
  Scenarios: 
  1. Request in: new session dest: any
  2. Request in: existing session dest: primary server
  3. Request in: existing session dest: non-primary server (fail over)
  4. Request in: existing session dest: backup server 
  5. Backup node crash
  6. Primary node crash (step 3)
  7. Sequence 6. then 2. (hit counter)
  8. Sequence 5. then 4. (swap)
  9. New member joins 
  10. Backup and primary crash -> death!
  11. Sequence 2. in: invalid hit counter 
  12. Sequence 4. in: invalid hit counter
  13. Session expires
  14. Sequence 3. then 13. on the old primary
  
  Notes:
  Ability to add a custom "Session expired" redirect for global 
  redirects to never forward an invalid session into the server
  when the session data has gone lost.
  Ie, in a normal scenario, the session is viewed as "new"
  but they may wanna display a session expired redirect.
  Security:
  Dont store any sensitive info in the new cookie.
  only pointers to members.

  
- break out membership/messaging module and improve JMX
  The clustering code has become very cluttered.
  We need to break out the membership and the messaging 
  away from the logic of session handling and JMX.
  The latter should be built on top of the clustering module, and not baked into it.
  (maybe complete refactor for Tomcat 6)
  Steps:
  1. Build a group object, to allow sending of messages
     The group object will handle encompass membership and 
     messaging under one umbrella.
  2. Enable group RPC
     Through the group object, enable RPC style messaging
     This should be used for message like state transfers, instead of waiting for a timeout.
     RCP strategies - first response, majority response, all response
  3. Group/Message interceptors - to perform logic such as fragmentation, ordering etc.
  4. JMX  
     JMX should be added on top of the core code, not inside the core code.
     The idea of JMX as management should be added outside to the core components.
     For example, SimpleTCPCluster should not contain JMX code, if I want to create
     a different cluster class, I have to rewrite all the JMX code again.
     Instead, create JMX beans that reference core components through interfaces.

- Create stats object
   Instead of keeping stats embedded in the code, make it reusable in
   a shared component.

- support CrossContext session replication
    Idea: use endAccess to signal replicationValve that session from other app has changed.
    refactor ReplicationValve 
    When session is serialzed from other app, set right app classloader
    endaccess is also send at CoyoteAdpater for context session as recycle the request
- Add MbeanFactory to generate dynamic cluster at runtime.
    Problem: How we can start those central services?
    - StandardEngine support load Mbean from external file.     
- when a lot of messages expire it comes to burst of messages 
     - all 60 Sec when ManagerBase#processExpires is called a lot of messages are send!
     - Better is to transfer a spezial epxire message with an array of expired session messages.
     - This reduce message transfer and reduce waits for acks.
     - complicated implementation thing , sessions expires when call isValid :-(     
- build a tool that receive the stage and present it as simple web app.
    	detect some problems
    	different active sessions
    	long queues
    	detect long wait acks
        Idea: Wrote a ant script with the new jmx tasks!
        Optimzied information access:
        	Create a MBean.attribute that store the complete cluster state 
        	information inside one attribute from type TabularData and CompositeData!
        	Implement as SimpleTcpCluster operation
        	Cluster <CompositeData>
        	  -- Attributes <SimpleType>
        	  -- ClusterMembership
        	      -- Attributes <SimpleType>
        	      -- Members <CompositeType>
        	         -- Attributes <SimpeType>
        	  -- ClusterReceiver
        	      -- Attributes <SimpleType>
        	  -- ClusterSender
        	      -- Attributes <SimpleType>
        	      -- DataSenders <TabularData>
        	         -- Sender <CompositeType>
        	         	-- Attributes <SimpeType>
        	         	-- (optional)Queue Stats <CompositeData>
 	       	         		-- Attributes <SimpeType>      	    
- add cluster setup template (src)	
- documentation
	wrote a complete new how-to
	add example configurations
	add complete attribute descriptions
	add JMX information
	add deployment help
	Need help (pero)
- implement fragmentation of large replication objects
  (note this will be done in the Group object, see top.)
	Compress at message level - this will go away! instead there will be a GZip interceptor
	Splitting Messages ala FarmDeployer war handling
- add a message type to the message header.
    - filtering at receiver that drop message before build Object
    - define short type definition.
       - Every Session message with differenz type
       - type with String
- add test cluster project
	functional testing a lot of szenarios
		differen replication mode
		restart after failure
		crash under load
		compress and uncompressed
    Junit test ( started)
    automated regression testing with some standard configs
    wrote a test client that can get a JMX State to verify the different cluster testszenarios
	 - direct JSR 160 client - preferred option
	 - via mx4j http adaptor ( xml protocol)
	 - setup different szenarios
	use new remote jmx ant tasks to grab information from mbeans 
- filter all cluster messages 
     - Filtering that no all message send to all member.
     - Now with domain mode registeration session message only send to
       members from same domain.
- setup a Cluster LifecycleListener to send the cluster status! (monitoring)
     - Make the compact state via JMX API avialable first.
     - setup cluster Listener that send own state to spezial member (sender from a message like GET_ALL_SESSIONS)
     - create a cluster status app (html/xml)
         - attributes
            - send message
            - current members
            - receive message
            - active session at different clustered manager
            - which app are clusters (registered managers)
            - avg processing times
         
         - operations
            - resetStats
            - send a message (String)
            - getDisplay state from other members??
            - stop queue to send
            - queue receiver message from some members ( later send after redeploying)
            - configure some send parameters
                 - keepalive
                 - compress
                 - wait for ack
         - other things
            - based on Cluster JMX API                     
            - watch some values from complete cluster and display some graphs
            - display the informations from all nodes
            - display informations from other cluster domains
            	via XML documents and http
            - display stats as xml
            - operation via JMX (MX4J adaptor)	    

================
problems
================ 
- How we can stop the request traffic when restart an application?
    currently the jk 1.2.10 can only disable the complete loadbalancer,
    but this detect only the new session request desicion.
    Request with sessions marks send to tomcat. 
    Fix: jk > 1.2.11 has a stopped flag, but then all application stop traffic
    and session transfer from other nodes not stopped!!
    	Stop MembershipService via JMX and after redeploying application,
    	start MembershipService and restart the application Session Manager or send GETALL Session from backup?
    	then enable apache worker again !! :-(
    	easier stop tomcats, deploy new app and start again... 
- Can't stop message replication for a spezial member and application
   - this need a spezial cluster message and send filter at SimpleTcpCluster 
- Don't generate cluster message when no member is at cluster!
   - Register DeltaManager as Cluster LifecycleListener and stop cresting and sending
   - Reduce memory consume when only local node is active
   - Important feature when nodes crashed, and only one server exists under load...
   
==============================
Nice to have:
==============================
- Replication ContextAttributes	
- Configure the McastService no accept every member.
	- receive a secret key
	- have a allow or deny list like RequestFilterValve
	- Also receiver don't accept request from not allowed members
- ReplicationListener and SocketReplicationListener only accept data from cluster member (low level ip restriction)
- PooledSocketSender
	Add more stats
	check all Pooled sender checkKeepAlive
- Implement a NonSerializable interface for session attributes that do not
wish to be replicated
        Then we must have ClusterNonSerializable at common classloader
- Extend StandardSession if possible
- Implement context attribute replication (?)
	pero:
		Also send Start/Stop messages from Context to complete cluster!
        With 5.5.10 you can wrote a Cluster Lifecycle Manager that do this.
            Register for AFTER_MANAGERREGISTER_EVENT and AFTER_MANAGERUNREGISTER_EVENT
            and also to the context ServletContextAttributeEvent Listener
            Access the Context
            	((Manager)event.getData()).getContainer()                     	
- Fix farm deployment for 5.5
   pero:
     Every start all application are deployed only to all running cluster nodes
     	New registered nodes don't get the applications!
     	Deploy must send a GET_ALL_APP to all other Deployers.
        FH: Correction, you should not send GET_ALL_APP to all deployers, only
to the main one. Which could be another property of the Member object, it
would not make sense to transfer the same webapp over and over again.
     		Only watchEnabled Deployer send this member all deployed application.
		pero:
		   Yes very true, but currently we distribute also all wars from watch node at begining.
		   Waiting to start other nodes is only change to not got these war's!
		   I have made some experiments to register war deployment at new memberAdded to cluster.	   
     Add JMX Support
     	Resend Deployed Applications to all or one cluster node.
     	Show all watch Resource
     	Processing Time
     	Change fileMessage Buffersize.
     	Start/Stop Cluster wide application
     Deployer and Watcher sync with engine background thread! 
     	Fixed!		
     Last FileMessage fragment need longe ackTimeout	
    	<Cluster ..> <Sender ... ackTimeout="60000"/> </Cluster> 	
- Change the cluster protocol that developer can add there own data serialzable/deserialzable format (high risk)
  Note from Filip: Lets not do this, last attempt caused a broken clustering piece for 4 versions. 
   Currently 
   	header 		    6 bytes (FLT2002)
    compressflag    4 byte
   	data.length     4 bytes
   	data, 
   	end header      6 bytes (TLF2003)
   	
   Optimized to 
      header 		2 bytes (TC)
      type   		1 byte
      compressflag  1 byte
      data.length 	4 bytes, 
      data | <real uncompressed data.length (4 bytes)> data
      
      "type" means user defined type and receiver extract bytes and type and sende it to callback
      	s. ObjectReader or SocketObjectReader
                       
       compress     1
       first data 4 data bytes are the real uncompressed data length. ( Is for better memory management atr recevier side, S. XByteBuffer)
       
   change at DataSender.writeData and XByteBuffer and add flexible handling to ClusterSender and ClusterReceiver
   
- Add single sign on support

==============================   
COMPLETED
==============================
5.5.16 (fhanik)
- Extensive logging when a member crashes
  When a member crashes and you use pooled socket mode then 
  the logs are spewing out info.
  Instead, now we only report one IOException if it happens, and 
  mark the member as suspect.

5.5.16 (fhanik)
- Average message size
  For my test application, the average message size has increased from
  526 bytes per message to over 750 bytes per message from version 5.0.28 to 5.5.15.
  Need to investigate why this is the case and where the 200+ extra bytes are coming from.
  Fix: The default compress value had changed. The new default is that no compression is used.
  To re-enable compression, <Sender compress="true" in the sender element.

5.5.15 (pero)
- Optimized getMembers access from McastMembership
    Member list was copied at every access. (Used inside ReplicationValve.invoke)
    SimpleTcpCluster need a alive time sorted array

5.5.11 (pero)
- MemoryUser principal from UserDatabaseRealm not handled to replicated 
	- look inside DeltaRequest.setPrincipal(Principal,boolean)
		 detected by Dirk de Kok (tomdev 16.8.2005)
	- only GenericPrincipal from all other realms are handled well.
    - change JAASRealm and UserDatabaseRealm that also used GenericPrincipal
5.5.10 (pero)
- Cluster config at engine level (user request 06/05)
	 Register a cluster infrastructur for many vhosts
	 configure backup systems!
	 Add Cluster Element to digister		
- add mapping sender mapping properties file (IDataSenderFactory)
	- let advanced people eaiser implemented there own sender mode
- We register different application with same name from different host?
	SimpleTcpManager register manager with app name + hostname when Cluster is configured as Engine element.
- Configured DeltaManager inside context
	- SimpleTcpCluster setProperty and transferproperty reflect changes only to defaultMode managers
    - Look inside SimpleTcpCluster.addManager and DeltaManager.start?
- Session serialization eat memory but now we can send session messages with blocks...
	When all sessions serialze after GET_ALL_SESSION is received following works
	- find all sessions
	- serialize a block or all sessions as byte array
	- serialize the complete SessionMessageImpl to transfer message
- WaitForAck mode and resend probleme
   - Now message creator can configure resend and compress mode!
- Add a default simple cluster config with good defaults and only
  one cluster element inside server.xml. Setup with fastasyncmode.
  Service Elements
  	ReplicationTransmitter,
  	SocketReplicationListener,
  	McastService,
  	ClusterSessionListener
  	ReplicationValve
  You can change property setting with SimpleTcpCluster prefix "sender.XXX, receiver.XXX, valve.XXX, listener.XXX, service.XXX"	
- Fix resend GET_ALL_SESSIONS when wait ACK failed at receiver side
- Fix that ClusterValve not remove when cluster stops 
- Set timestamp only at first time inside SessionMessageImpl
- Set timestamp from findsessions when handling GET_ALL_SESSION
    - Set this timestamp to all SEND_SESSION_DATA and TRANSFER Complete messages
    - Drop all received message inside GET_ALL_SESSION message queue (DeltaManager)    

- Mcast Service as JMX MBean (change cluster domain at runtime)    
- send cluster domain with mcast ping
    - With sendClusterDomainOnly=true only session message from same domain are received
    - Session only replicated to members from same domain, with sendClusterDomainOnly=false
      at Sender (ReplicationTransmitter) session messages send to all members.
    - GET ALL Session send to first member inside same cluster domain
- better restart szenario at DeltaManager after failure restart (java service wrapper).
   queue all other session events
   as STATE Transfer Complete is received, dequeue all received sessions messages.   
- restructure methods at DeltaManager
- extract handleXXS methods for better DeltaManager subclassing.
- split big get all sessions from one server into blocks of sessions and separate STATE Transfer message!
- no complete sync sessions when GET ALL Sessions event is received.
- add JMX API for ClusteRreceiver
- ClusterReceiver is now Callback when message is received
- SimpleTcpCluster only receive ClusterMessage (API change)
       Redesign SimpleTcpCluster message receiving to ClusterReceiverBase:
          - optimized data uncompressed
          - better extendablity
          - XByteBuffer only buffer bytes and don't uncompress.
          - Add receiver JMX stats with new attribute 'Receiver.doReceivedProcessingStats'   
          - optimized createManager and addManager that also can configured normal StandardManager 
            to use cluster message transfer without replication.
- add support to dynamic property transfer from SimpleTcpCluster to the Manager
    like ReplicationTransmitter
        All manager attributes can be configured:
          - expireSessionsOnShutdown (false)
          - notifySessionListenersOnReplication (true)
          - notifyListenersOnReplication (true)
          - maxActiveSessions (-1)
          - timeoutAllSession (60 sec)
          - sessionIdLength (16)
          - processExpiresFrequency (6 - exipre all six engine background periodic event (60 sec))
          - algorithm (MD5)
          - entropy
          - randomFile
          - randomClass
- Setup the cluster without SessionReplication Manager
         Only a message bus.
         Configure the bus with you ClusterListener and valves and a StandardManager         
- reduce cpu and memory consume (Receiver)
    - set new compress sender flag at default=false ( < CPU usage)
    - Make compact algo 
        currently message receive data is split at XbyteBuffer#extractPackage and
        SimpleTcpCluster#messageDataReceived     
- reduce memory and cpu consume (send message)
    - set new compress sender flag at default=false ( < CPU usage)
    - don't copy the buffer to add message header
        transfer this from SimpleTcpCluster to DataSender pushMessage
        successfull refactored
    - make it possible that a subclass crypt the transfered messages
       sub class ReplicationTransmitter and override createMessageData
    - don't copy START and END Header for every message, instead send dirctly and DataSender.writeData.   
- Add a flag for replicated attribute events, to enable or disable them 
     Now can configued with notifyListenersOnReplication=false at SimpleTCPCluster
     Also can drop HttpSessionLsitener events
         can configued with notifySessionListenersOnReplication=false at SimpleTCPCluster
- Refactoring DeltaManager
- Transfer attributes from Cluster config to DeltaManager
- Fix at 5.5.9 cluster hang bug! 
- Add more Valve to direct cluster config
- Add Lifecycle Listener support to direct cluster config
- Add ClusterListener support to direct cluster config
- Add new SocketReplicationListener
- Add Stats to DeltaManager

5.5.9  (pero)
- JMX friendly
    pero:  Add some MBeanSupport to SimpleTCPCluster, ReplicationTransmitter and Senders
- Add Keep Alive and WaitForAck at async mode implementation.
       Make this feature configurable to Sender element at server.xml
       Is include with 5.5.8
- Add support to new Async Mode from Rainer Jung
       Integrated with 5.5.9
       fastasyncqueue mode