Table of Contents

• 1. Rocketmq-redis-replicator
  • 1.1. Brief introduction
  • 1.2. Architecture
• 2. Install
  • 2.1. Requirements
  • 2.2. Install from source code
• 3. Simple usage
  • 3.1. Downstream via socket
  • 3.2. Deploy as an independent service
  • 3.3. Consume redis event
• 4. Configuration
  • 4.1. Rocketmq configuration
  • 4.2. Specify your own configuration
• 5. Other topics
  • 5.1. Built-in command parser
  • 5.2. EOFException
  • 5.3. Trace event log
  • 5.4. Auth
  • 5.5. Avoid full sync

1. Rocketmq-redis-replicator

1.1. Brief introduction

Rocketmq redis replicator implement Redis Replication protocol written in java. It can parse, filter, broadcast the RDB and AOF events in a real time manner and downstream these event to RocketMQ.

1.2. Architecture

+-------+     PSNC      +--------------+
|       |<--------------|              |   event      +--------------+
| Redis |               |              |------------->|              |
|       |-------------->|Rocketmq-redis|   event      |              |
+-------+     data      | (parse data) |------------->|   Rocketmq   |
                        |              |   event      |              |
                        |              |------------->|              |
                        +--------------+              +--------------+

2. Install

2.1. Requirements

jdk 1.8+
maven-3.3.1+
redis 2.6 - 5.0.x
rocketmq 4.2.0 or higher

2.2. Install from source code

    $mvn clean install package -Dmaven.test.skip=true

3. Simple usage

3.1. Downstream via socket

        Configure configure = new Configure();
        Replicator replicator = new RocketMQRedisReplicator(configure);
        final RocketMQRedisProducer producer = new RocketMQRedisProducer(configure);
        producer.open();
        replicator.addEventListener(new EventListener() {
            @Override public void onEvent(Replicator replicator, Event event) {
                try {
                    if (!producer.send(event)) {
                        LOGGER.error("Failed to send Event");
                    }
                } catch (Exception e) {
                    LOGGER.error("Failed to send Event", e);
                }
            }
        });
        
        replicator.addCloseListener(new CloseListener() {
            @Override public void handle(Replicator replicator) {
                producer.close();
            }
        });

        replicator.open();

3.2. Deploy as an independent service

1. mvn clean package -Dmaven.test.skip
2. sh target/rocketmq-redis-pack/bin/start.sh

3.3. Consume redis event

        Configure configure = new Configure();
        RocketMQRedisConsumer consumer = new RocketMQRedisConsumer(configure);
        consumer.addEventListener(new EventListener() {
            @Override public void onEvent(Event event) {
                if (event instanceof PreRdbSyncEvent) {
                    // pre rdb sync
                    // your code goes here
                } else if (event instanceof AuxField) {
                    // rdb aux field event
                    // your code goes here
                } else if (event instanceof KeyValuePair) {
                    // rdb event
                    // your code goes here
                } else if (event instanceof PostRdbSyncEvent) {
                    // post full sync
                    // your code goes here
                } else if (event instanceof Command) {
                    // aof command event
                    // your code goes here
                } else if (event instanceof PreCommandSyncEvent) {
                    // pre command sync
                    // your code goes here
                } else if (event instanceof PostCommandSyncEvent) {
                    // post command sync
                    // your code goes here
                }
            }
        });
        consumer.open();

4. Configuration

The config file located at target/rocketmq-redis-pack/conf/replicator.conf

4.1. Rocketmq configuration

4.2. Specify your own configuration

By default the configuration file replicator.conf loaded from your classpath.
But you can specify your own configuration using Configure like following:

        Properties properties = new Properties()
        properties.setProperty("zookeeper.address", "127.0.0.1:2181");
        properties.setProperty("redis.uri", "redis://127.0.0.1:6379");
        properties.setProperty("rocketmq.nameserver.address", "localhost:9876");
        properties.setProperty("rocketmq.producer.groupname", "REDIS_REPLICATOR_PRODUCER_GROUP");
        properties.setProperty("rocketmq.consumer.groupname", "REDIS_REPLICATOR_CONSUMER_GROUP");
        properties.setProperty("rocketmq.data.topic", "redisdata");
        properties.setProperty("deploy.model", "single");
        Configure configure = new Configure(properties);
        

5. Other topics

5.1. Built-in command parser

5.2. EOFException

• Adjust redis server setting like the following. more details please refer to redis.conf

    client-output-buffer-limit slave 0 0 0

WARNNING: this setting may run out of memory of redis server in some cases.

5.3. Trace event log

• If you are using log4j2, add logger like the following:

    <Logger name="com.moilioncircle" level="info">
        <AppenderRef ref="YourAppender"/>
    </Logger>

    // redis uri
    "redis://127.0.0.1:6379?verbose=yes"

5.4. Auth

    // redis uri
    "redis://127.0.0.1:6379?authPassword=foobared"

5.5. Avoid full sync

• Adjust redis server setting like the following

    repl-backlog-size
    repl-backlog-ttl
    repl-ping-slave-periods

repl-ping-slave-period MUST less than readTimeout, default readTimeout is 30 seconds