docs/operations/operations.rst - distributedlog - Git at Google

 DistributedLog Operations
 =========================

 Feature Provider
 ~~~~~~~~~~~~~~~~

 DistributedLog uses a `feature-provider` library provided by Apache BookKeeper for managing features
 dynamically at runtime. It is a feature-flag_ system used to proportionally control what features
 are enabled for the system. In other words, it is a way of altering the control in a system without
 restarting it. It can be used during all stages of development, its most visible use case is on
 production. For instance, during a production release, you can enable or disable individual features,
 control the data flow through the system, thereby minimizing risk of system failure in real time.

 .. _feature-flag: https://en.wikipedia.org/wiki/Feature_toggle

 This `feature-provider` interface is pluggable and easy to integrate with any configuration management
 system.

 API
 ___

 `FeatureProvider` is a provider that manages features under different scopes. The provider is responsible
 for loading features dynamically at runtime. A `Feature` is a numeric flag that control how much percentage
 of this feature will be available to the system - the number is called `availability`.

 ::

     Feature.name() => returns the name of this feature
     Feature.availability() => returns the availability of this feature
     Feature.isAvailable() => returns true if its availability is larger than 0; otherwise false


 It is easy to obtain a feature from the provider by just providing a feature name.

 ::

     FeatureProvider provider = ...;
     Feature feature = provider.getFeature("feature1"); // returns the feature named 'feature1'


 The `FeatureProvider` is scopable to allow creating features in a hierarchical way. For example, if a system
 is comprised of two subsystems, one is *cache*, while the other one is *storage*. so the features belong to
 different subsystems can be created under different scopes.

 ::

     FeatureProvider provider = ...;
     FeatureProvider cacheFeatureProvider = provider.scope("cache");
     FeatureProvider storageFeatureProvider = provider.scope("storage");
     Feature writeThroughFeature = cacheFeatureProvider.getFeature("write_through");
     Feature duralWriteFeature = storageFeatureProvider.getFeature("dural_write");

     // so the available features under `provider` are: (assume scopes are separated by '.')
     // - 'cache.write_through'
     // - 'storage.dural_write'


 The feature provider could be passed to `DistributedLogNamespaceBuilder` when building the namespace,
 thereby it would be used for controlling the features exposed under `DistributedLogNamespace`.

 ::

     FeatureProvider rootProvider = ...;
     FeatureProvider dlFeatureProvider = rootProvider.scope("dlog");
     DistributedLogNamespace namespace = DistributedLogNamespaceBuilder.newBuilder()
         .uri(uri)
         .conf(conf)
         .featureProvider(dlFeatureProvider)
         .build();


 The feature provider is loaded by reflection on distributedlog write proxy server. You could specify
 the feature provider class name as below. Otherwise it would use `DefaultFeatureProvider`, which disables
 all the features by default.

 ::

     featureProviderClass=com.twitter.distributedlog.feature.DynamicConfigurationFeatureProvider


 Configuration Based Feature Provider
 ____________________________________

 Beside `DefaultFeatureProvider`, distributedlog also provides a file-based feature provider - it loads
 the features from properties files.

 All the features and their availabilities are configured in properties file format. For example,

 ::

     cache.write_through=100
     storage.dural_write=0


 You could configure `featureProviderClass` in distributedlog configuration file by setting it to
 `com.twitter.distributedlog.feature.DynamicConfigurationFeatureProvider` to enable file-based feature
 provider. The feature provider will load the features from two files, one is base config file configured
 by `fileFeatureProviderBaseConfigPath`, while the other one is overlay config file configured by
 `fileFeatureProviderOverlayConfigPath`. Current implementation doesn't differentiate these two files
 too much other than the `overlay` config will override the settings in `base` config. It is recommended
 to have a base config file for storing the default availability values for your system and dynamically
 adjust the availability values in overlay config file.

 ::

     featureProviderClass=com.twitter.distributedlog.feature.DynamicConfigurationFeatureProvider
     fileFeatureProviderBaseConfigPath=/path/to/base/config
     fileFeatureProviderOverlayConfigPath=/path/to/overlay/config
     // how frequent we reload the config files
     dynamicConfigReloadIntervalSec=60


 Available Features
 __________________

 Check the :doc:`../references/features` reference page for the features exposed by DistributedLog.

 `dlog`
 ~~~~~~

 A CLI is provided for inspecting DistributedLog streams and metadata.

 .. code:: bash

    dlog
    JMX enabled by default
    Usage: dlog <command>
    where command is one of:
        local               Run distributedlog sandbox
        example             Run distributedlog example
        tool                Run distributedlog tool
        proxy_tool          Run distributedlog proxy tool to interact with proxies
        balancer            Run distributedlog balancer
        admin               Run distributedlog admin tool
        help                This help message

    or command is the full name of a class with a defined main() method.

    Environment variables:
        DLOG_LOG_CONF        Log4j configuration file (default $HOME/src/distributedlog/distributedlog-service/conf/log4j.properties)
        DLOG_EXTRA_OPTS      Extra options to be passed to the jvm
        DLOG_EXTRA_CLASSPATH Add extra paths to the dlog classpath

 These variable can also be set in conf/dlogenv.sh

 Create a stream
 _______________

 To create a stream:

 .. code:: bash

    dlog tool create -u <DL URI> -r <STREAM PREFIX> -e <STREAM EXPRESSION>


 List the streams
 ________________

 To list all the streams under a given DistributedLog namespace:

 .. code:: bash

    dlog tool list -u <DL URI>

 Show stream's information
 _________________________

 To view the metadata associated with a stream:

 .. code:: bash

    dlog tool show -u <DL URI> -s <STREAM NAME>


 Dump a stream
 _____________

 To dump the items inside a stream:

 .. code:: bash

    dlog tool dump -u <DL URI> -s <STREAM NAME> -o <START TXN ID> -l <NUM RECORDS>

 Delete a stream
 _______________

 To delete a stream, run:

 .. code:: bash

    dlog tool delete -u <DL URI> -s <STREAM NAME>


 Truncate a stream
 _________________

 Truncate the streams under a given DistributedLog namespace. You could specify a filter to match the streams that you want to truncate.

 There is a difference between the ``truncate`` and ``delete`` command. When you issue a ``truncate``, the data will be purge without removing the streams. A ``delete`` will delete the stream. You can pass the flag ``-delete`` to the ``truncate`` command to also delete the streams.

 .. code:: bash

    dlog tool truncate -u <DL URI>
	DistributedLog Operations
	=========================

	Feature Provider
	~~~~~~~~~~~~~~~~

	DistributedLog uses a `feature-provider` library provided by Apache BookKeeper for managing features
	dynamically at runtime. It is a feature-flag_ system used to proportionally control what features
	are enabled for the system. In other words, it is a way of altering the control in a system without
	restarting it. It can be used during all stages of development, its most visible use case is on
	production. For instance, during a production release, you can enable or disable individual features,
	control the data flow through the system, thereby minimizing risk of system failure in real time.

	.. _feature-flag: https://en.wikipedia.org/wiki/Feature_toggle

	This `feature-provider` interface is pluggable and easy to integrate with any configuration management
	system.

	API
	___

	`FeatureProvider` is a provider that manages features under different scopes. The provider is responsible
	for loading features dynamically at runtime. A `Feature` is a numeric flag that control how much percentage
	of this feature will be available to the system - the number is called `availability`.

	::

	Feature.name() => returns the name of this feature
	Feature.availability() => returns the availability of this feature
	Feature.isAvailable() => returns true if its availability is larger than 0; otherwise false


	It is easy to obtain a feature from the provider by just providing a feature name.

	::

	FeatureProvider provider = ...;
	Feature feature = provider.getFeature("feature1"); // returns the feature named 'feature1'


	The `FeatureProvider` is scopable to allow creating features in a hierarchical way. For example, if a system
	is comprised of two subsystems, one is cache, while the other one is storage. so the features belong to
	different subsystems can be created under different scopes.

	::

	FeatureProvider provider = ...;
	FeatureProvider cacheFeatureProvider = provider.scope("cache");
	FeatureProvider storageFeatureProvider = provider.scope("storage");
	Feature writeThroughFeature = cacheFeatureProvider.getFeature("write_through");
	Feature duralWriteFeature = storageFeatureProvider.getFeature("dural_write");

	// so the available features under `provider` are: (assume scopes are separated by '.')
	// - 'cache.write_through'
	// - 'storage.dural_write'


	The feature provider could be passed to `DistributedLogNamespaceBuilder` when building the namespace,
	thereby it would be used for controlling the features exposed under `DistributedLogNamespace`.

	::

	FeatureProvider rootProvider = ...;
	FeatureProvider dlFeatureProvider = rootProvider.scope("dlog");
	DistributedLogNamespace namespace = DistributedLogNamespaceBuilder.newBuilder()
	.uri(uri)
	.conf(conf)
	.featureProvider(dlFeatureProvider)
	.build();


	The feature provider is loaded by reflection on distributedlog write proxy server. You could specify
	the feature provider class name as below. Otherwise it would use `DefaultFeatureProvider`, which disables
	all the features by default.

	::

	featureProviderClass=com.twitter.distributedlog.feature.DynamicConfigurationFeatureProvider



	Configuration Based Feature Provider
	____________________________________

	Beside `DefaultFeatureProvider`, distributedlog also provides a file-based feature provider - it loads
	the features from properties files.

	All the features and their availabilities are configured in properties file format. For example,

	::

	cache.write_through=100
	storage.dural_write=0


	You could configure `featureProviderClass` in distributedlog configuration file by setting it to
	`com.twitter.distributedlog.feature.DynamicConfigurationFeatureProvider` to enable file-based feature
	provider. The feature provider will load the features from two files, one is base config file configured
	by `fileFeatureProviderBaseConfigPath`, while the other one is overlay config file configured by
	`fileFeatureProviderOverlayConfigPath`. Current implementation doesn't differentiate these two files
	too much other than the `overlay` config will override the settings in `base` config. It is recommended
	to have a base config file for storing the default availability values for your system and dynamically
	adjust the availability values in overlay config file.

	::

	featureProviderClass=com.twitter.distributedlog.feature.DynamicConfigurationFeatureProvider
	fileFeatureProviderBaseConfigPath=/path/to/base/config
	fileFeatureProviderOverlayConfigPath=/path/to/overlay/config
	// how frequent we reload the config files
	dynamicConfigReloadIntervalSec=60


	Available Features
	__________________

	Check the :doc:`../references/features` reference page for the features exposed by DistributedLog.

	`dlog`
	~~~~~~

	A CLI is provided for inspecting DistributedLog streams and metadata.

	.. code:: bash

	dlog
	JMX enabled by default
	Usage: dlog <command>
	where command is one of:
	local Run distributedlog sandbox
	example Run distributedlog example
	tool Run distributedlog tool
	proxy_tool Run distributedlog proxy tool to interact with proxies
	balancer Run distributedlog balancer
	admin Run distributedlog admin tool
	help This help message

	or command is the full name of a class with a defined main() method.

	Environment variables:
	DLOG_LOG_CONF Log4j configuration file (default $HOME/src/distributedlog/distributedlog-service/conf/log4j.properties)
	DLOG_EXTRA_OPTS Extra options to be passed to the jvm
	DLOG_EXTRA_CLASSPATH Add extra paths to the dlog classpath

	These variable can also be set in conf/dlogenv.sh

	Create a stream
	_______________

	To create a stream:

	.. code:: bash

	dlog tool create -u <DL URI> -r <STREAM PREFIX> -e <STREAM EXPRESSION>


	List the streams
	________________

	To list all the streams under a given DistributedLog namespace:

	.. code:: bash

	dlog tool list -u <DL URI>

	Show stream's information
	_________________________

	To view the metadata associated with a stream:

	.. code:: bash

	dlog tool show -u <DL URI> -s <STREAM NAME>


	Dump a stream
	_____________

	To dump the items inside a stream:

	.. code:: bash

	dlog tool dump -u <DL URI> -s <STREAM NAME> -o <START TXN ID> -l <NUM RECORDS>

	Delete a stream
	_______________

	To delete a stream, run:

	.. code:: bash

	dlog tool delete -u <DL URI> -s <STREAM NAME>


	Truncate a stream
	_________________

	Truncate the streams under a given DistributedLog namespace. You could specify a filter to match the streams that you want to truncate.

	There is a difference between the ``truncate`` and ``delete`` command. When you issue a ``truncate``, the data will be purge without removing the streams. A ``delete`` will delete the stream. You can pass the flag ``-delete`` to the ``truncate`` command to also delete the streams.

	.. code:: bash

	dlog tool truncate -u <DL URI>