Mesos handles the logs of each Mesos component differently depending on the degree of control Mesos has over the source code of the component.
Roughly, these categories are:
The Mesos Master and Agent use the Google's logging library. For information regarding the command-line options used to configure this library, see the configuration documentation. Google logging options that are not explicitly mentioned there can be configured via environment variables.
Both Master and Agent also expose a /logging/toggle HTTP endpoint which temporarily toggles verbose logging:
The effect is analogous to setting the
GLOG_v environment variable prior to starting the Master/Agent, except the logging level will revert to the original level after the given duration.
For background, see the containerizer documentation.
Mesos does not assume any structured logging for entities running inside containers. Instead, Mesos will store the stdout and stderr of containers into plain files (“stdout” and “stderr”) located inside the sandbox.
In some cases, the default Container logger behavior of Mesos is not ideal:
ContainerLogger module was introduced in Mesos 0.27.0 and aims to address the shortcomings of the default logging behavior for containers. The module can be used to change how Mesos redirects the stdout and stderr of containers.
Mesos comes with two
SandboxContainerLoggerimplements the existing logging behavior as a
ContainerLogger. This is the default behavior.
LogrotateContainerLoggeraddresses the problem of unbounded log file sizes.
LogrotateContainerLogger constrains the total size of a container's stdout and stderr files. The module does this by rotating log files based on the parameters to the module. When a log file reaches its specified maximum size, it is renamed by appending a
.N to the end of the filename, where
N increments each rotation. Older log files are deleted when the specified maximum number of files is reached.
LogrotateContainerLogger can be loaded by specifying the library
liblogrotate_container_logger.so in the
--modules flag when starting the Agent and by setting the
--container_logger Agent flag to
Defaults to 10 MB. Minimum size of 1 (memory) page, usually around 4 KB. </td>
Defaults to <code>CONTAINER_LOGGER_</code>. </td>
Defaults to <code>/usr/local/libexec/mesos</code>. </td>
LogrotateContainerLoggerstarts up companion subprocesses of the
mesos-logrotate-loggerwill pipe the output into the “stdout”/“stderr” files. As the files grow,
logrotateto keep the files strictly under the configured maximum size.
mesos-logrotate-loggerwill finish logging before exiting as well.
LogrotateContainerLogger is designed to be resilient across Agent failover. If the Agent process dies, any instances of
mesos-logrotate-logger will continue to run.
For basics on module writing, see the modules documentation.
There are several caveats to consider when designing a new
ContainerLoggershould be resilient to Agent failover. If the Agent process dies (which includes the
ContainerLoggermodule), logging should continue. This is usually achieved by using subprocesses.
ContainerLoggeris not explicitly notified. Instead, encountering
EOFin the container's stdout/stderr signifies that the container has exited. This provides a stronger guarantee that the
ContainerLoggerhas seen all the logs before exiting itself.
ContainerLoggershould not assume that containers have been launched with any specific
ContainerLogger. The Agent may be restarted with a different
ContainerLogger. This means more than one
ContainerLoggermay be running in a single Agent. However, each Agent will only run a single type of