The pipe is an isolation concept in Satellite. Each pipe has one pipeline to process the telemetry data(metrics/traces/logs). Two pipes are not sharing data.
Satellite --------------------------------------------------------------------- | ------------------------------------------- | | | Pipe | | | ------------------------------------------- | | ------------------------------------------- | | | Pipe | | | ------------------------------------------- | | ------------------------------------------- | | | Pipe | | | ------------------------------------------- | ---------------------------------------------------------------------
Module is the core workers in Satellite. Module is constituted by the specific extension plugins. There are 3 modules in one namespace, which are Gatherer, Processor, and Sender.
Pipe -------------------------------------------------------------------- | ---------- ----------- -------- | | | Gatherer | => | Processor | => | Sender | | | ---------- ----------- -------- | --------------------------------------------------------------------
LifeCycle
Plugin is the minimal components in the module. Satellite has 2 plugin catalogs, which are sharing plugins and normal plugins.
Nowadays, there are 2 kinds of sharing plugins in Satellite, which are server plugins and client plugins. The reason why they are sharing plugins is to reduce the resource cost in connection. Server plugins are sharing with the ReceiverGatherer modules in the different pipes to receive the external requests. And the client plugins is sharing with the Sender modules in the different pipes to connect with external services, such as Kafka and OAP.
Sharing Server Sharing Client -------------------------------------------------------------------- | ------------------ ----------- -------- | | | ReceiverGatherer | => | Processor | => | Sender | | | ------------------ ----------- -------- | -------------------------------------------------------------------- -------------------------------------------------------------------- | ------------------ ----------- -------- | | | ReceiverGatherer | => | Processor | => | Sender | | | ------------------ ----------- -------- | -------------------------------------------------------------------- -------------------------------------------------------------------- | ------------------ ----------- -------- | | | ReceiverGatherer | => | Processor | => | Sender | | | ------------------ ----------- -------- | --------------------------------------------------------------------
There are 7 kinds of normal plugins in Satellite, which are Receiver, Fetcher, Queue, Parser, Filter, Forwarder, and Fallbacker.
Gatherer Processor ------------------------------- ------------------------------------------- | ----------- --------- | | ----------- ----------- | | | Receiver | ==> | Queue | |==>| | Filter | ==> ... ==> | Filter | | | | /Fetcher | | Mem/File | | | ----------- ----------- | | ----------- ---------- | | || || | -------------------------------- | \/ \/ | | --------------------------------------- | | | OutputEventContext | | | --------------------------------------- | ------------------------------------------- || \/ Sender ------------------------------------------ | --- --- | | | B | | D | ----------------- | | | A | | I | |Segment Forwarder| | | | T | | S | | (Fallbacker) | | | | C | | P | ----------------- | | | H | => | A | | ===> Kafka/OAP | | B | | T | => ...... | | | U | | C | | | | F | | H | ----------------- | | | F | | E | | Meter Forwarder| | | | E | | R | | (Fallbacker | | | | R | | | ----------------- | | --- --- | ------------------------------------------ 1. The Fetcher/Receiver plugin would fetch or receive the input data. 2. The Parser plugin would parse the input data to SerializableEvent that is supported to be stored in Queue. 3. The Queue plugin stores the SerializableEvent. However, whether serializing depends on the Queue implements. For example, the serialization is unnecessary when using a Memory Queue. Once an event is pulled by the consumer of Queue, the event will be processed by the filters in Processor. 4. The Filter plugin would process the event to create a new event. Next, the event is passed to the next filter to do the same things until the whole filters are performed. All created events would be stored in the OutputEventContext. However, only the events labeled with RemoteEvent type would be forwarded by Forwarder. 5. After processing, the events in OutputEventContext would be stored in the BatchBuffer. When the timer is triggered or the capacity limit is reached, the events in BatchBuffer would be partitioned by EventType and sent to the different Forwarders, such as Segment Forwarder and Meter Forwarder. 6. The Follower in different Senders would share with the remote client to avoid make duplicate connections and have the same Fallbacker(FallBack strategy) to process data. When all forwarders send success or process success in Fallbacker, the dispatcher would also ack the batch is a success. ============================================================================================