| Hardware |
| ======== |
| |
| Figure 1 describes the data flow of DistributedLog. Write traffic comes to `Write Proxy` |
| and the data is replicated in `RF` (replication factor) ways to `BookKeeper`. BookKeeper |
| stores the replicated data and keeps the data for a given retention period. The data is |
| read by `Read Proxy` and fanout to readers. |
| |
| In such layered architecture, each layer has its own responsibilities and different resource |
| requirements. It makes the capacity and cost model much clear and users could scale |
| different layers independently. |
| |
| .. figure:: ../images/costmodel.png |
| :align: center |
| |
| Figure 1. DistributedLog Cost Model |
| |
| Metrics |
| ~~~~~~~ |
| |
| There are different metrics measuring the capability of a service instance in each layer |
| (e.g a `write proxy` node, a `bookie` storage node, a `read proxy` node and such). These metrics |
| can be `rps` (requests per second), `bps` (bits per second), `number of streams` that a instance |
| can support, and latency requirements. `bps` is the best and simple factor on measuring the |
| capability of current distributedlog architecture. |
| |
| Write Proxy |
| ~~~~~~~~~~~ |
| |
| Write Proxy (WP) is a stateless serving service that writes and replicates fan-in traffic into BookKeeper. |
| The capability of a write proxy instance is purely dominated by the *OUTBOUND* network bandwidth, |
| which is reflected as incoming `Write Throughput` and `Replication Factor`. |
| |
| Calculating the capacity of Write Proxy (number of instances of write proxies) is pretty straightforward. |
| The formula is listed as below. |
| |
| :: |
| |
| Number of Write Proxies = (Write Throughput) * (Replication Factor) / (Write Proxy Outbound Bandwidth) |
| |
| As it is bandwidth bound, we'd recommend using machines that have high network bandwith (e.g 10Gb NIC). |
| |
| The cost estimation is also straightforward. |
| |
| :: |
| |
| Bandwidth TCO ($/day/MB) = (Write Proxy TCO) / (Write Proxy Outbound Bandwidth) |
| Cost of write proxies = (Write Throughput) * (Replication Factor) / (Bandwidth TCO) |
| |
| CPUs |
| ^^^^ |
| |
| DistributedLog is not CPU bound. You can run an instance with 8 or 12 cores just fine. |
| |
| Memories |
| ^^^^^^^^ |
| |
| There's a fair bit of caching. Consider running with at least 8GB of memory. |
| |
| Disks |
| ^^^^^ |
| |
| This is a stateless process, disk performances are not relevant. |
| |
| Network |
| ^^^^^^^ |
| |
| Depending on your throughput, you might be better off running this with 10Gb NIC. In this scenario, you can easily achieves 350MBps of writes. |
| |
| |
| BookKeeper |
| ~~~~~~~~~~ |
| |
| BookKeeper is the log segment store, which is a stateful service. There are two factors to measure the |
| capability of a Bookie instance: `bandwidth` and `storage`. The bandwidth is majorly dominated by the |
| outbound traffic from write proxy, which is `(Write Throughput) * (Replication Factor)`. The storage is |
| majorly dominated by the traffic and also `Retention Period`. |
| |
| Calculating the capacity of BookKeeper (number of instances of bookies) is a bit more complicated than Write |
| Proxy. The total number of instances is the maximum number of the instances of bookies calculated using |
| `bandwidth` and `storage`. |
| |
| :: |
| |
| Number of bookies based on bandwidth = (Write Throughput) * (Replication Factor) / (Bookie Inbound Bandwidth) |
| Number of bookies based on storage = (Write Throughput) * (Replication Factor) * (Replication Factor) / (Bookie disk space) |
| Number of bookies = maximum((number of bookies based on bandwidth), (number of bookies based on storage)) |
| |
| We should consider both bandwidth and storage when choosing the hardware for bookies. There are several rules to follow: |
| - A bookie should have multiple disks. |
| - The number of disks used as journal disks should have similar I/O bandwidth as its *INBOUND* network bandwidth. For example, if you plan to use a disk for journal which I/O bandwidth is around 100MBps, a 1Gb NIC is a better choice than 10Gb NIC. |
| - The number of disks used as ledger disks should be large enough to hold data if retention period is typical long. |
| |
| The cost estimation is straightforward based on the number of bookies estimated above. |
| |
| :: |
| |
| Cost of bookies = (Number of bookies) * (Bookie TCO) |
| |
| Read Proxy |
| ~~~~~~~~~~ |
| |
| Similar as Write Proxy, Read Proxy is also dominated by *OUTBOUND* bandwidth, which is reflected as incoming `Write Throughput` and `Fanout Factor`. |
| |
| Calculating the capacity of Read Proxy (number of instances of read proxies) is also pretty straightforward. |
| The formula is listed as below. |
| |
| :: |
| |
| Number of Read Proxies = (Write Throughput) * (Fanout Factor) / (Read Proxy Outbound Bandwidth) |
| |
| As it is bandwidth bound, we'd recommend using machines that have high network bandwith (e.g 10Gb NIC). |
| |
| The cost estimation is also straightforward. |
| |
| :: |
| |
| Bandwidth TCO ($/day/MB) = (Read Proxy TCO) / (Read Proxy Outbound Bandwidth) |
| Cost of read proxies = (Write Throughput) * (Fanout Factor) / (Bandwidth TCO) |
| |