tree 30104903da08e826b064b06115f864e32b9604b9
parent 45ddc9350e34c609a0a0b0279d0a70f8a4cbc63b
author Tony Sun <tony.sun427@gmail.com> 1599770117 -0700
committer GitHub <noreply@github.com> 1599770117 -0700
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsBcBAABCAAQBQJfWo4FCRBK7hj4Ov3rIwAAdHIIAAdymfdBDSO0g4pW1kE0x2TA
 DIZSPkreRb3YKdxwE6PruGDk6700LZ12Erv+WGOoPj6t/L4AaN1RhlEDUgwd4crj
 QUFpDPFZzLeDEUFEG/rFASEid+CDqlI9MwcB6BwpnXCUx7gwHY+P0/I8vCwzY6NG
 XLpVDwB7sSKIjB8oRR2Bbk5XV+0D1f6cjwFUPPHHEjTCJjR6eit7MfsfSMjXA4ct
 NGVHi4HUURdKgQRQ6jwhwWxMy5q3wAxs1QezuQVn7kjz4g38194NvFPJXYhqepF6
 F4r9Bq5JRhcPSqnnhCqAaxb9JJgquaQ1s/OHTdmwb9Knm3YrkUFO7iAQIkkb8wc=
 =EzWz
 -----END PGP SIGNATURE-----
 

add remonitor code to DOWN message (#3144)

Smoosh monitors the compactor pid to determine when the compaction jobs
finishes, and uses this for its idea of concurrency. However, this isn't
accurate in the case where the compaction job has to re-spawn to catch up on
intervening changes since the same logical compaction job continues with
another pid and smoosh is not aware. In such cases, a smoosh channel with
concurrency one can start arbitrarily many additional database compaction jobs.

To solve this problem, we added a check to see if a compaction PID exists for
a db in `start_compact`. But wee need to add another check because this check
is only for shard that comes off the queue. So the following can still occur:

1. Enqueue a bunch of stuff into channel with concurrency 1
2. Begin highest priority job, Shard1, in channel
3. Compaction finishes, discovers compaction file is behind main file
4. Smoosh-monitored PID for Shard1 exits, a new one starts to finish the job
5. Smoosh receives the 'DOWN' message, begins the next highest priority job,
Shard2
6. Channel concurrency is now 2, not 1

This change adds another check into the 'DOWN' message so that it checks for
that specific shard. If the compaction PID exists then it means a new process
was spawned and we just monitor that one and add it back to the queue. The
length of the queue does not change and therefore we won’t spawn new
compaction jobs.