tree 6fba6f00d9173d5a2c1fd9d27a81dace5823c2fa
parent c84d08146c183e2dcf83a4acbbbbe2da4cbaeec4
author Stephan Erb <serb@apache.org> 1528981368 +0000
committer Stephan Erb <serb@apache.org> 1528981476 +0200

Speedup regular Thermos observer checkpoint refresh

Profiling indicates that a significant part of the refresh time os spend in `os.path.realpath`.
This was introduced in https://reviews.apache.org/r/35580/ to properly handle the `latest`
symlink in the Mesos folder layout.

This patch takes a slightly different approach to solve this problem based on `os.path.islink`.
The latter is faster as it just needs to look at a single folder rather than an entire path.

Testing Done:
I have tested this build on a node with 55 running tasks and 2004 finished ones.

Before this patch:

    D0320 22:20:44.887248 25771 task_observer.py:142] TaskObserver: finished checkpoint refresh in 0.92s
    D0320 22:20:50.746316 25771 task_observer.py:142] TaskObserver: finished checkpoint refresh in 0.93s
    D0320 22:20:56.590157 25771 task_observer.py:142] TaskObserver: finished checkpoint refresh in 0.89s

With this patch:

    D0320 22:18:53.545236 16250 task_observer.py:142] TaskObserver: finished checkpoint refresh in 0.48s
    D0320 22:18:59.031919 16250 task_observer.py:142] TaskObserver: finished checkpoint refresh in 0.49s
    D0320 22:19:04.512358 16250 task_observer.py:142] TaskObserver: finished checkpoint refresh in 0.48s

Reviewed at https://reviews.apache.org/r/66139/