The tail at scale

Dealing with latency variability is a Hard Problem™ and really not much to do with Ruby or process vs thread parallelism. https://dl.acm.org/doi/pdf/10.1145/2408776.2408794

My pick is Jeff Dean's The Tail At Scale (2013) [1]. If you're interested in performance of web services, it's very, very well-written and describes a counterintuitive (but crucial!) phenomenon: your p99.99 latency can drive the entire user experience because of head-of-line blocking. As microservice architectures have gotten more popular, this has only become more important. It's a fiendishly difficult problem to get around, but Dean proposes a few mechanisms. I learned a lot and think back to this paper often when designing real software. Reading it will probably make you a better engineer. [1] https://dl.acm.org/doi/abs/10.1145/2408776.2408794

And similarly, within a microservice architecture the tail latency is amplified through all its downstream requests or if there is fan-out: Consider a system where each server typically responds in 10ms but with a 99th-percentile latency of one second. If a user request is handled on just one such server, one user request in 100 will be slow (one second). The figure here outlines how service-level latency in this hypothetical scenario is affected by very modest fractions of latency outliers. If a user request must collect responses from 100 such servers in parallel, then 63% of user requests will take more than one second (marked “x” in the figure). "The tail at scale", Jeffrey Dean and Luiz André Barroso: https://dl.acm.org/doi/abs/10.1145/2408776.2408794?download=...