PromCon 2017Talk abstract
The Uninstrumentable; Getting Apache Spark and Prometheus to Play Nicely
Speaker 1: Dan Rathbone
Speaker 2: Joe Stringer
Instrumenting your code with Prometheus is simple and easy. Or so we thought until we tried to instrument our Python application running under Apache Spark… The distributed nature of Spark presents some interesting challenges when it comes to instrumenting your code effectively, for example a lack of global state, transient processes and no control over the execution profile.
We’ll talk about our myriad failed attempts at instrumenting under Spark and our journey to finally getting something working effectively, without DOSing Prometheus with millions of time series! :)
Back to schedule