2018-08-09 2018-08-10 2 00:00 00:15 8:15 01:00 Elsewhere talk True 11:00 00:15 Elsewhere talk True 12:15 00:15 Elsewhere talk True 13:30 01:00 Elsewhere talk True 15:30 00:15 Elsewhere talk True 16:45 00:15 Elsewhere talk True 18:00 00:30 Elsewhere talk True 18:30 2:00 Elsewhere talk True 9:15 00:15 Main room talk True Richard Hartmann (SpaceNet) Richard opens the conference and gives an overview of the state of Prometheus. Talk information, slides, and video. Richard Hartmann (SpaceNet) 9:30 00:30 Main room talk True Sneha Inguva (DigitalOcean) The pillars of observability have long been accepted as key components of any microservice-in-production. But what about those new products - those new features - that have yet to be released? Properly instrumenting and leveraging metrics at this stage is perhaps even more crucial; when a product is yet to be released, identifying and addressing early bugs is critical.
Within this talk, I will discuss how we have leveraged Prometheus to properly instrument and test features within the software-defined networking pillar at Digital Ocean. I will highlight instrumentation, key visualizations, as well as takeaways from our experience. Perhaps even more importantly, I will touch upon areas that we can certainly improve upon. Listeners - especially those on product teams - will be able to utilize these learnings and hopefully apply them before their releases as well. Talk information, slides, and video. Sneha Inguva (DigitalOcean) 10:00 00:30 Main room talk True Stuart Nelson (SoundCloud) Alerting is a key concept to remediating issues that arise in the systems we monitor. In order to alert effectively, it's helpful to understand the concepts available in Alertmanager and how they are actually executed within the code. In this talk, I'll give a brief, high-level description of the wondrous journey an alert takes, from its triggering in Prometheus to the email in your mailbox. Then, I'll break down in-depth how alerts are ingested, grouped, and processed, including high availability mode. At the end of this exciting trip you'll understand how your configuration affects Alertmanager, what an alerting pipeline is, and why you sometimes get two emails for the same alert. Talk information, slides, and video. Stuart Nelson (SoundCloud) 10:30 00:30 Main room talk True Ivan Babrou (Cloudflare) While there are plenty of readily available metrics for monitoring the Linux kernel, many gems remain hidden. With the help of recent developments in eBPF, it is now possible to run safe programs in the kernel to collect arbitrary information with little to no overhead. A few examples include:
* Disk latency and IO size histograms
* Run queue (scheduler) latency
* Page cache efficiency
* Directory cache efficiency
* LLC (aka L3 cache) efficiency
* Kernel timer counters
* System-wide TCP retransmits
Practically any event from "perf list" output and any kernel function can be traced, analyzed, and turned into a metric with almost arbitrary labels attached to it.
If you are already familiar with BCC tools, you may think of ebpf\_exporter as BCC tools turned into Prometheus metrics.
In this talk we'll go over eBPF basics, how to write programs and get insights into a running system.
https://github.com/cloudflare/ebpf\_exporter https://github.com/cloudflare/ebpf\_exporter Talk information, slides, and video. Ivan Babrou (Cloudflare) 11:15 00:30 Main room talk True Tom Wilkie (Grafana Labs) Prometheus offers powerful open source monitoring and alerting - but that comes with higher degrees of freedom, making pre-configured monitoring "packages" hard to build. Simultaneously, it's becoming accepted wisdom that the developers of a given software package are best placed to operate said software, or at least construct the basic monitoring configuration.
In this talk we present a technique for using Jsonnet (a configuration language from Google) for packaging and deploying "Monitoring Mixins" - extensible and customisable combinations of dashboards, alert definitions and exporters. This technique allows developers of open source projects to publish best-practice monitoring configurations alongside their code, and for users to consume it, customise it and stay up to date. We will present example Mixins for Kubernetes and other services such as Consul, Vault, and Cassandra. Talk information, slides, and video. Tom Wilkie (Grafana Labs) 11:45 00:30 Main room talk True Michael Hausenblas Frederic Branczyk (Red Hat) Autoscaling in Kubernetes used to be an inconsistent concept, however using the new metrics APIs defined by Kubernetes SIG Instrumentation the monitoring system of choice can be used. As it turns out, Prometheus is a popular system being used alongside Kubernetes, and the community has already developed a custom-metrics-api adapter to be used for Prometheus. This means, we can now perform autoscaling on the cluster and application level with metrics collected by Prometheus.
While for some use cases single values are enough, for others more sophisticated historic metrics are necessary. In the context of the SIG Autoscaling, we're working on a Vertical Pod Autoscaler (VPA), allowing for vertical autoscaling of pods (that is, adapting resource limits and requests) based on metrics from Prometheus (see https://github.com/kubernetes/community/blob/master/contributors/design-proposals/autoscaling/vertical-pod-autoscaler.md).
Frederic and Michael will review the history of metrics in Kubernetes, discuss the current state of metrics and autoscaling on Kubernetes using Prometheus with a focus on VPAs as well as show it in action. Talk information, slides, and video. Michael Hausenblas Frederic Branczyk (Red Hat) 12:30 00:30 Main room talk True Julien Pivotto (Inuits) Relabeling is a powerful feature of Prometheus but it is also quite specific to the project. This talk will explain and demonstrate relabeling, with some real use cases. After the presentation, you will understand relabeling and stop to copy/paste relabeling rules you don't understand. Talk information, slides, and video. Julien Pivotto (Inuits) 13:00 00:30 Main room talk True Cody Boggs (FreshTracks.io) Series cardinality can be a large foot-gun that can take out a Prometheus instance quickly. It can have similar impacts to downstream systems when using remote write. Finding the offending series and recovering from such an event can cause a large downtime for your Prometheus installation. In this talk I will show an implementation of a sidecar to Prometheus that automatically detects and squelches series whose cardinality is growing quickly. Using a combination of recording rules, dynamic metric relabel configs, and alerting, we can detect, throttle, and alert on any explosion in progress. Talk information, slides, and video. Cody Boggs (FreshTracks.io) 14:30 00:30 Main room talk True Bartłomiej Płotka (Improbable) Fabian Reinartz (Google) The Prometheus Monitoring system has been thriving for several years. Along with its powerful data model, operational simplicity and reliability have been a key factor in its success. However, some questions were still largely unaddressed to this day. How can we store historical data at the order of petabytes in a reliable and cost-efficient way? Can we do so without sacrificing responsive query times? And what about a global view of all our metrics and transparent handling of HA setups?
Thanos takes Prometheus' strong foundations and extends it into a clustered, yet coordination free, globally scalable metric system. It retains Prometheus's simple operational model and even simplifies deployments further. Under the hood, Thanos uses highly cost-efficient object storage that's available in virtually all environments today. By building directly on top of the storage format introduced with Prometheus 2.0, Thanos achieves near real-time responsiveness even for cold queries against historical data. All while having virtually no cost overhead beyond that of the underlying object storage.
We will show the theoretical concepts behind Thanos and demonstrate how it seamlessly integrates into existing Prometheus setups. Talk information, slides, and video. Bartłomiej Płotka (Improbable) Fabian Reinartz (Google) 15:00 00:30 Main room talk True Paul Dix (InfluxData) The new open source Flux query engine for InfluxDB is designed to be decoupled from storage. This makes it possible to query many different data sources, including Prometheus, with the new engine. The engine also decouples the representation of the query language from the execution representation. This makes it possible to add support for additional languages, like PromQL.
This talk will show how we've added support to the Flux engine to query multiple Prometheus servers at the same time, combining their results. This makes it trivial to query across a federated setup for scaling out the query. We'll also show how to query across two Prometheus servers configured for high availability, while guaranteeing a consistent query result.
We'll close out the talk by looking at how PromQL support is being added to the new engine. This will enable PromQL queries across a distributed system where the individual Prometheus and storage servers maintain their shared-nothing non-clustered failure modes. Talk information, slides, and video. Paul Dix (InfluxData) 15:45 01:00 Main room talk True Julius Volz (Prometheus) Fabian Reinartz (Google) Nikunj Aggarwal (Uber) Paul Dix (InfluxData) Tom Wilkie (Grafana Labs) In this panel, we will explore some of the emerging and competing approaches to Prometheus scalable long-term storage. Talk information, slides, and video. Julius Volz (Prometheus) Fabian Reinartz (Google) Nikunj Aggarwal (Uber) Paul Dix (InfluxData) Tom Wilkie (Grafana Labs) 17:00 01:00 Main room talk True Lightning talks are 5 minutes each.
All LTs submitted via the CfP are scheduled here. For the remaining time,
we will have a write-in poster for people interested in giving a LT. Talk information, slides, and video. 8:30 01:00 Elsewhere talk True 11:00 00:15 Elsewhere talk True 12:15 00:15 Elsewhere talk True 13:30 01:00 Elsewhere talk True 15:30 00:15 Elsewhere talk True 16:45 00:15 Elsewhere talk True 9:30 00:30 Main room talk True Robin van Zijll (ING N.V.) By definition, SREs are responsible for the reliability of sites, but what if they don't own any sites themselves? Within ING, the largest bank of the Netherlands, BizDevOps teams are autonomous and responsible for building and running their services. In theory, that could make the existence of SRE obsolete, right? How can you improve availability for end customers in an environment of engineers with full service ownership? How to convince without the power of intervention? How to improve without being blameful?
We'll explain how we, a team of 8 SREs among 1700 DevOps engineers, try to improve stability by focussing on software engineering. We created the Reliability Toolkit to help BizDevOps teams with their reliability challenges in the fields of white box monitoring and alerting while minimizing toil.
This talk will explain:
- Our SRE team purpose and why we think our approach with heavy focus on software engineering works for our organization
- The concept of the Reliability Toolkit and introduction of its components and their setup (Prometheus, Alertmanager, Grafana, NGINX Log Aggregator, SMS and ChatOps functionalities)
- How we provision Reliability Toolkit
- How we convince, onboard and educate BizDevOps teams to use the Reliability Toolkit
During this talk we will demo:
- A Kafka to Prometheus consumer (and why this is not what we want)
- Prometheus Model Builder, generate expectation models out of any Prometheus metric
- A collection of templated Grafana dashboards to give every team a kickstart Talk information, slides, and video. Robin van Zijll (ING N.V.) 10:00 00:30 Main room talk True David Kaltschmidt (Grafana Labs) Grafana is the de-facto dashboarding solution for Prometheus. Now imagine you received a page. Grafana is often the starting point for your incident response. You look at a time series panel, form a hypothesis, and would like to dive deeper. We've built a whole new section that allows you to do just that by iterating quickly through Prometheus queries while leaving your dashboards intact. I'll show-case our new Explore UI and how it can fit into your workflows for both troubleshooting and data exploration. Talk information, slides, and video. David Kaltschmidt (Grafana Labs) 10:30 00:30 Main room talk True Darragh Grealish (56k.Cloud GmbH) We show how PromQL and Alertmanager can be used to detect abnormalities in LAN and WiFi networks, using various Prometheus exporters to take data from WLAN controllers and SNMP. We use Unifi, Grafana and of course Prometheus + Alertmanager, along with a few exporters. Talk information, slides, and video. Darragh Grealish (56k.Cloud GmbH) 11:15 00:30 Main room talk True Richard Hartmann (SpaceNet) Prometheus has established a de-facto standard for cloud-native metric exposition. OpenMetrics aims to achieve true interoperability, and to bridge the gap towards other types of monitoring. Talk information, slides, and video. Richard Hartmann (SpaceNet) 11:45 00:30 Main room talk True Jorge Salamero Sanz (Sysdig) Users have been looking for a better understanding of how Prometheus monitoring and other commercial monitoring tools compare and contrast when it comes to Docker and Kubernetes monitoring. Are they enemies? Lovers? Twins separated at birth? Let's go there. This talk will discuss the Prometheus ecosystem from a vendor perspective. Talk information, slides, and video. Jorge Salamero Sanz (Sysdig) 12:30 00:30 Main room talk True Jonas Große Sundrup Prometheus is capable of and used for monitoring large infrastructures, from corporate networks to world-spanning CDNs. But while performing excellent on setups of large scales, it is also extremely well suited for small setups due to its simplicity and modularity both in operation as well in usage.
To further simplify leveraging the power of Prometheus for monitoning, we developed a cooperative, centrally managed multi-tenant-capable Prometheus system that minimizes the entry barrier for participants, introducing Prometheus to small-scale monitoring including out-of-the-box capabilities for blackbox montioring.
The multi-tenancy layer in this case serves two distinct purposes: First, it ensures that every user can only access their own metrics when querying. Second, it ensures that every user can only create alerts or alert silences for their own set of metrics.
In the process of doing so, we will investigate how the Prometheus frontend handles queries and how alerts and silences are submitted and modified accordingly, to implement multi-tenancy in Prometheus.
The entire setup is usable with out-of-the-box standard Prometheus components, no patches necessary. Talk information, slides, and video. Jonas Große Sundrup 13:00 00:30 Main room talk True Marcus Barczak (Fastly) Over the past 6 months at Fastly we've migrated away from our legacy monitoring systems and have deployed Prometheus as our primary system for infrastructure and application monitoring.
The Prometheus approach posed some unique challenges over traditional monitoring systems, whilst at the same time enabling us to easily scale our monitoring infrastructure alongside our global network growth.
It hasn't been completely smooth sailing and deploying Prometheus across a globe spanning network serving over 10% of the world's internet traffic has raised its fair share of challenges both technical and cultural.
In this presentation you will learn how we addressed these issues in ways that deviate slightly from conventional wisdom, the mistakes we made along the way, and how the new system has been received by our teams. We hope that our experiences can help you succeed in deploying Prometheus within your organization. Talk information, slides, and video. Marcus Barczak (Fastly) 14:30 00:30 Main room talk True Krasi Georgiev (Red Hat) Harsh Agarwal (IIT Hyderabad) We will show you how we are trying to make Prometheus more stable by running automated benchmarking for risky PRs and before every release.
In other words, let's try to break it in our tests so it doesn't break in your production.
We will cover why we decided to use Prow and how it is integrated with GitHub.
We will also cover the current progress, the project roadmap and of course do a short demo. Prow Talk information, slides, and video. Krasi Georgiev (Red Hat) Harsh Agarwal (IIT Hyderabad) 15:00 00:30 Main room talk True Brian Brazil (Robust Perception) Prometheus client libraries are notably different from most other options in the space. In order to get the best insights into your applications it helps to know how they are designed, and why they are designed that way. This talk will look at how client libraries are structured, how that makes them easy to use, some tips for instrumentation, and why you should use them even if you aren't using Prometheus. Talk information, slides, and video. Brian Brazil (Robust Perception) 15:45 01:00 Main room talk True The Prometheus maintainers A panel of Prometheus maintainers will answer questions about Prometheus, alternating between audience questions and presubmitted ones. Submit your questions in advance here!. Submit your questions in advance here! Talk information, slides, and video. 17:00 01:00 Main room talk True Lightning talks are 5 minutes each.
We will have a write-in poster for people interested in giving a LT. Talk information, slides, and video. 18:00 0:15 Main room talk True Richard Hartmann (SpaceNet) Richard will close the conference with a few parting words. Talk information, slides, and video. Richard Hartmann (SpaceNet)