The complexity of doing large scale data processing is not only about the volume of data. Often this large volume of data is sourced from (and needs to move through) different data stores and services. There are mapping, filtering, join and aggregation steps along the way which span across these. Moreover in a large organization there are several distinct teams managing one or more of these data stores and services. All of this frequently leads to each team modeling their subset of the processing as a sub-pipeline and coming up with a bunch of APIs (along with SLOs/SLAs) to exchange data between these different sub pipelines to form the overall processing pipeline. Monitoring such a processing pipeline can be a challenge not just with managing large number of complex inter-linked dependencies but also the fact that the ownership model is distributed across many teams across divisions and regions. In order to monitor such a pipeline you should be able to answer a few high level questions such as:

• How do you visualize and query the status of the entire processing pipeline in its entirety across the organization without getting bogged down in a lot of granular details?
• How do you know when something is wrong or is about to go wrong?
• How do you know the impact of failure in one part of the pipeline on another part?
• How do you monitor if there are delays in one part of the processing pipeline and the effect that delay has for the timeliness SLO of the entire pipeline?

A key strategy that can be used here is to model the entire processing pipeline as a graph with nodes representing processing steps and edges representing dependencies between steps. In this session, targeted for engineers, architects as well as managers, Atin will take you through an approach to model and interpret the processing pipeline as a graph and track key metrics across the graph that articulate the current status, model SLOs on top of these metrics to enable meaningful alerting signals as well as drive key analytical insights into how the pipeline is doing over a period of time.