1 Abstract¶

The First-Look Analysis and Feedback Functionality (FAFF) working group, identified the need for a service that is capable of executing user-defined operations/computation following user-defined rules. This service has been referred to as Catcher. In essence, the Catcher is not that different from the Watcher or other stream processing off-the-shelf technologies like Kapacitor. This technote explores the idea of using InfluxDB OSS 2.x new task engine as an alternative implementation for the Catcher.

2 Introduction¶

One of the finding of the First-Look Analysis and Feedback Functionality (FAFF) working group SITCOMTN-025 [1], was the need for a service to …

handle the low-level coordiation of identifying when a specific condition is met, then launching and monitoring an analysis process.

The original proposal from the FAFF working group was to develop a Commandable SAL Component (CSC), similar to the Watcher, that would provide be capable of monitoring data from the system and trigger certain actions (e.g. perform some computation and/or generate reports) based on some pre-defined conditions.

Following additional discussions, we have decided to experiment with an alternative approach, replacing the CSC with an off-the-shelf solution provided by provided by Sasquatch (see SQR-068 []).

As discussed in SQR-068 [], Sasquatch is based on InfluxDB OSS 2.x, a time series database. The new version of InfluxDB brings a number of important updates, amongst which two are relevant for stream processing; Flux and InfluxDB task.

Flux is a new functional data scripting language that is designed to process data streams. It contains extensive libraries that allows users to express complex operations, compatible with what can be done with programming languanges. Furthermore, InfluxDB task provides a way to schedule Flux scripts.

Each Catcher “rule” or “alarm” would be comprised of an InfluxDB task to execute a user defined Flux script periodically. The Flux script performs a set of actions/calculations based on EFD data, additional processes/tasks can be triggered depending on the results of these calculations and results can be stored in the EFD for future reference.

Below we describe how some of the Catcher functionality would be implemented.

2.1 Alarms¶

One of the key features expected from the Catcher was the ability to issue alarms that can be displayed in LOVE. One option to support this feature would be to create a custom LOVE producer that can be interacted with using a REST API. The alert information would be persisted in Sasquatch by the Flux script, which would then post a message to the LOVE producer of the new alert. The producer would then read the alert information from the EFD and display it.

2.2 Process Image Data¶

Another important feature of the Catcher was the capability to process camera image data using the OCPS. The OCPS is a CSC that is capable of executing DM-pipeline processes. It is going to be key in allowing remote processing of Main Camera data during commissioning and operations.

Nevertheless, if the Catcher is not a CSC it won’t be able to interact direcly with OCPS to process image data. One alternative would be to interact with the OCPS remote backend direcly, instead of going through the CSC. For that, the OCPS backend implements a REST API that we could leverage to post requests and receive responses back.

2.3 Generate reports¶

The Catcher should also be able to generate reports, with graphs and performing additional lightweight computation. The original idea was to rely on parameterized Jupyter notebooks that would later be converted to webpages and rendered online so users could inspect the results, and use as reference for additional followup.

This could still be achieved using a similar solution or relying on a service like noteburst SQR-065 [2] and Times Square SQR-066 [3] to execute parameterized notebooks using a web request and publishing the results afterwards.

3 Service Architecture¶

The Catcher would be a service built on top of RSP functionalities (some existing and others still under development), with ancillary support from LOVE and OCPS. It will mostly rely on Flux scripts and InfluxDB task to execute operations based on data streams from the EFD bucket. The results can be persisted in the EFD, along with summary information to be displayed by LOVE.

4 Prototype¶

TBD

References

1: [SITCOMTN-025]. Keith Bechtol, Patrick Ingraham (chair), Tim Jenness, Tony Johnson, Simon Krughoff, Robert Lupton, Tiago Ribeiro, and Yousuke Utsumi. First-look analysis and feedback functionality breakout group report. 2022. URL: https://sitcomtn-025.lsst.io/
2: [SQR-065]. Jonathan Sick. Design of noteburst, a programatic jupyterlab notebook execution service for the rubin science platform. 2022. URL: https://sqr-065.lsst.io/
3: [SQR-066]. Adam Thornton. Proposal for separate rsp user lab spawning service. 2022. URL: https://sqr-066.lsst.io/