# The Low-Latency Pipeline¶

 General information General information on online Online Flowcharts Some figures explaining the various steps Directory structure Infrastructure of the involved directory Configuration file List of parameters in the configuration file Online Running How to run Figures of merit Page overview

## General information¶

The online is structured in the following way:

• perform a zero-lag analysis on available frames which are located in the RAM memory, send to lumin the triggers with a certain significance
• collects zero-lag triggers, produce results page, produce CED for significant triggers
• performing background analysis with a greater latency

Differing from the rest of the infrastructure, the online scripts are python based. There is a general configuration file cWB_conf.py which contains all the information, then there are various cWB-like configuration files for:

• user_parameters.C: zero lag
• user_parameters_bkg.C: background
• user_parameters_pe.C: parameter estimation

N.B. the name of these files are put in cWB_conf.py. Even if it should be safe to change them, we do not suggest to do.

## Directory structure¶

In the configuration file is specified the general directory in which the analysis is performed. Then there is a list of sub-directories where the various part of the analysis is distributed.
Even if in principle the path of these directories are specified in the configuration file, we suggest not to change this setup.

Below we show the directory structure for the default value. Each time we use the cwb_online command two directories are created:

• Main directory (with the name chosen in the configuration file, see Configuration file

• WWW directory (WWW/LSC/online on ATLAS or public_html/online on CIT)

## Online Configuration file¶

The general configuration file should be called cWB_conf.py and put in the RUN_cWB directory. It contains a list of the following parameters:

General setup

Where to find frames, involved detectors, working directory

• title: A title for the web pages
• label: working name of the directory
• online_dir: path of working dir
• frames_dir: directory of the frames saved in RAM, used for zero lag
• bkg_dir: directory for frames saved permanently, used for background analysis
• bkg_fnames: pre-fix of file names, used for background
• ifos: involved interferometers
• channelname: Channel name in the frames containing h(t)
• debug: level of debugging

Zero lag setup

• job_offset: scratch for job selection
• job_timeout:
• min_seg_duration: minimum segment duration for zero lag
• seg_duration: typical segment duration for zero lag
• look_back: search for frames at time before the official start
• sleep: pause during the creation of segments
• max_jobs:
• science_segment_offset=[]/moving_step: possibility to perform more analyses shifted of the values specified here.
• run_start: starting time, if set to -1 or not defined: it is the local time
• run_stop:: stop time, if set to -1 or not defined: it is never ending
• cwb_par: list of parameters written in C++ code that are common for all cWB user_parameters files (zero lag, bkg)

Background setup

• log_path: path where to put log files of condor submission (depend on cluster)
• bkg_delay: time to wait from last available segment to start job submission
• bkg_nlags: number of lags (not accounting superlags)
• bkg_split: number of jobs in which splitting the total nlags
• bkg_job_duration: typical and maximum job lenght
• bkg_job_duration: minimum job lenght
• bkg_njobs: number of super-lags
• bkg_framesize: frame length
• accounting_group: tag for condor submission
• condor_requirements: requirements for condor machine selection

Plugins

• qveto_plugin: Path of plugin used for Qveto definition. The Plugin is copied inside the run_dir/config directory. After the creation of the all dir the code asks to compile it.
• pe_plugin: Path of plugin used for Parameter estimation. The Plugin is copied inside the run_dir/config directory. After the creation of the all dir the code asks to compile it.

Directories

These directories follows the general structure, we suggest not to modify them

• run_dir: contains zero lag script
• jobs_dir: contains zero lag job
• seg_dir: contains segments list
• segs1: files containing total segment list
• zerolag_par: cWB user_parameters for zero lag
• bkg_par: cWB user_parameters for background
• pe_par: cWB user_parameters for Parameter estimation
• bkg_run_dir: contains background script
• bkg_postprod_dir: contains figure of merit script for background
• bkg_job_dir: contains background jobs
• bkg_segments_dir: contains background segments list, i.e. the following files:
• bkg_considered_segments_file
• bkg_processed_segments_file
• bkg_running_segments_file
• bkg_missing_segments_file
• bkg_run_segments_file
• bkg_job_segments_file
• web_dir: local directory where to put web pages (depend on cluster)

Data quality

If Data Quality are provided

• DQ_Channel=[]: channel inserted in the frames to value if the data are good or not
• DQ_channel_samples=[]: lenght of frames
• bitmask=[]: information contained in the frames for good times to be analyzed

E-mails

List of e-mails to advertize in case of triggers

• emails=[“”,””,..]
• phone_mail: mail to send alert on phone.

Library information

These information are written in the cWB user_parameters files

• version: analysis version
• version_wat: library wat version
• search: cWB search type
• optim: user of SRA or MRA

gracedb

For significative triggers, the information are send to gracedb.

• gracedb_group: destination (Burst/Test)
• gracedb_analysys: cWB
• gracedb_search: Allsky

Injections

If injections have been produced, these parameters set where to find information and where to put working directory

• inj_name=[]: type of Hardware injection to flag (BURST, CBC, …)

Threshold

Post-production thresholds:

• id_rho: choice of rho[0] or rho[1] for event significance
• th_rho_off/th_far_off: rho/far threshold for considering offline significant
• th_rho_lum: rho threshold for considering to send to gracedb
• id_cc: choice of netcc[0] or netcc[1] for event selection
• th_cc: netcc threshold
• th_qveto: threshold on qveto
• Cuts_file: .hh file which contains the list of pp classes
• Cuts_list=[]: list of classes for the classification of events
• Cuts_name=[]: legends for the list of class to be eventually reported in the web page (not necessary)

## Flowcharts¶

General Analysis

The pipeline considers from the coincidence time from the various detector and then on this it applies the zero lag and the background analyses. If the False Alarm Rate is lower than a threshold set in Configuration file, then the trigger informations are sent to GraceDB. FAR is estimated using the Background analysis as a reference.

Detector coincidences

Detectors have their proper Data Quality (DQ) times where the analysis can be performed.<br> The analysis consider the coincidence between the various detectors and consider only the segments which has lenght greater than the minimum value in Configuration file

Various instances

Data arrived in chuncks of usual 4s (frame update). As soon as the data arrives, the pipeline check for coincidences. When coincidence has enough coincidence time (Minimum time in Configuration file), the first job is performed. Then, the pipeline waits for a minor time of coincidence data (Moving time in Configuration file) and perform a job with an overlap with the previous one.

The Summary jobs are updated once the last job is finished and a new small time has been analyzed.

In this way the same trigger can be found more times in different job. In case the same trigger is found multiple time, the one with lower FAR is reported in the Summary.

## How to run¶

There is a script that create all the working directories and the related path for the web pages.

Create Working Directory

First of all create a configuration file for online (config_file in this example). If you do not have an example you can copy the standard one using the command:

cwb_online file config_file


Then change appropriately the configuration file according to the desire of the analysis. Remind to change the working dir.

Then use the command:

cwb_online create config_file


The dir RUN_cWB/config should contain all the files used as configuration:

• user_parameters.C: cWB file for zero lag
• user_parameters_bkg.C: cWB file for background
• user_parameters_bkg_split.C: cWB file for application of superlags in the background
• user_pparameters.C: cWB file for standard postproduction web pages

The automatic procedure put the same values parameters in all cWB user_parameters files, except for the obvious differences. Typical expected differences are:

• diff user_parameters.C user_parameters_bkg.C

13,14c13,17
<   lagSize = 1;
<   lagOff = 0;
---
>   lagSize    = 201;   // number of lags (simulation=1)
>   lagStep    = 1.;    // time interval between lags [sec]
>   lagOff     = 0;     // first lag id (lagOff=0 - include zero lag )
>   lagMax     = 0;   // 0/>0 -  standard/extended lags
>   lagFile    = NULL;  // lag file list
17c20
<   segLen = 60;
---
>   segLen = 600;


Note: usually background are made on jobs of duration 600 s.

Run zero lag

The script run.py in the RUN_cWB directory makes the zero lag analysis, in particular (names in italic are variable of cWB_conf.py file):

• Check the frames present in the online_dir directory
• Launch cWB analysis each job_timeout, if not already done in jobs_dir.
• Check trigger and send to gracedb the ones with enough significance

If more offsets are requested, the run.py command should be performed in each OFFSET_* directory, where as the same script in RUN_cWB directory merges the triggers coming from the different analyses, eventually checking if the same triggers coming from various analyses, to be considered only once.

All the scripts are launched by crontab in the RUN_cWB directory:

crontab run.crontab


CED are produced automatically in the zero lag analysis.

The collection of figures are made by the script web_pages.py, which produces same figures for various time periods, according to the given value:

• daily: each day
• hour: last hour
• mid: last 12 hours
• day: last day
• week: last week
• run: complete run
• all: all the possible solution

These are launched by another crontab script:

crontab web.crontab


The same scripts launch the corresponding figures for background.

Run time shifts

Background is run on dedicated machines, for instance at CIT the ones satysfing the following requirements:

condor_status -constraint TARGET.online_Burst_cWB

Name               OpSys      Arch   State     Activity LoadAv Mem   ActvtyTime

slot4@node1.cluste LINUX      X86_64 Claimed   Busy      1.080 3829  0+01:16:52
slot4@node10.clust LINUX      X86_64 Claimed   Busy      1.120 3829  0+03:58:01
slot4@node2.cluste LINUX      X86_64 Claimed   Busy      1.090 3829  0+02:33:58
slot4@node3.cluste LINUX      X86_64 Claimed   Busy      1.070 3829  0+03:43:11
slot4@node4.cluste LINUX      X86_64 Claimed   Busy      1.100 3829  0+03:28:32
slot4@node5.cluste LINUX      X86_64 Claimed   Busy      0.140 3829  0+00:31:59
slot4@node6.cluste LINUX      X86_64 Claimed   Busy      1.140 3829  0+02:22:15
slot4@node7.cluste LINUX      X86_64 Claimed   Busy      1.050 3829  0+04:26:10
slot4@node8.cluste LINUX      X86_64 Claimed   Busy      1.050 3829  0+01:41:54
slot4@node9.cluste LINUX      X86_64 Claimed   Busy      1.100 3829  0+03:05:21
Total Owner Claimed Unclaimed Matched Preempting Backfill

X86_64/LINUX    10     0      10         0       0          0        0

Total    10     0      10         0       0          0        0

Background consider the files inside the SEGMENT directory to select the time period to be analyzed. Then it submits jobs according to available segments.

To launch background enter in TIME_SHIFTS directory and use the command:

./run_all.sh

Background figures are already produced from the same postproduction script of online, however it is possible to procued manually with the following commands:

Figures are already produced during the web pages production, in any case to update, enter in the directory TIME_SHIFTS/POSTPRODUCTION and launch:

./restart_last_N.sh .


## Figures of merit¶

Results from online are collected in a wiki page with a simple structure:

• Status: which contains:
• uptime: running time of the whole run
• hostname: running machine of zero lag
• Page generation: GPS time of the page creation (referred to the calendar)
• Tab menu with pages reporting results collected in following periods
• last hour
• last 12 hours
• last day
• last week
• the whole run
• Calendar (collection of pages for each day)

The pages referred in Tab menu shows all the same structure according to different situations:

• No jobs during the selected period: a table reporting the start and stop of the considered period.
• No triggers during the selected period: a table reporting run statistic (running time, analized segments, …)
• Triggers: same table as previous case, information and FOM on the triggers.

We show an example for each table and figure.

Run statistic table

• Number of completed jobs:
• 2815: number of zero lag jobs, with link of the time list
• Analized segments: time periods after data quality selection
• More details: A web page reporting histograms and table of running times, and time delay between arrival data and starting job.
• Job run time: Maximum, minimum and average running time for the zero lag jobs
• Jobs completed within 10 minutes: percentage of zero lags jobs completed within 10 minutes.

Trigger table