.. include:: images.rst

XMDC Engine
-----------

This page provides an overview of the **XMDC** (eXtended MDC) engine, used
to generate large sets of MDC injections distributed either uniformly in
volume or according to a Galactic-disk model, and to produce the
post-production reports (visible volume, hrss50/dist50 vs. IFAR, ...) used
to characterise the detection efficiency and the visible volume of a search.

XMDC builds directly on top of the :doc:`mdc engine <mdc>`: instead of a
fixed waveform/time grid, it draws a **Monte Carlo population** of
injections (waveform, sky position, distance/amplitude) from the
configured distributions -- typically a large population, e.g. 50k-100k
events. This population carries **no injection time**: time only enters
the picture when the population is actually injected into a given
observation period (on-the-fly or via frames, see below), by drawing
events at random (or cycling through them) from the population. Because
of this, the same Monte Carlo population can be reused across several,
different observation periods, or cycled more than once if a period is
long enough to need more injections than the population contains.

See class :cwb_library:`CWB::xmdc`.

Overview
~~~~~~~~~~~~~~~~

An XMDC study is organised around the single
`cwb_xmdc <commands.html#cwb-xmdc>`__ command:

- `1 - Make injections <#make-injections>`__ -- generate the Monte Carlo
  injection population (waveforms, sky/distance distribution, snr/hrss
  thresholds); no injection time is assigned at this stage
- `2 - Inject the signals <#inject-the-signals>`__ -- the preferred,
  frame-less **on-the-fly** injection draws an event from the population
  and adds it sample-by-sample while the data is read, with no frame file
  ever written and no separate step required; the alternative,
  **optional** MDC-frame path instead writes a time-tagged selection of
  the population into MDC frame files, one XMDC job at a time
- `3 - Postproduction reports <#postproduction-reports>`__ -- visible
  volume / hrss50 / dist50 / snr-distance / GD reports

The output of stage 1 is a **XMDC file** (a ROOT file holding the
:cwb_library:`CWB::xmdc::config`, the injection population and, for
**builtin** waveforms, the waveform tree). This file is read directly by
the on-the-fly plugin, or by the frame-generation macro if that path is
used, and by stage 3 to compute the reports once the analysis has
produced the merged ``.S_ifar.root`` file.

XMDC configuration
~~~~~~~~~~~~~~~~~~~~~~

An XMDC run is configured through a :cwb_library:`CWB::xmdc::config`
structure, filled by a **user configuration macro** (analogous to a plugin
configuration macro, see :doc:`plugins`). The main fields are:

- **Population size** -- ``inj_size`` is the number of events in the
  Monte Carlo population to generate (can be large, e.g. 50k-100k); it
  carries no injection time (see `Overview <#overview>`__).
- **Injection thresholds** -- ``inj`` is a list of
  ``{snr_min, fiducial_hrss}`` pairs (one entry per "trial"): each entry
  sets the network SNR threshold used to select an injection and the
  fiducial hrss value used to normalise the visible-volume calculation for
  that trial.
- **Injection waveform buffer** -- ``inj_len``, ``inj_srate``,
  ``inj_resample``, ``inj_flow``, ``inj_fhigh``: length, sample rate and
  frequency band of the generated injection buffers.
- **Sky/distance distribution** -- ``sky_distribution``, one of:

  - ``uv`` -- uniform in volume (distance :sup:`2` weighting)
  - ``poly...`` -- polynomial in distance,
    p\ :sub:`0` + p\ :sub:`1`\ ·dist + p\ :sub:`2`\ ·dist\ :sup:`2`
  - ``mngd`` -- Miyamoto-Nagai Galactic-disk model
  - ``mmgd`` -- McMillan Galactic-disk model

- **Injection type** -- ``umacro_inj_type``, either:

  - ``builtin`` -- waveforms are generated on the fly with the
    :cwb_library:`CWB::mdc` built-in generator (Gaussians, sine-Gaussians,
    WNBs, ...), configured in a ``InitMDC``-style function
    (``umacro_func_name``).
  - ``external`` -- waveforms are read from file (e.g. CCSN catalogs),
    configured in a ``GetWaveforms``-style function.

- **detector_name** / **psd_fname** -- network detectors and the matching
  noise PSD files used to compute the injected SNRs.
- **seed**, **tag** -- random seed and a label used in the output
  directories/file names.

The default configuration can be retrieved and edited starting from
``CWB::xmdc().GetConfig()``, see the full example below.

The ``cwb_xmdc`` command
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Make injections
^^^^^^^^^^^^^^^^^^

.. code-block:: bash

    cwb_xmdc xmdc_user_config_macro.C '--ninj number_of_injections --seed seed --tag tag_name --odir dir_name --rtype all'

runs the user configuration macro (see :cwb_library:`cwb_xmdc_make.C`) and
produces the XMDC file, generating ``--ninj`` injections above the SNR
threshold(s) declared in ``cfg.inj``. Setting ``--rtype none`` skips the
preproduction plots.

Inject the signals
^^^^^^^^^^^^^^^^^^^^^^

Once the XMDC file is available, the injections it contains can be added
to the analysis in either of two ways:

- **On-the-fly (preferred)** -- for each injection, one event is drawn
  from the Monte Carlo population and added sample-by-sample while the
  data is read, with no frame file ever written to disk. Declare
  ``CWB_Plugin_MDC_OTF.C`` as ``plugin`` and
  ``CWB_Plugin_MDC_OTF_Config_XMDC.C`` as ``plugin_config`` in
  ``user_parameters.C`` (see :doc:`plugins`), pointing ``xmdc_fname`` at
  the XMDC file produced above. No further ``cwb_xmdc`` step is required:
  the analysis reads the injection tree directly, job by job. The
  ``xmdc_read`` plugin option selects how the population is consumed:
  ``random`` (default) draws a uniformly random entry for each injection;
  ``sequential`` cycles through the population in order (wrapping around
  if the observation period requires more injections than the population
  contains).

- **Frame-based (optional)** --

  .. code-block:: bash

      cwb_xmdc job_id

  pre-generates real MDC frame files for the given XMDC job (an integer
  job ID), see :cwb_library:`cwb_xmdc_frame.C`, using
  ``CWB_Plugin_MDC_Frames_Config_XMDC.C`` with the standard
  ``CWB_Plugin_MDC_Frames.C`` plugin (see :doc:`plugins`). The resulting
  frames are then analysed like real data. This is only needed when frame
  files are actually required (e.g. to share/replay the same injected
  data set, or to feed a pipeline stage that expects frames).

Postproduction reports
^^^^^^^^^^^^^^^^^^^^^^^^^^

Once the analysis of the XMDC frames has been merged into a
``.S_ifar.root`` file:

.. code-block:: bash

    cwb_xmdc merge_label '--xmdc_fname xmdc_file_name --rtype report_type [options]'

(see :cwb_library:`cwb_xmdc_report.C`) with **report_type** one of:

- ``visible_volume`` -- visible volume vs. IFAR
- ``hrss50`` -- hrss at 50% detection efficiency vs. IFAR
- ``dist50`` -- distance at 50% detection efficiency vs. IFAR
- ``snr_distance`` -- recovered SNR vs. injected distance, at a given IFAR
- ``gd`` -- Galactic-disk view of the injection/detection distribution
- ``eff2vv`` -- convert an efficiency curve into a visible-volume vs. IFAR curve
- ``eff2hrss50`` -- convert an efficiency curve into a hrss50 vs. IFAR curve
- ``eff2dist50`` -- convert an efficiency curve into a dist50 vs. IFAR curve

Two XMDC reports (e.g. from two different pipeline configurations) can be
overlaid with ``--compare_dir``/``--compare_label``, to compare the visible
volume or efficiency gained/lost between the two.

Run ``cwb_xmdc`` with no arguments for the full, up-to-date list of options.

A Full Example
~~~~~~~~~~~~~~~~~~

The following configuration macro (adapted from
`XMDCConfig_BurstLF_StdInj.C <tutorials.html>`__, a **builtin** waveform
set) generates a set of Gaussian, sine-Gaussian and white-noise-burst
injections, uniformly distributed in volume, for a two-detector (L1, H1)
network:

::

        #include "xmdc.hh"

        void XMDCConfig_BurstLF_StdInj() {}    // mandatory default macro function

        CWB::xmdc::config
        XMDCConfig_GetConfig(int inj_size, int seed, TString tag, TString umacro_path_name) {

          CWB::xmdc XMDC;
          CWB::xmdc::config cfg = XMDC.GetConfig();          // start from the default configuration

          cfg.seed               = seed;
          cfg.inj_size           = inj_size;
          cfg.tag                = tag;
          cfg.umacro_path_name   = umacro_path_name;
          cfg.umacro_source_code = CWB::Toolbox::readFile(umacro_path_name);

          cfg.inj_len            = 4;                        // injection buffer length (sec)
          cfg.inj_srate          = 16384;                     // injection sample rate (Hz)
          cfg.inj_resample       = 4096.;                     // injection resample rate (Hz)
          cfg.inj_flow           = 16;                        // low frequency (Hz)
          cfg.inj_fhigh          = 2046;                       // high frequency (Hz)

          cfg.detector_name      = {"L1","H1"};
          cfg.psd_fname          = {"L1_PSD.dat", "H1_PSD.dat"};

          cfg.inj = {                                         // {snr_min, fiducial_hrss}
                       { 5.0, 1.1e-23}, { 5.0, 0.9e-23}, { 5.0, 1.5e-23}, /* ... */
                    };

          cfg.umacro_inj_type    = "builtin";
          cfg.umacro_func_name   = "XMDCConfig_InitMDC";
          cfg.sky_distribution   = "uv";                      // uniform in volume

          return cfg;
        }

        void XMDCConfig_InitMDC() {

          CWB::mdc* MDC;
          CWB_PLUGIN_IMPORT(CWB::mdc*,MDC);

          vector<mdcpar> par;
          par.resize(1);
          par[0].name="duration"; par[0].value=0.0025;
          MDC->AddWaveform(MDC_GA, par);                      // ... add as many waveforms as needed
        }

Run it with:

.. code-block:: bash

    cwb_xmdc XMDCConfig_BurstLF_StdInj.C '--ninj 1000 --seed 1 --tag StdInj'

For an **external**-waveform example (reading numerical-relativity CCSN
waveforms from file instead of the builtin generator), see
`XMDCConfig_BurstLF_CCSN.C <tutorials.html>`__ in the same tutorials
directory.