.. include:: images.rst OpenMP Parallelization ----------------------- Since cWB-6.4.8.0, the **Coherence** and **SuperCluster** stages of the 2G pipeline (``cwb2G.cc``) can run multi-threaded via OpenMP, on top of the job-level parallelism already provided by condor/cwb_inet (see `production `__). This page explains what is actually parallelized, when it helps, and what to expect in terms of memory and speed. What is parallelized ~~~~~~~~~~~~~~~~~~~~~~~~ In both stages, the loop that is parallelized runs over the **time-shift lags** of the job (one iteration per lag). Each lag is processed independently on its own per-thread scratch buffers; the only shared mutable state -- the ROOT job-file I/O and the shared sparse-table update -- is serialized in a critical section, so the result is bit-identical to the serial code regardless of the thread count. Nothing else in the pipeline is parallelized this way: the pre-processing stages (data conditioning, whitening, likelihood/reconstruction) and the post-production stage remain single-threaded. When it actually helps ~~~~~~~~~~~~~~~~~~~~~~~~~~ The achievable speedup is bounded by the number of lags processed by the job: - **Background (BKG) jobs** -- typically process many time-shift lags, commonly thousands (e.g. 2399 lags for a 1200 s segment) to accumulate background statistics. These are where OpenMP gives a real benefit, since there are many independent iterations to spread across threads. For simplicity, the zero-lag is normally **not** produced as a separate single-lag job: it is bundled into the same BKG job as just one more lag among the many, so it benefits from the same parallelism as the rest of the background. - **Simulation (SIM) / MDC injection jobs** -- these are deliberately run at a lagged (non-zero-shift) time, rather than at the real zero-lag time, precisely to avoid reusing the same zero-lag data as the actual search result. If such a job only iterates over a handful of lags, there is correspondingly little for OpenMP to parallelize, and little or no speedup should be expected from raising ``OMP_NUM_THREADS``. As always, the relevant number is how many independent lag iterations the specific job processes -- not whether it is labelled "BKG" or "SIM". As a rule of thumb, requesting more threads than there are lags in the job does not help further -- the extra threads simply stay idle. Bottlenecks ~~~~~~~~~~~~~~~ - **I/O** -- the ROOT job-file read/write and the shared sparse-table update are serialized in a critical section (one at a time, across all threads). This section does not parallelize: as the thread count grows, it becomes a larger share of the wall-clock time (Amdahl's law), capping the achievable speedup well below the thread count. - **Memory** -- each thread keeps its own private copy of the per-lag scratch buffers (the network/monster SSE/AVX working buffers used by the coherence and sub-network-cut calculations). Memory usage therefore grows roughly linearly with ``OMP_NUM_THREADS``, on top of the memory already used by the job itself. On memory-constrained nodes this can become the limiting factor before CPU time does. Expected speedup ~~~~~~~~~~~~~~~~~~~~ There is no fixed speedup factor to quote: it depends on the number of lags in the job, the fraction of time spent in the serialized I/O section versus the parallel per-lag computation, and the memory/CPU layout of the node. As a starting point for a BKG job: - start with a modest thread count (e.g. 4-8) and measure the actual wall-clock time and peak memory of a representative job before scaling up further; - do not request more threads than the job has lags to process; - watch memory headroom, not just CPU count, when deciding how many threads to use per node. Combining staged execution with OpenMP ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Since only the Coherence and SuperCluster stages benefit from OpenMP, the per-job pipeline can be split into stages -- each stage saving its output to a ROOT file that the next stage resumes from -- to concentrate threads only where they help, and avoid paying their memory cost everywhere else. This is the standard **multistage** ``cwb_inet2G`` mechanism (see `How to do an interactive multistages 2G analysis `__), for example: .. code-block:: bash # earlier stages: low/default thread count cwb_inet2G config/user_parameters.C INIT 1 cwb_inet2G data/init_..._job1.root STRAIN cwb_inet2G data/strain_..._job1.root CSTRAIN # the OpenMP bottleneck: raise OMP_NUM_THREADS just for these two export OMP_NUM_THREADS=8 cwb_inet2G data/cstrain_..._job1.root COHERENCE cwb_inet2G data/coherence_..._job1.root SUPERCLUSTER # remaining stages: back to a low/default thread count export OMP_NUM_THREADS=1 cwb_inet2G data/supercluster_..._job1.root LIKELIHOOD Each stage reads the ROOT file saved by the previous one, so it is safe to change ``OMP_NUM_THREADS`` between invocations. On a shared/memory-limited node this lets a job use several threads only while it is actually in the per-lag bottleneck, instead of holding that many private scratch buffers for the whole run. How to control it ~~~~~~~~~~~~~~~~~~~~~ The thread count is controlled with the standard ``OMP_NUM_THREADS`` environment variable -- there is no cWB-specific configuration parameter for it. OpenMP is an optional build-time dependency (see the `library README `__): a build where it could not be detected simply runs the same code, serially, and produces the same result.