OpenMP Parallelization

Since cWB-6.4.8.0, the Coherence and SuperCluster stages of the 2G pipeline (cwb2G.cc) can run multi-threaded via OpenMP, on top of the job-level parallelism already provided by condor/cwb_inet (see production). This page explains what is actually parallelized, when it helps, and what to expect in terms of memory and speed.

What is parallelized

In both stages, the loop that is parallelized runs over the time-shift lags of the job (one iteration per lag). Each lag is processed independently on its own per-thread scratch buffers; the only shared mutable state – the ROOT job-file I/O and the shared sparse-table update – is serialized in a critical section, so the result is bit-identical to the serial code regardless of the thread count.

Nothing else in the pipeline is parallelized this way: the pre-processing stages (data conditioning, whitening, likelihood/reconstruction) and the post-production stage remain single-threaded.

When it actually helps

The achievable speedup is bounded by the number of lags processed by the job:

  • Background (BKG) jobs – typically process many time-shift lags, commonly thousands (e.g. 2399 lags for a 1200 s segment) to accumulate background statistics. These are where OpenMP gives a real benefit, since there are many independent iterations to spread across threads. For simplicity, the zero-lag is normally not produced as a separate single-lag job: it is bundled into the same BKG job as just one more lag among the many, so it benefits from the same parallelism as the rest of the background.

  • Simulation (SIM) / MDC injection jobs – these are deliberately run at a lagged (non-zero-shift) time, rather than at the real zero-lag time, precisely to avoid reusing the same zero-lag data as the actual search result. If such a job only iterates over a handful of lags, there is correspondingly little for OpenMP to parallelize, and little or no speedup should be expected from raising OMP_NUM_THREADS. As always, the relevant number is how many independent lag iterations the specific job processes – not whether it is labelled “BKG” or “SIM”.

As a rule of thumb, requesting more threads than there are lags in the job does not help further – the extra threads simply stay idle.

Bottlenecks

  • I/O – the ROOT job-file read/write and the shared sparse-table update are serialized in a critical section (one at a time, across all threads). This section does not parallelize: as the thread count grows, it becomes a larger share of the wall-clock time (Amdahl’s law), capping the achievable speedup well below the thread count.

  • Memory – each thread keeps its own private copy of the per-lag scratch buffers (the network/monster SSE/AVX working buffers used by the coherence and sub-network-cut calculations). Memory usage therefore grows roughly linearly with OMP_NUM_THREADS, on top of the memory already used by the job itself. On memory-constrained nodes this can become the limiting factor before CPU time does.

Expected speedup

There is no fixed speedup factor to quote: it depends on the number of lags in the job, the fraction of time spent in the serialized I/O section versus the parallel per-lag computation, and the memory/CPU layout of the node. As a starting point for a BKG job:

  • start with a modest thread count (e.g. 4-8) and measure the actual wall-clock time and peak memory of a representative job before scaling up further;

  • do not request more threads than the job has lags to process;

  • watch memory headroom, not just CPU count, when deciding how many threads to use per node.

Combining staged execution with OpenMP

Since only the Coherence and SuperCluster stages benefit from OpenMP, the per-job pipeline can be split into stages – each stage saving its output to a ROOT file that the next stage resumes from – to concentrate threads only where they help, and avoid paying their memory cost everywhere else. This is the standard multistage cwb_inet2G mechanism (see How to do an interactive multistages 2G analysis), for example:

# earlier stages: low/default thread count
cwb_inet2G config/user_parameters.C INIT 1
cwb_inet2G data/init_..._job1.root      STRAIN
cwb_inet2G data/strain_..._job1.root    CSTRAIN

# the OpenMP bottleneck: raise OMP_NUM_THREADS just for these two
export OMP_NUM_THREADS=8
cwb_inet2G data/cstrain_..._job1.root    COHERENCE
cwb_inet2G data/coherence_..._job1.root  SUPERCLUSTER

# remaining stages: back to a low/default thread count
export OMP_NUM_THREADS=1
cwb_inet2G data/supercluster_..._job1.root LIKELIHOOD

Each stage reads the ROOT file saved by the previous one, so it is safe to change OMP_NUM_THREADS between invocations. On a shared/memory-limited node this lets a job use several threads only while it is actually in the per-lag bottleneck, instead of holding that many private scratch buffers for the whole run.

How to control it

The thread count is controlled with the standard OMP_NUM_THREADS environment variable – there is no cWB-specific configuration parameter for it. OpenMP is an optional build-time dependency (see the library README): a build where it could not be detected simply runs the same code, serially, and produces the same result.