DP3 performance analysis and optimization

Parallelism, memory accesses and vectorization

Hamza Chouh, EPFL

DP3 in the SKA Science Data Processor

  • DP3: LOFAR pipeline software (ASTRON): averaging, flagging, calibration, prediction
  • Used for the preprocessing / self-calibration engine of the SKA Science Data Processor
  • Self-calibration alternates predict (simulate a model sky) and solve (fit corrections against it)

The predict step

  • Simulates model visibilities for every baseline × channel × correlation, from a sky model and a telescope model
  • Per source: simulate its contribution, accumulate into a per-patch buffer
  • Per patch: compute the beam response, apply it to that patch’s buffer
  • Mostly DP3 internal implementation; calls EveryBeam for the beam response

Test configuration used for benchmarking in this talk

quantity value
stations 68
baselines 2 278
channels 416
sky components 25 085
polarization unpolarized (Stokes I only)
patches 14 365
timesteps 100
machine 72 cores (2x Intel Xeon Platinum 8360Y)

Original work distribution

Structure:

  • Sources statically split into equal-count slices, one per thread
  • Each thread simulates its sources and computes beam responses independently into a private accumulator

Potential issues:

  • Can result in a workload discrepancy
  • Beam computations are duplicated when shared among multiple threads
metric value
predict 1 161 s
wall 1 612 s
user CPU 66 623 s
mean parallelism 41.3× of 72 (57%)

Performance measurements

  • 32.0% effective CPU utilization — 23.0 of 72 cores (VTune, spin and overhead discounted)
  • 3.45x cycle-count spread, busiest vs. idlest core

  • 4→72 threads: 9.0x against an ideal 18x — 50% scaling efficiency

Per-thread load

  • Flat item count: 348–349 sources on every thread, yet thread time spreads 2.0x
  • simulate() itself only spreads 1.7x — 93% of components are Gaussian, so an equal-count split gives every thread much the same mix
  • The spread is in the rest: the beam-apply remainder varies 7.2x across threads and is 40% of thread busy time
  • Equal source counts are not equal patch counts — and the beam fires per patch

Dynamic dispatch via Intel Thread Building Blocks (TBB) library

  • Same per-source granularity as the static split, dispatched dynamically through tbb::parallel_for instead of a fixed slice per thread
  • Work-stealing scheduler: idle threads pick up remaining sources instead of sitting still
  • Low overhead scheduler + minimal code to maintain

TBB attempt: results

6–24% faster than the static baseline at every measured thread count: 24% at 24 threads, but 7% at 72. Still far short of what the profiling budget suggested was available.

Per-thread load, TBB

  • Work stealing redistributes work ☭: between 249 and 498 items per thread
  • Better per-thread time balance
  • Only 5% of the makespan saved at 72 threads: total work grew 22% (456 → 556 thread-seconds)
  • 89 of those 100 extra thread-seconds are beam applies (183 → 272, +49%): the same patch is re-applied in every range that touches it
metric static split TBB stealing
thread utilization 77.4% 99.6%
time spread 64.2% 0.6%
idle, per timestep 1.88s 0.03s

Over a thread’s whole work, per timestep, averaged over 28 timesteps.

Removing the redundant beam applies

  • Structural redundancy: a patch split across a range boundary is applied once in each range — applies ≈ patches + ranges, so the finer the stealing, the more applies

  • Four variants tried, verdicts under each panel

    ==> General conclusion: tradeoff between parallelism efficiency and memory overhead

Memory traffic

  • Next step was to look at memory reads/writes and how to optimize them

metric static TBB
Memory Bound 77% 76%
L3 Bound 1.4% 1.3%
DRAM Bound 16.7% 19.3%
Load ops 1.9T 2.1T
LLC misses 4.8B 5.1B

Work stealing moves +6% LLC misses in our test case.

Tile by channel and source

  • Each thread works a channel slab × source block tile sized to stay inside L2, instead of every source sweeping the full channel range
  • Current implementation only works on unpolarized sky models
  • Hypothesis: expected to cut cache misses and reduce pressure on DDR

Impact of the new layout

  • 1 161s → 562s predict time, 2.07x
  • Beam applications are done on-the-fly in the Stokes-I path, so the per-patch buffer round-trip disappears: 183s of thread time → 0.5s
  • Hypothesis confirmed — the tile really does keep a thread’s working set in cache:
original tiled
memory bound 77% 13% 6.0×
DRAM bound 17% 3% 5.8×
store bound 10.6% 0.9% 11.8×
LLC misses 4.8 B 0.33 B 14.6×

Memory traffic analysis of the simulation phase of the prediction step.

Vectorizing the inner loops

  • Structure of arrays pattern for processing, then written into array of structures
  • Slab rounded to a multiple of an SIMD lane to avoid scalar epilogues
  • xsimd::sincos in phases() and xsimd::exp in the Gaussian envelope
  • Combined performance gain: 3.12s → 0.74s per normal timestep, 4.2x (1.67x on the whole predict step)

Overall effect

  • The work was on the simulation step only, which was dominant.
  • Beam compute is now 79% of the predict step.
  • On the simulation phase, memory bound 77% → 13% and 14.6x fewer LLC misses
  • Over the whole predict step both shrink less: the untouched beam dominates what is left

Per-thread load, final version

  • Beam application is now contained in simulate()
  • Residual spread is a granularity floor: going finer leads to worse performance

Final results

  • 3.4x faster predict step, upstream → current (1 161s → 337s over the full 100 timesteps; the sweep above is the 30-timestep one)

Full dataset, end to end

  • The production pipeline over the whole 900-timestep set, one node, one binary: 11 811s → 4 250s wall clock, 7 561s saved per run
  • Predict step 3.57x, and 2.17x against FastPredict

Conclusion / discussion

  • Correctness: unit tests + results validated on a realistic test case
  • Good performance gains through a loop reordering
  • Takeaway:
    • a better scheduler alone had a ceiling close to the baseline
    • it balanced the threads much better
    • but paid for it in both compute and memory accesses from redundant per-patch work
    • the real gain came from reordering memory access
  • Perspective:
    • polarized sky models. The tiled path needs Stokes I today; applying the Jones matrix per patch per channel slab keeps the tile in L2 and saves memory traffic
    • GPU implementation
    • applying FastPredict improvements (look into trigonometry lookup tables, source type batching, some possible precompute)