Detail publikace

Large-scale Ultrasound Simulations Using the Hybrid OpenMP/MPI Decomposition

JAROŠ, J. NIKL, V. TREEBY, B.

Originální název

Large-scale Ultrasound Simulations Using the Hybrid OpenMP/MPI Decomposition

Typ

článek ve sborníku mimo WoS a Scopus

Jazyk

angličtina

Originální abstrakt

The simulation of ultrasound wave propagation through biological tissue has a wide range of practical applications including planning therapeutic ultrasound treatments of various brain disorders such as brain tumours, essential tremor, and Parkinson's disease. The major challenge is to ensure the ultrasound focus is accurately placed at the desired target within the brain because the skull can significantly distort it. Performing accurate ultrasound simulations, however, requires the simulation code to be able to exploit several thousands of processor cores and work with datasets on the order of tens of TB.We have recently developed an efficient full-wave ultrasound model based on the pseudospectral method using pure-MPI with 1D slab domain decomposition that allows simulations to be performed  using up to 1024 compute cores. However, the slab decomposition limits the number of compute cores to be less or equal to the size of the longest dimension, which is usually below 1024. This paper presents an improved implementation that exploits 2D hybrid OpenMP/MPI decomposition. The 3D grid is first decomposed by MPI processes into slabs. The slabs are further partitioned into pencils assigned to threads on demand. This allows 8 to 16 times more compute cores to be employed compared to the pure-MPI code, while also reducing the amount of communication among processes due to the efficient use of shared memory within compute nodes. The hybrid code was tested on the Anselm Supercomputer (IT4Innovations, Czech Republic) with up to 2048 compute cores and the SuperMUC supercomputer (LRZ, Germany) with up to 8192 compute cores. The simulation domain sizes ranged from 256^3 to 1024^3 grid points. The experimental results show that the hybrid decomposition can significantly outperform the pure-MPI one for large simulation domains and high core counts, where the efficiency remains slightly below 50%. For a domain size of 1024^3, the hybrid code using 8192 cores enables the simulations to be accelerated by a factor of 4 compared to the pure-MPI code. Deployment of the hybrid code has the potential to eventually bring the simulation times within clinically meaningful timespans, and allow detailed patient specific treatment plans to be created.

Klíčová slova

Ultrasound simulations, 2D domain decomposition, OpenMP/MPI, Hybrid programming; Performance evaluation, Supercomputing, k-Wave toolbox.

Autoři

JAROŠ, J.; NIKL, V.; TREEBY, B.

Rok RIV

2015

Vydáno

31. 5. 2015

Nakladatel

Association for Computing Machinery

Místo

Edinburgh

ISBN

978-0-9926615-1-9

Kniha

Proceedings of the 3rd International Conference on Exascale Applications and Software

Strany od

115

Strany do

119

Strany počet

5

URL

BibTex

@inproceedings{BUT119818,
  author="Jiří {Jaroš} and Vojtěch {Nikl} and Bradley {Treeby}",
  title="Large-scale Ultrasound Simulations Using the Hybrid OpenMP/MPI Decomposition",
  booktitle="Proceedings of the 3rd International Conference on Exascale Applications and Software",
  year="2015",
  pages="115--119",
  publisher="Association for Computing Machinery",
  address="Edinburgh",
  isbn="978-0-9926615-1-9",
  url="http://www.easc2015.ed.ac.uk/proceedings"
}