Human Arterial Tree Project
Software
Nektar
High-order accuracy in numerical methods is important for simulating unsteady and transitional flows. This need has been recognized by the biofluids community, and efforts have been made to increase the standard methods.
Our research code Nektar is based on the spectral/hp element method developed by Professors Karniadakis and Sherwin. It employs an unstructured finite element mesh with a spectral expansion within each element. Resolution can be increased by increasing either the polynomial order (p-type refinement) of the element or the number of elements (h-type refinement). Discretization for complex geometries is efficient and achieves global spectral accuracy. A stiffly stable time stepping scheme is utilized with temporal accuracy up to third-order. Nektar is freeware and is being used by several research teams around the world.
Nektar has already been applied with great success to hemodynamical flows within bypass grafts and has been compared with in-vitro models using MRI imaging to capture the velocity profiles.
The Challenge
The human vascular system has 55 main arteries with 27 bifurcations, creating a problem of enormous size that presents a great computational challenge. Memory consumption for full resolution is about 7 terabytes. This problem is currently beyond the capacity of a single supercomputer available to the open research community, necessitating the use of many connected supercomputers.
Our Solution: Nektar-G2
Nektar-G2 is a grid-enabled parallel version of Nektar that allows us to solve problems on geographically distributed supercomputers. It was featured in the winter issue 1998 of NCSA/ACCESS magazine with the title Code the keeps blood flowing.
What makes it possible: MPICH-G2
MPICH-G2 is the message passing library that is used within Nektar-G2. It is an MPI library implementation that utilizes the Globus Toolkit and allows MPI applications to be run across multiple computers at different sites. Its performance is on par with native MPI implementations within a machine.
Our Approach
We take a hybrid approach that allows the detailed 3D hemodynamics at arterial bifurcations to be coupled with a representation, obtained from reduced 1D equations, of the entire arterial tree that models waveform coupling between bifurcations and incorporates information about sectional area, velocity, and pressure.
Computation Algorithm
Click on the image to view the video. (550KB)
The overall simulation consists of a 1D computation throughout the full arterial tree coupled with a detailed 3D simulation of flow at arterial bifurcations. The 1D system of equations provides the appropriate boundary conditions describing flow rate and pressure for the 3D Navier-Stokes computations.
The topology-aware feature of MPICH-G2 is utilized to enforce the data distribution strategy. The following process groups and MPI communicators are involved: 1D group for 1D model computation, which involves only intra-site communications; bifurcation groups, one for each arterial bifurcation, which require only intra-site communications; boundary condition groups, one for each arterial bifurcation, which require inter-site communications.
