Ginkgo: Linear Algebra Powered by SYCL

Rob_Mueller-Albrecht · ‎07-25-2023

Solve Sparse Linear Systems on your Platform of Choice

In our series of blogs on ready-to-use projects, solutions and products that are available for use with oneAPI and SYCL*, it is time to talk a bit more about the specifics of one of the projects I briefly introduced a short time ago.

The Innovative Computing Lab at the University of Tennessee at Knoxville under the leadership of Hartwig Anzt has been working on Ginkgo*, making it available with full support for the SYCL cross-platform abstraction layer and oneAPI.

Ginkgo is a modern C++ math library for scientific high performance computing. It is modular by design with the intent of actively being able to take advantage of GPUs and other accelerators. For GPU offload of its math algebra functions and solvers it relies on the implementation of executors. One such executor is the DPC++ Executor, a module specifically dedicated to enabling the use of Ginkgo Linear Operators with SYCL and thus allowing offload compute acceleration on GPUs and compute devices from a variety of vendors including Intel.

The key functionality Ginkgo provides consists of

Sparse Linear Solvers and Preconditioners
Sparse Matrix Vector Multiplication Functions

In short, Ginkgo is a high-performance linear algebra library for many core systems, with a focus on solution of sparse linear systems. It is implemented using C++ with GPU kernels supporting CUDA*, HIP*, and of course SYCL/oneAPI.

In this blog we will talk about the great work being done at the University of Tennessee at Knoxville as part of the oneAPI Academic Centers of Excellence.

We will also briefly talk about how Ginkgo fits into the oneAPI ecosystem and how it and its SYCL support augments the Intel® oneAPI Math Kernel Library (oneMKL).

University of Tennessee oneAPI Center of Excellence

The University of Tennessee in Knoxville joined the Intel oneAPI Academic Centers of Excellence in fall 2021. They are focusing on two projects:

oneAPI and SYCL Support with Ginkgo expanding hardware vendor choice for sparse linear solvers and vector math in high performance computing .
Enabling high-end visualization as a service through oneAPI that can be instantly available and universally accessible.

Professor Hartwig Anzt is the Director of the Innovative Computing Lab (ICL) and Professor in the Electrical Engineering and Computer Science Department of the University of Tennessee. He and his team lead the effort to allow Ginkgo to take full advantage of oneAPI, an open sourced, unified programming model that simplifies development across multiple types of architectures (CPUs, GPU, FPGAs, and other accelerators).

Their interest in algorithm research into sparse linear algebra, iterative and asynchronous methods, Krylov solvers and preconditioning is complemented with efforts aiming at sustainable software development in an academic setting, and a healthy software lifecycle. One of the goals of these research efforts is to help prepare Ginkgo for the Aurora supercomputer at Argonne National Laboratory.

Jian Huang, a professor in the Min H. Kao Department of Electrical Engineering and Computer Science, and his team employ oneAPI to help end the perception that high-end visualization has to be heavy-weight, cumbersome to deploy, and expensive to use. They do this by providing high-end visualization as an easily accessible service, taking advantage of Intel’s family of rendering and ray-tracing Libraries.

These projects along with additional work on technology development and curriculum design help drive ecosystem adoption of oneAPI and thus adoption of a common open-standards based paradigm for flexible multi-vendor cross-architecture compute acceleration.

Get Started with Ginkgo

With its focus on sparse linear Krylov solvers and matrix vector multiplication Ginkgo is perfectly positioned to augment and add to the domains already covered by the oneMKL domains as part of the oneAPI specification such as dense and sparse linear algebra functions (BLAS, LAPACK), vector math functions (VM), discrete Fourier transforms (DFT), random number generation (RAND).

It also augments the parallel direct sparse solver interface (PARDISO) additionally covered by the Intel® oneAPI Math Kernel Library product.

Ginkgo ensures oneAPI ecosystem access to a comprehensive sparse linear solver solution.

It is implemented in modern C++ and provides interfaces to common simulation frameworks, including MFEM*, SUNDIALS*, deal.ii*, HyTeg*, openCARP*, and OpenFOAM*.

You can access Ginkgo on GitHub. There is a dedicated development branch for SYCL & oneAPI support as well as a rich set of code examples.

Please consult the documentation for details.

The main Krylov subspace solver classes it provides are:

BICG: Biconjugate gradient method
BiCGSTAB: Bi-Conjugate Gradient-Stabilized solver
CB-GMRES: Compressed basis generalized minimal residual method. An iterative method suitable for nonsymmetric linear systems.
CG: Conjugate gradient method. An iterative method suitable for symmetric positive definite methods.
CGS: Conjugate gradient square method. An iterative method suitable for general systems.
FCG: Flexible conjugate gradient method. An iterative method suitable for symmetric positive definite data configurations.
GCR: Generalized conjugate residual method. An iterative method similar to GMRES suitable for nonsymmetric linear systems.
GMRES: Generalized minimal residual method is an iterative method suitable for nonsymmetric linear systems.

Other solver classes:

IDR: An efficient method for solving large nonsymmetric systems of linear equations.
IR: Iterative refinement uses a coarse method to approximate the error of the current solution via the current residual.
Multigrid methods have a hierarchy of many levels, whose coarse level is a subset of the fine level, of the problem.
LowerTrs: Triangular solver which solves the system L x = b, when L is a lower triangular matrix.
UpperTrs: Triangular solver which solves the system U x = b, when U is an upper triangular matrix.

Vector Math matrix formats classes:

Vector stores (multiple) distributed column vectors in a dense storage format.
COO stores a matrix in the coordinate matrix format.
CSR stores only the nonzero coefficients by compressing each row of the matrix (compressed sparse row format).
Dense explicitly stores all values of the matrix.
Diagonal implements the diagonal matrix (a linear operator which scales a vector row wise).
ELL: stride with explicit zeros is used such that all rows have the same number of stored elements.
Fixed-block compressed sparse row storage.
Fft implements a 1D Fourier matrix using the FFT algorithm.
Fft2 implements a 2D Fourier matrix using the FFT algorithm.
Fft3 implements a 3D Fourier matrix using the FFT algorithm.
Hybrid splits the matrix into ELLPACK and COO format.
Identity efficiently implements the identity matrix (a linear operator which maps each vector to itself).
IdentityFactory generates Identity operators.
Permutation stores the row and column permutation arrays for re-ordering the rows and columns a matrix.
SELL-P is a matrix format similar to ELL format.
SparsityCsr stores only the sparsity pattern of a sparse matrix by compressing each row of the matrix.

To build Ginkgo with oneAPI and SYCL support on a Linux* systems it is recommended to have the following packages installed:

cmake 3.13+
Intel® oneAPI Base Toolkit 2023.x+ including

Intel® oneAPI DPC++/C++ Compiler
Intel® oneAPI Math Kernel Libraty (oneMKL)
Intel® oneAPI DPC++ Library (oneDPL)

Set dpcpp as the CMAKE_CXX_COMPILER
Your code should C++17 compliant

Detailed instruction can be found on the Ginkgo project GitHub installation page and example program documentation. Help is available through issue submission on GitHub.

Join Us

Become part of the effort to make high-performance cross-architecture compute transparent, portable, and flexible. Include SYCL as the accelerator and GPU offload solution in your code path. Adopt oneAPI as the means to implementations free from vendor-lock.

Get started with both Ginkgo and Intel oneAPI Math Kernel Library (oneMKL).

Contribute to the Ginkgo project and the oneMKL SYCL API.

Make it part of your software solution.

Please stay tuned for next week’s oneAPI project focus.

Additional SYCL Resources

Additional Ginkgo Resources

Notices and Disclaimers

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex. Results may vary.

Performance results are based on testing as of dates shown in configurations and may not reflect all publicly available updates. No product or component can be absolutely secure.

Your costs and results may vary.

Intel technologies may require enabled hardware, software or service activation.

Intel does not control or audit third-party data. You should consult other sources to evaluate accuracy.

© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries.
*Other names and brands may be claimed as the property of others.