11. AOCL-Sparse#

AOCL-Sparse is a library containing basic linear algebra subroutines for sparse matrices and vectors (Sparse BLAS). It is designed to be used with C, C++, and compatible languages.

The current functionality of AOCL-Sparse is organized in the following categories:

Sparse Level 1 functions perform vector operations, such as dot product, vector additions on sparse vectors, gather, scatter, and other similar operations.
Sparse Level 2 functions describe the operations between a matrix in sparse format and a vector in dense format, including matrix-vector product (SpMV), triangular solve (TRSV), and so on.
Sparse Level 3 functions describe the operations between a matrix in sparse format and a matrix in dense/sparse format. The operations comprise of matrix additions (SpADD), matrix-matrix product (SpMM, SpMMd, Sp2M, Sp2Md), triangular solver with multiple right-hand sides (TRSM), symmetric product of 3 matrices (sypr, syprd) and matrix-its transpose product (syrk, syrkd).
Iterative sparse solvers based on Krylov subspace methods (CG, GMRES) and preconditioners (such as SymGS, SOR and ILU0).
Sparse format conversion functions for translating matrices in a variety of sparse storage formats.
Auxiliary functions to allow basic operations, including create, copy, destroy, and modify matrix handles and descriptors.

Additional Highlights

Supported data types: single, double, and the complex variants
0-based and 1-based indexing of sparse formats
Hint & Optimize framework to accelerate the supported functions by a prior matrix analysis based on the users’ hints of expected operations.

Multi-Thread Support

You can set the total number of threads using the environment variables AOCLSPARSE_NUM_THREADS or OMP_NUM_THREADS. If both environment variables are set, AOCL-Sparse gives higher precedence to AOCLSPARSE_NUM_THREADS. If neither variable is set, the default number of threads is equal to the number of processors identified by the OpenMP library. The functions with multi-thread support include SpMV variants for float and double datatypes, TRSM, SpMM, Sp2M and SpAdd.

For more information on the AOCL-Sparse APIs, refer AOCL-Sparse_API_Guide.pdf in the docs folder (amd/aocl-sparse).

11.1. Installation#

AOCL-Sparse can be installed from the source or pre-built binaries.

11.1.1. Building AOCL-Sparse from Source on Linux#

The following prerequisites must be met:

Git
CMake. For more information on CMake versions validated, refer to Build Utilities
Boost library versions 1.80 through 1.85

Note

This is needed only for benchmarks (BUILD_CLIENTS_BENCHMARKS=ON). For more information, refer to AOCL-Sparse - CMake Build Options

Complete the following steps to build different packages of the library, including dependencies and test application:

Install AOCL-BLAS, AOCL-LAPACK, and AOCL-Utils.
Define the environment variable AOCL_ROOT to point to AOCL Libs installation:
```
$ export AOCL_ROOT=/opt/aocl
```
For the case where AOCL_ROOT cannot be exported by placing AOCL-BLAS, AOCL-LAPACK, and AOCL-Utils libraries in the same path, you can use the following variables during the CMake configuration to point to the location of the dependent libraries and headers:
- AOCL_BLIS_LIB
- AOCL_LIBFLAME
- AOCL_UTILS_LIB
- AOCL_BLIS_INCLUDE_DIR
- AOCL_LIBFLAME_INCLUDE_DIR
- AOCL_UTILS_INCLUDE_DIR
Download the latest release of AOCL-Sparse (amd/aocl-sparse),

or

Clone the Git repository (amd/aocl-sparse.git).

Configure the project using the following tables:

Table 11.1 **Compiler and Library Type**#
Compiler	Library Type	ILP 64 Support	Command
G++ (Default)	Static	OFF (Default)	`cmake -S . -B out_sparse -DBUILD_SHARED_LIBS=OFF`
		ON	`cmake -S . -B out_sparse -DBUILD_SHARED_LIBS=OFF -DBUILD_ILP64=ON`
	Shared (Default)	OFF (Default)	`cmake -S . -B out_sparse`
		ON	`cmake -S . -B out_sparse -DBUILD_ILP64=ON`
AOCC	Static	OFF (Default)	`cmake -S . -B out_sparse -DCMAKE_CXX_COMPILER=clang++ -DBUILD_SHARED_LIBS=OFF`
		ON	`cmake -S . -B out_sparse -DCMAKE_CXX_COMPILER=clang++ - DBUILD_SHARED_LIBS=OFF -DBUILD_ILP64=ON`
	Shared (Default)	OFF (Default)	`cmake -S . -B out_sparse -DCMAKE_CXX_COMPILER=clang++`
		ON	`cmake -S . -B out_sparse -DCMAKE_CXX_COMPILER=clang++ -DBUILD_ILP64=ON`

Table 11.2 **AOCL-Sparse - CMake Build Options**#
Build Option	Feature
`CMAKE_INSTALL_PREFIX`	Use `-DCMAKE_INSTALL_PREFIX=<path>` to choose the custom path. The default install path is `/opt/aoclsparse/`
`CMAKE_BUILD_TYPE`	Release => Release Library (Default) Debug => Debug Library
`CMAKE_CXX_COMPILER`	Use `-DCMAKE_CXX_COMPILER=clang++` for AOCC builds
`BUILD_SHARED_LIBS`	OFF => Build Static Library ON => Build Dynamic/Shared library (Default)
`BUILD_ILP64`	Integer length: OFF => 32-bit integer length (Default) ON => 64-bit integer length
`SUPPORT_OMP`	Multi-threading using OpenMP: OFF => Disable OpenMP ON => Enable OpenMP (Default)
`USE_AVX512`	OFF => Disables AVX512 kernel support (Default) ON => Enables AVX512 kernel support
`BUILD_CLIENTS_BENCHMARKS`	OFF => Disable building benchmarks (Default) ON => Build benchmark framework (requires Boost library)
`BUILD_CLIENTS_SAMPLES`	OFF => Disable building sparse API examples ON => Enable building sparse API examples (Default)
`BUILD_UNIT_TESTS`	OFF => Unit tests are not built ON => Unit tests are built and target `test` is activated. This target should be used to test the correctness of the compiled library (requires GTest)
`BUILD_DOCS`	OFF => No documentation target is activated. (Default) ON => Build PDF and HTML API documentation, this adds target `docs` (requires Linux and modern LaTeX distribution). Uses `rocm-docs-core` python package version 0.32 (`python3 -m pip install --upgrade rocm-docs- core==0.32`)

Build the AOCL-Sparse library:
```
$ cmake --build out_sparse --config <build_type> --parallel <nproc>
```
where,
- build_type can be Release/Debug as configured during configuration.
- nproc is the number of cores for the build to run in parallel.
Install AOCL-Sparse to the directory /opt/aoclsparse or a custom path given by CMAKE_INSTALL_PREFIX:
```
cmake --build out_sparse --target install
```

Note

If AOCL_ROOT contains the shared and static libraries in the same directory for any dependent library, Sparse will link to the shared library irrespective of BUILD_SHARED_LIBS.

11.1.2. Building AOCL-Sparse from Source on Windows#

AOCL-Sparse uses CMake along with Microsoft Visual Studio for building binaries from the source on Windows. The following sections explain the GUI and command-line schemes of building the binaries and test suite.

Prerequisites

For more information, refer to the Prerequisites in section Build AOCL-BLAS from Source on Windows.

11.1.2.1. Using CMake and Visual Studio GUI#

Complete the following steps to prepare the project with CMake GUI:

Install AOCL-BLAS, AOCL-LAPACK, and AOCL-Utils.
Define the environment variable AOCL_ROOT to point to the AOCL Libs installation:
```
$ set AOCL_ROOT="C:\Program Files\AMD\AOCL-Windows"
```
For the cases where AOCL_ROOT cannot be exported by placing AOCL-BLAS, AOCL-LAPACK, and AOCL-Utils libraries in the same path, define the following variables during CMake configuration in cmake-gui to point to the corresponding libraries and headers:
- AOCL_BLIS_LIB
- AOCL_LIBFLAME
- AOCL_UTILS_LIB
- AOCL_BLIS_INCLUDE_DIR
- AOCL_LIBFLAME_INCLUDE_DIR
- AOCL_UTILS_INCLUDE_DIR
Launch CMake GUI using the Windows command line:
```
$ cmake-gui
```
Set the source (folder containing the AOCL-Sparse source code) and build (folder in which the project files will be generated) folder paths. It is not recommended to use the folder named build as it is already used for Linux build system.
Click on the Configure button to prepare the project options.
Set the generator to Visual Studio 17 2022 and the platform toolset to clangCl:

Figure 11.1 Specify Generator#
Update the options based on the project requirements. All the available options are listed in AOCL-Sparse - CMake Build Options. Select the available and recommended options as follows:

Figure 11.2 AOCL-Sparse CMake Config Options#
Click the Generate button and then Open Project.
Complete the following steps in Microsoft Visual Studio GUI:
1. Open the AOCL-Sparse Visual Studio project from the build folder using the aoclsparse.sln file or the Open Project button in CMake GUI.
2. To generate the AOCL-Sparse binaries, choose the appropriate build configuration Debug or Release and then build the AOCL-Sparse project. The library files would be generated at <build_dir>\library\Release.

11.1.2.2. Using Windows Command-Line#

Complete the following steps to configure the project using command prompt:

Install AOCL-BLAS, AOCL-LAPACK, and AOCL-Utils.
Define the environment variable AOCL_ROOT to point to the AOCL Libs installation:
```
$ set AOCL_ROOT="C:\Program Files\AMD\AOCL-Windows"
```
For the cases where AOCL_ROOT cannot be exported by placing AOCL-BLAS, AOCL-LAPACK, and AOCL-Utils libraries in the same path, define the following variables during CMake configuration to point to the corresponding libraries and headers:
- AOCL_BLIS_LIB
- AOCL_LIBFLAME
- AOCL_UTILS_LIB
- AOCL_BLIS_INCLUDE_DIR
- AOCL_LIBFLAME_INCLUDE_DIR
- AOCL_UTILS_INCLUDE_DIR
Checkout the AOCL-Sparse directory:
```
$ cd aocl-sparse
```

Configure the project along with the following options depending on the required build. For CMake configure options, refer to AOCL-Sparse - CMake Build Options. When AOCL_ROOT is defined:

$ cmake -S . -B out_sparse -T clangcl -G "Visual Studio 17 2022"
-DCMAKE_CXX_COMPILER=clang-cl -DCMAKE_INSTALL_PREFIX="<aoclsparse_install_path>"

When CMake variables are used to define AOCL-BLAS, AOCL-LAPACK, and AOCL-Utils libraries/header:

$ cmake -S . -B out_sparse -T clangcl -G "Visual Studio 17 2022"
-DCMAKE_CXX_COMPILER= clang-cl
-DCMAKE_INSTALL_PREFIX="<aoclsparse_install_path>"
-DAOCL_LIBFLAME="Lapack/Library/with/path"
-DAOCL_LIBFLAME_INCLUDE_DIR="path/to/Lapack/Headers"
-DAOCL_BLIS_LIB="Blas/Library/with/path"
-DAOCL_BLIS_INCLUDE_DIR="path/to/Blas/Headers"
-DAOCL_UTILS_LIB="Utils/Library/with/path"
-DAOCL_UTILS_INCLUDE_DIR="path/to/Utils/Headers"

Refer to AOCL-Sparse - CMake Build Options to update the parameter options in the command according to the project requirements.

Export the paths for AOCL-BLAS, AOCL-LAPACK, and AOCL-Utils libraries:

$ set PATH="C:\Users\Program Files\AMD\AOCL-Windows\amd-blis\lib\LP64;%PATH%"
$ set PATH="C:\Users\Program Files\AMD\AOCL-Windows\amd-libflame\lib\LP64;%PATH%"
$ set PATH="C:\Users\Program Files\AMD\AOCL-Windows\amd-utils\lib;%PATH%"

Build the project:

$ cmake --build out_sparse --config Release --target install --verbose

Install AOCL-Sparse:

$ cmake --build out_sparse/ --target install

11.1.3. Using Pre-Built Libraries#

You can install the AOCL-Sparse binaries using the packages available at the following URL:

https://www.amd.com/en/developer/aocl/sparse.html

Also, you can install AOCL-Sparse binary from the AOCL master installer tar file available at the following URL:

https://www.amd.com/en/developer/aocl.html

Note

The pre-built libraries are prepared on a specific platform having dependencies related to operating system, compiler (GCC, Clang, Visual Studio), and glibc. Your platform must adhere to the same versions of these dependencies to use those libraries.

11.2. Usage#

The library includes sample programs demonstrating the usage of AOCL-Sparse APIs and they can be used as a starting point to build your application, executable binaries to perform benchmarking and unit tests to check the correctness of the library build. These are located in tests/examples, tests/benchmarks and tests/unit_tests directories respectively.

11.2.1. Use by Applications on Linux#

To use AOCL-Sparse in your application, compile your source with C or C++ compiler including the AOCL-Sparse header files (-I$SPARSE_ROOT/include). If 64-bit integers are used, define the appropriate macro (-Daoclsparse_ILP64). Afterwards, link the objects with C++ linker to the AOCL-Sparse library, dependencies (libflame, libblis, and libaoclutils in that order), and pthreads library. You must match the size of integers for the dependency libraries (LP64 versus ILP64). If multi-threaded library is used, appropriate compiler flags must be defined and the corresponding dependent libraries must be linked. Either static or shared libraries can be used. In the case of shared libraries, you must set their location in LD_LIBRARY_PATH before running the resulting executables.

The following sections provide commands to manually build two sample programs from AOCL-Sparse examples directory (tests/examples) in two different scenarios.

11.2.1.1. Link to Single-Threaded, LP64, Static Sparse Library#

$ export AOCL_ROOT=/opt/aocl
$ export SPARSE_ROOT=<aoclsparse_install_path>
$ export LD_LIBRARY_PATH=$AOCL_ROOT/lib_LP64:$LD_LIBRARY_PATH

$ g++ sample_spmv.cpp -I$SPARSE_ROOT/include -I$AOCL_ROOT/include_LP64
-I$AOCL_ROOT/include_LP64/ alci/.. $SPARSE_ROOT/lib/libaoclsparse.a
-Wl,-rpath,$AOCL_ROOT/lib_LP64 $AOCL_ROOT/ lib_LP64/libflame.so
$AOCL_ROOT/lib_LP64/libblis.so $AOCL_ROOT/lib_LP64/libaoclutils.so
-lpthread -o sample_spmv

$ g++ sample_spmv_c.c -I$SPARSE_ROOT/include -I$AOCL_ROOT/include_LP64
-I$AOCL_ROOT/include_LP64/ alci/.. $SPARSE_ROOT/lib/libaoclsparse.a
-Wl,-rpath,$AOCL_ROOT/lib_LP64 $AOCL_ROOT/ lib_LP64/libflame.so
$AOCL_ROOT/lib_LP64/libblis.so $AOCL_ROOT/lib_LP64/libaoclutils.so
-lpthread -o sample_spmv_c

./sample_spmv
./sample_spmv_c

11.2.1.2. Link to Multi-Threaded, ILP64, Shared Sparse Library#

$ export AOCL_ROOT=/opt/aocl
$ export SPARSE_ROOT=<aoclsparse_install_path>
$ export LD_LIBRARY_PATH=$AOCL_ROOT/lib_ILP64:$LD_LIBRARY_PATH
$ export LD_LIBRARY_PATH=$SPARSE_ROOT/lib:$LD_LIBRARY_PATH

$ g++ sample_spmv.cpp -I$SPARSE_ROOT/include -I$AOCL_ROOT/include_ILP64
-I$AOCL_ROOT/ include_ILP64/alci/.. -Daoclsparse_ILP64 -fopenmp
$SPARSE_ROOT/lib/libaoclsparse.so - Wl,-rpath,$AOCL_ROOT/lib_ILP64
$AOCL_ROOT/lib_ILP64/libflame.so $AOCL_ROOT/lib_ILP64/ libblis.so
$AOCL_ROOT/lib_ILP64/libaoclutils.so -lpthread -o sample_spmv

$ g++ sample_spmv_c.c -I$SPARSE_ROOT/include -I$AOCL_ROOT/include_ILP64
-I$AOCL_ROOT/ include_ILP64/alci/.. -Daoclsparse_ILP64 -fopenmp
$SPARSE_ROOT/lib/libaoclsparse.so - Wl,-rpath,$AOCL_ROOT/lib_ILP64
$AOCL_ROOT/lib_ILP64/libflame.so $AOCL_ROOT/lib_ILP64/ libblis.so
$AOCL_ROOT/lib_ILP64/libaoclutils.so -lpthread -o sample_spmv_c

./sample_spmv
./sample_spmv_c

To build all the example programs using CMake build system, refer to README.md in tests/examples.

11.2.2. Use by Application on Windows#

Complete the following steps to use by application on Windows:

Move to installed examples directory:
```
$ cd <install_dir>\examples
```
Define the environment variable AOCL_ROOT to point to AOCL libs installation that has AOCL-BLAS, AOCL-LAPACK and AOCL-Utils libraries:
```
$ set AOCL_ROOT="C:\Program Files\AMD\AOCL-Windows"
```
Define SPARSE_ROOT to point to the AOCL-Sparse package installation path:
```
$ set SPARSE_ROOT="%HOME%\amd\aocl-sparse"
```

Add AOCL-Sparse, AOCL-BLAS, AOCL-LAPACK and AOCL-Utils library paths to the environment path variable:

$ set PATH="%SPARSE_ROOT%\lib;%PATH%"
$ set PATH="%AOCL_ROOT%\amd-blis\lib\ILP64;%PATH%"
$ set PATH="%AOCL_ROOT%\amd-libflame\lib\ILP64;%PATH%"
$ set PATH="%AOCL_ROOT%\amd-utils\lib;%PATH%"

Configure the build system to compile sample applications. Ensure that the build configuration is same as the one used to build the AOCL-Sparse library for the variable BUILD_ILP64:
```
$ cmake -S . -B out_sparse -G "Visual Studio 17 2022" -T "clangcl"
-DCMAKE_CXX_COMPILER=clang-cl
-DBUILD_SHARED_LIBS=ON -DBUILD_ILP64=OFF -DSUPPORT_OMP=OFF
```
Compile the sample applications:
```
$ cmake --build out_sparse
```
Run the application:
```
$ .\out_sparse\sample_spmv.exe
```

Note

The environment variable SPARSE_ROOT takes precedence when searching for sparse library and headers. If the environment variable SPARSE_ROOT is not defined, SPARSE_ROOT inherits the path from AOCL_ROOT and looks for AOCL-Sparse installation in AOCL_ROOT. If the sparse library/headers are not found neither in SPARSE_ROOT nor in AOCL_ROOT, an error is returned.

11.2.3. Performance Benchmarking on Linux#

The AOCL-Sparse benchmark executable (called aoclsparse-bench) accepts various input parameters including matrix data and triggers the desired operation while measuring the API performance.

Optionally, it can also check the results against its reference implementation. The matrix data can be randomly generated or read from the Matrix Market format (mtx) input file. The MTX inputs can be downloaded from the SuiteSprase Matrix Collection website (https://sparse.tamu.edu/). Usage of both the input types is as follows:

Enable BUILD_CLIENTS_BENCHMARKS during AOCL-Sparse installation process (refer to Building AOCL-Sparse from Source on Linux).

Export paths to the dependent libraries in LD_LIBRARY_PATH if building shared library of AOCL-Sparse:

$ export AOCL_ROOT=/opt/aocl
$ export SPARSE_ROOT=<aoclsparse_install_path>
$ export LD_LIBRARY_PATH=$AOCL_ROOT/lib_ILP64:$LD_LIBRARY_PATH
$ export LD_LIBRARY_PATH=$SPARSE_ROOT/lib:$LD_LIBRARY_PATH

Navigate to the folder containing the AOCL-Sparse executable:
```
$ cd out_sparse/tests/staging/
```

Run the benchmark:

**Random Data:**

$ ./aoclsparse-bench --function=optmv --precision=d --sizem=1000
--sizen=1000 --sizennz=4000 - verify=1

**MTX Input:**

$ ./aoclsparse-bench --function=optmv --precision=d --mtx=LFAT5.mtx
--verify=1

Run the multi-threaded (4 threads) CSR-SPMV on a randomly generated matrix:

$ export AOCLSPARSE_NUM_THREADS=4

$ ./aoclsparse-bench --function=csrmv --precision=d --sizem=1000
--sizen=1000 --sizennz=4000 -- verify=1

Note

Running aoclsparse-bench without any arguments will provide overview of its parameters.

11.2.4. Performance Benchmarking on Windows#

Complete the following steps to run performance benchmarking on Windows:

Enable BUILD_CLIENTS_BENCHMARKS during AOCL-Sparse installation process (refer to Building AOCL-Sparse from Source on Linux).
Define the environment variable AOCL_ROOT to point to AOCL libs installation that has AOCL-BLAS, AOCL-LAPACK and AOCL-Utils libraries:
```
$ set AOCL_ROOT="C:\Program Files\AMD\AOCL-Windows"
```
Define SPARSE_ROOT to the AOCL-Sparse package installation path:
```
$ set SPARSE_ROOT="%HOME%\amd\aocl-sparse"
```

Add AOCL-Sparse, AOCL-BLAS, AOCL-LAPACK and AOCL-Utils library paths to the environment path variable:

$ set PATH="%SPARSE_ROOT%\lib;%PATH%"
$ set PATH="%AOCL_ROOT%\amd-blis\lib\ILP64;%PATH%"
$ set PATH="%AOCL_ROOT%\amd-libflame\lib\ILP64;%PATH%"
$ set PATH="%AOCL_ROOT%\amd-utils\lib;%PATH%"

Navigate to the folder containing the AOCL-Sparse executable:
```
$ cd out_sparse\tests\staging\
```

Run the benchmark:

**Random Data:**

$ .\aoclsparse-bench.exe --function=optmv --precision=d --sizem=1000
--sizen=1000 --sizennz=4000 -verify=1

**MTX Input:**

$ .\aoclsparse-bench.exe --function=optmv --precision=d --mtx=LFAT5.mtx
--verify=1

11.2.5. Running the Test Suite#

The AOCL-Sparse library comes with a comprehensive set of tests which you might want to run when you build the library from source (refer to Building AOCL-Sparse from Source on Linux and Building AOCL-Sparse from Source on Windows). During the configuration, enable BUILD_UNIT_TESTS, BUILD_CLIENTS_BENCHMARKS and BUILD_CLIENTS_SAMPLES. CMake will then generate all unit tests, examples, and a selection of benchmarking tests using CMake’s ctests framework. The tests can be executed from the command prompt as follows:

$ cd <build_directory>
$ ctest -VV

Refer to the ctests help on all the possible ways to trigger an individual test or a specific selection.

AOCL-Sparse - 5.0 English - 57404

AOCL User Guide (57404)

11. AOCL-Sparse#

11.1. Installation#

11.1.1. Building AOCL-Sparse from Source on Linux#

11.1.2. Building AOCL-Sparse from Source on Windows#

11.1.2.1. Using CMake and Visual Studio GUI#

11.1.2.2. Using Windows Command-Line#

11.1.3. Using Pre-Built Libraries#

11.2. Usage#

11.2.1. Use by Applications on Linux#

11.2.1.1. Link to Single-Threaded, LP64, Static Sparse Library#

11.2.1.2. Link to Multi-Threaded, ILP64, Shared Sparse Library#

11.2.2. Use by Application on Windows#

11.2.3. Performance Benchmarking on Linux#

11.2.4. Performance Benchmarking on Windows#

11.2.5. Running the Test Suite#