HLS - 2024.1 English

Table of Contents

Building the Design

Hardware Design Details

Software Design Details

Performance Details

Building the Design

Make Steps

To run the following make steps (for example, make kernels, make xsa, make application, and make package), you must be in the gemm_dsp58/ folder. The following options can be specified in the make steps. Instructions for how to apply them are provided later in this section.

TARGET: This option can be set to hw or hw_emu to build the design in the hardware or hardware emulation flow. The default is hw_emu.

GEMM_SIZE: This option can be set to 32, 64, 128, 256, 512 or 1024

The Makefile uses the following directory references:

## Relative directory
RELATIVE_PROJECT_DIR := ./
PROJECT_REPO := $(shell readlink -f $(RELATIVE_PROJECT_DIR))
DESIGN_REPO  := $(PROJECT_REPO)/design
PL_SRC_REPO  := $(DESIGN_REPO)/pl_src
CONSTRAINTS_REPO  := $(PL_SRC_REPO)/constraints
HOST_APP_SRC := $(DESIGN_REPO)/host_app_src
SYSTEM_CONFIGS_REPO    := $(DESIGN_REPO)/system_configs
VIVADO_METRICS_SCRIPTS_REPO := $(DESIGN_REPO)/vivado_metrics_scripts

BASE_BLD_DIR := $(PROJECT_REPO)/build_$(PL_FREQ)
GEMM_BLD_DIR     := $(BASE_BLD_DIR)/gemm_$(MAT_DIMS)
BUILD_TARGET_DIR := $(GEMM_BLD_DIR)/$(TARGET)

VIVADO_REPORTS_REPO := $(PROJECT_REPO)/vivado_reports_dir
BLD_VIVADO_REPORTS_DIR := $(VIVADO_REPORTS_REPO)/gemm_$(MAT_DIMS)

EMBEDDED_PACKAGE_OUT := $(BUILD_TARGET_DIR)/package
EMBEDDED_EXEC_SCRIPT := run_script.sh

Build the Entire Design with a Single Command

If you are already familiar with Vitis kernel compilation flows, you can build the entire design with one command:

make run (default TARGET=hw_emu, GEMM_SIZE=64) 

or

make run TARGET=hw (Target is hardware, GEMM_SIZE=64)

This command runs the make kernels, make xsa, make application, make package, and make run_emu steps for hardware emulation or to run on hardware (VCK190 board) depending on the TARGET you specify. The settings also apply to individual make steps listed below.

The generated files are placed under an individual directory: $(BUILD_TARGET_DIR)/. Each make step to build the design is specified in the following sections. These sections also detail the options used and the location of input and output files in each case.

${PL_SRC_REPO}/rtl/BDELAY.vhd
${PL_SRC_REPO}/rtl/FIXGEMM.vhd
${PL_SRC_REPO}/rtl/SDELAY.vhd
${PL_SRC_REPO}/rtl/sfixed_pkg.vhd
${PL_SRC_REPO}/rtl/cfixed_pkg.vhd
${PL_SRC_REPO}/rtl/DSP_GW.vhd
${PL_SRC_REPO}/rtl/FIXGEMM_WRAPPER.vhd
${PL_SRC_REPO}/rtl/control_logic.sv
${PL_SRC_REPO}/rtl/gemm_top.sv
${PL_SRC_REPO}/rtl/ps_slave.sv
${PL_SRC_REPO}/rtl/DSP_data_controller.sv
${PL_SRC_REPO}/rtl/op_uram.sv
${PL_SRC_REPO}/rtl/row_uram.sv
${PL_SRC_REPO}/rtl/col_uram.sv
${PL_SRC_REPO}/rtl/gemm_large_ocm.sv
${PL_SRC_REPO}/rtl/partial_sum_bram.sv
${PL_SRC_REPO}/rtl/synchronizer.sv

$(PROJECT_REPO)/build/gemm_GEMM_SIZExGEMM_SIZExGEMM_SIZE/gemm_large_ocm.xo

make kernels: Generates the PL Kernels

This step uses the RTL and mem_init_files specified above to generate the PL kernel (gemm_large_ocm.xo)

make xsa: Using the Vitis Tools to Link HLS Kernels with the Platform

After the kernel is generated, you can use the Vitis compiler to link it with the platform to generate an XSA file.

The Vitis tools allow you to integrate the kernels into an existing extensible platform. This is an automated step from a software developer perspective where the platform chosen is provided by the hardware designer (or you can opt to use one of the many extensible base platforms provided by Xilinx and the Vitis tools build the hardware design and integrate the kernels into the design).

The command to run this step is shown as follows

make xsa TARGET=<hw/hw_emu> GEMM_SIZE=<64,128,256,512,1024>

The expanded command is as follows:

cd $(BUILD_TARGET_DIR);	\

v++ -l --platform xilinx_vck190_base_202410_1 --save-temps --temp_dir $(BUILD_TARGET_DIR)/_x \
   --verbose -g --clock.freqHz 500000000:gemm_large_ocm_0 --clock.defaultTolerance 0.001 \
   --config $(SYSTEM_CONFIGS_REPO)/gemm.cfg --vivado.prop fileset.sim_1.xsim.simulate.log_all_signals=true \
   --vivado.prop run.synth_1.{STEPS.SYNTH_DESIGN.ARGS.CONTROL_SET_OPT_THRESHOLD}={16} \
   --vivado.prop run.synth_1.{STEPS.SYNTH_DESIGN.ARGS.KEEP_EQUIVALENT_REGISTERS}={true} \
   --xp vivado_prop:run.impl_1.STEPS.PLACE_DESIGN.TCL.PRE=$(CONSTRAINTS_REPO)/gemm_dsp58.tcl
   -t hw_emu -o $(BUILD_TARGET_DIR)/gemm.hw_emu.xclbin $(PROJECT_REPO)/build/gemm_GEMM_SIZExGEMM_SIZExGEMM_SIZE/gemm_large_ocm.xo

See this page for a detailed description of Vitis linking options. The following table provides a summary of the switches used.

The information to tell the linker how to connect the PL kernels together is described in a configuration file, system_configs/gemm.cfg. The file describes the overall connection scheme of the system.

[connectivity]
nk=gemm_large_ocm:1:gemm_large_ocm_0

[clock]
id=0:gemm_large_ocm_0.S_AXI_ACLK

[advanced]
## Disable Profiling in hw_emu so that it is faster...
param=hw_emu.enableProfiling=false
## Export the xsa of the design..
param=compiler.addOutputTypes=hw_export
param=compiler.worstNegativeSlack=-1.0
[vivado]
prop=run.synth_1.STRATEGY=Flow_PerfOptimized_high
prop=run.impl_1.STEPS.OPT_DESIGN.is_enabled=true
prop=run.impl_1.STEPS.OPT_DESIGN.ARGS.DIRECTIVE=Explore
prop=run.impl_1.STEPS.PLACE_DESIGN.ARGS.DIRECTIVE=ExtraTimingOpt
prop=run.impl_1.STEPS.PHYS_OPT_DESIGN.is_enabled=true
prop=run.impl_1.STEPS.PHYS_OPT_DESIGN.ARGS.DIRECTIVE=AggressiveExplore
#prop=run.impl_1.STEPS.ROUTE_DESIGN.ARGS.MORE OPTIONS=-tns_cleanup
prop=run.impl_1.STEPS.ROUTE_DESIGN.ARGS.DIRECTIVE=AggressiveExplore

See this page for a detailed description of the Vitis compiler configuration file. A summary of the configuration options used is provided in the following table.

The Vitis™ compiler calls the Vivado™ IP integrator under the hood to build the design. The platform and kernels are input to the Vivado Design Suite, which produces a simulation XSA or an XSA after running place and route on the design. The point at which the XSA is produced from Vivado depends on the -target option set on the Vitis compiler command line.

You can now view the Vivado project, which is located in the $(BUILD_TARGET_DIR)/_x/link/vivado/vpl/prj directory. You have now generated the XCLBIN file, $(BUILD_TARGET_DIR)/gemm.hw_emu.xclbin, that is used to execute your design on the platform.

make application: Compile the Host Application

You can compile the host application by following the typical cross-compilation flow for the Cortex A72 processor. To build the application, run the following command

make application 

or

cd $(BUILD_TARGET_DIR);	\

aarch64-xilinx-linux-g++ -mcpu=cortex-a72.cortex-a53 -march=armv8-a+crc -fstack-protector-strong \
   -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=$(SDKTARGETSYSROOT) -O -c \
   -std=c++14 -D__linux__ \
   -DM_LARGE=$(GEMM_SIZE) -DN_LARGE=$(GEMM_SIZE) -DL_LARGE=$(GEMM_SIZE) \
   -I$(SDKTARGETSYSROOT)/usr/include/xrt -I$(SDKTARGETSYSROOT)/usr/include -I$(SDKTARGETSYSROOT)/usr/lib -I$(HOST_APP_SRC)/$(MAT_DIMS) \
$(HOST_APP_SRC)/main.cpp -o $(BUILD_TARGET_DIR)/gemm_top_app.o \
   -L$(SDKTARGETSYSROOT)/lib -lxrt_coreutil

aarch64-xilinx-linux-g++  -mcpu=cortex-a72.cortex-a53 -march=armv8-a+crc -fstack-protector-strong \
   -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=$(SDKTARGETSYSROOT) \
   $(BUILD_TARGET_DIR)/gemm_top_app.o -L$(SDKTARGETSYSROOT)/usr/lib -lxrt_coreutil \
   -o $(BUILD_TARGET_DIR)/gemm_dsp_xrt.elf

See this page for XRT documentation. See this page for details of host application programming.

The following is a description of the input sources compiled by the cross-compiler compiler command.

The following is a description of the output objects that results from executing the cross-compiler command with the above inputs and options.

make package: Packaging the Design

With the Kernel outputs created, as well as the new platform, you can now generate the programmable device image (PDI) and a package to be used on an SD card. The PDI contains all the executables, bitstreams, and configurations of the device. The packaged SD card directory contains everything to boot Linux, the generated applications, and the XCLBIN.

The command to run this step is as follows (default TARGET=hw_emu):

make package

or

cp $(PROJECT_REPO)/run_script.sh $(BUILD_TARGET_DIR)/
cd $(BUILD_TARGET_DIR);	\

v++ -p -t hw --save-temps --temp_dir $(BUILD_TARGET_DIR)/_x -f xilinx_vck190_base_202410_1 \
   --package.rootfs $(XLNX_VERSAL)/rootfs.ext4 --package.kernel_image $(XLNX_VERSAL)/Image --package.boot_mode=sd \
   --package.out_dir $(BUILD_TARGET_DIR)/package --package.image_format=ext4 --package.sd_file $(BUILD_TARGET_DIR)/gemm_dsp_xrt.elf \
   $(BUILD_TARGET_DIR)/gemm.hw.xclbin

If the XRT_ROOT is set, the following Vitis compiler flags are also set:

   --package.sd_dir $(XRT_ROOT)

See this page for more details about packaging the system.

The output of the V++ Package step is the package directory that contains the contents to run hardware emulation.

make run_emu: Running Hardware Emulation

After packaging, everything is set to run hardware emulation. To run emulation, use the following command (default TARGET=hw_emu):

make run_emu 

or

###########################################################################
Hardware Emulation Goto:
$(BUILD_TARGET_DIR)/package

and do:
./launch_hw_emu.sh or ./launch_hw_emu.sh -g (for waveform viewer)...

When hardware emulation is launched, you see the QEMU simulator load. Wait for the autoboot countdown to go to zero. After a few minutes, the root Linux prompt comes up:

root@versal-rootfs-common-2024.1:~#

After the root prompt comes up, run the following commands to run the design:

cd /mnt
export XILINX_XRT=/usr
./gemm_dsp_xrt.elf a.xclbin

The gemm_dsp_xrt.elf executes. After a few minutes, you should see the output with TEST PASSED on the console. When this is shown, run the following keyboard command to exit the QEMU instance:

#To exit QEMU Simulation
Press CtrlA, let go of the keyboard, and then press x 

To run with waveform, do the following:

cd $(BUILD_TARGET_DIR)/package
./launch_hw_emu.sh -g

The XSIM Waveform Viewer is launched. Drag and drop the signals into the viewer and click Play to start the emulation. Go back to the terminal and wait for the Linux prompt to show up. In the XSIM Waveform Viewer, you will see the signals you added to the waveform adjusting over the execution of the design. When this is done, hit the pause button and close the window to end the emulation.Data Integrity mismatch due to software issue in Hardware Emulation,Design works in Hardware run.

Running on Hardware

To run the design on hardware, rerun the following make steps with TARGET=hw and other applicable options (see the preceding make steps specified above).

make kernels TARGET=hw 
make xsa TARGET=hw 
make application TARGET=hw
make package TARGET=hw 

These commands create a $(BUILD_TARGET_DIR) folder with the kernels, xsa, and package for a hardware run.

Run the following step to set up the execution file, generated images, and base images ($(BUILD_TARGET_DIR)/package/sd_card and $(BUILD_TARGET_DIR)/package/sd_card.img).

make run_emu TARGET=hw 

These commands create a build/hw folder with the kernels, XCLBIN, and package for a hardware run. Follow steps 1-9 to run the gemm_dsp_xrt.elf executable on your VCK190 board.

Step 1. Ensure your board is powered off.

Step 2. Use an SD card writer (such as balenaEtcher) to flash the sd_card.img file to an SD card.

Step 3. Plug the flashed SD card into the top slot of the VCK190 board.

Step 4. Set the switch (SW1 Mode\[3:0\]=1110 = OFF OFF OFF ON).

Step 5. Connect your computer to the VCK190 board using the USB cable included with the board.

Step 6. Open a TeraTerm terminal and select the correct COM port. Set the port settings to the following:

Port: <COMMXX>
Speed: 115200
Data: 8 bit
Parity: none
Stop Bits: 1 bit
Flow control: none
Transmit delay: 0 msec/char 0 msec/line

Step 7. Power on the board.

Step 8. Wait until you see the root@versal-rootfs-common-2024_1 Linux command prompt. Press enter a few times to get past any xinit errors.

Step 9. Run the following commands in the TeraTerm terminal:

mount /dev/mmcblk0p1 /mnt
cd /mnt
export XILINX_XRT=/usr

./gemm_dsp_xrt.elf a.xclbin