Establishing Interrupt Communication Using AMD IPIs - Establishing Interrupt Communication Using AMD IPIs - 2026.1 English - UG1186

IPI Overview

Versal devices use IPI as the hardware mechanism for fast, low-latency signaling between processing units. Key features include the following:

• Hardware interrupt generation between APU, RPU, and PMC
• Use two variants:
  • Buffered IPI: Includes a small message buffer of 512 bytes.
  • No-buffer IPI: Provides interrupt-only signaling and supports OpenAMP.
• Each IPI channel includes the following:
  • Control registers for trigger, status, and enable
  • An optional message buffer

IPI Configuration for OpenAMP

OpenAMP uses no-buffer IPI channels for VirtIO kick and notify operations:

• Linux (APU) side:
  • IPI mailbox: ipi0@eb330000 with IPI ID 0x2
  • Destination: ipi_1_nobuf_to_ipi_2_nobuf@eb3b1000 with IPI ID 0xb
  • Purpose: Signal the RPU when new messages become available in the vring

• Zephyr (RPU) side:
  • IPI mailbox: ipi_nobuf2@eb3b1000 with IPI ID 0xb
  • Destination: ipi_2_nobuf_to_ipi_1_nobuf@eb3b0000 with IPI ID 0xa
  • Purpose: Signal the APU when the RPU consumes messages or sends responses

IPI Flow in RPMsg Communication

The following is the message send flow from the APU to RPU:

1. Linux writes the message to the shared buffer:
   1. The application calls write() on /dev/ttyRPMSG0
   2. The RPMsg driver allocates a buffer from the shared pool.
   3. The driver copies the message data into the buffer.
2. Linux updates the VRing:
   • Linux adds the buffer descriptor to VRing0, the TX ring.
   • Linux updates the available index.
3. Linux triggers the IPI:
   • The VirtIO layer calls mailbox_notify().
   • The IPI mailbox driver writes to the IPI control registers.
   • The hardware generates an interrupt to the RPU.
4. RPU receives the IPI
   • GIC-R52 receives interrupt IRQ 0x41.
   • IPM driver ISR calls OpenAMP callback
   • OpenAMP processes VRing0 and retrieves the buffer
5. The Zephyr application processes the message:
   • The endpoint callback receives the message data.
   • The application logic handles the message.
6. Zephyr optionally sends a response:
   • It calls rpmsg_send() or rpmsg_sendto().
   • Zephyr updates VRing1, the RX ring from the Linux perspective.
   • Zephyr triggers an IPI back to the APU.
7. Linux receives the response:
   • IPI interrupt IRQ 0x39 wakes remoteproc.
   • VirtIO processes VRing1.
   • The system delivers the data to your application.

IPI Configuration Details

Assign IPI interrupt numbers for processor notifications as follows:

• On the APU (GICv3), use IRQ 0x39 (SPI 57) to receive notifications from the RPU.
• On the RPU (GIC-R52), use IRQ 0x41 (SPI 65) to receive notifications from the APU.

Zephyr IPI Implementation

The following is the platform callback from main_remote.c:

static
                  const struct device *const ipm_handle = 
                DEVICE_DT_GET(DT_CHOSEN(zephyr_ipc)); static void
                platform_ipm_callback(const struct device *dev, void *context, 
                uint32_t id, volatile void *data) {     LOG_DBG("%s: msg
                received from mb %d\n", __func__, id); 
                k_sem_give(&data_sem);  /* Wake OpenAMP processing thread */ } /* Setup IPM during
                initialization */ int platform_init(void)
                {     /* ... */
                    /* Register IPI
                callback */ 
                ipm_register_callback(ipm_handle, platform_ipm_callback, NULL);     /* Enable IPI
                interrupts */     status =
                ipm_set_enabled(ipm_handle, 1);     /* ... */
                } 

The following is the VirtIO notify function:

static int
                  mailbox_notify(void *priv, uint32_t id) {     /* Trigger IPI to
                remote processor */     ipm_send(ipm_handle,
                0, id, NULL, 0);     return 0;
                } /* Register with VirtIO
                device */ vdev =
                rproc_virtio_create_vdev(VIRTIO_DEV_DEVICE, VDEV_ID, 
                rsc_table_to_vdev(rsc_table), 
                rsc_io, NULL,  
                mailbox_notify,  /* IPI notify callback */ 
                NULL); 

IPI Performance Characteristics

Latency describes timing characteristics of IPIs.

• You can expect IPI trigger latency under 1 microsecond.
• You can expect interrupt delivery in one to five microseconds.
• Processor state affects interrupt delivery time.
• You can expect total wakeup latency under ten microseconds.

Throughput describes how many interrupts you can send per time.

• Use lightweight IPIs with no message buffer overhead.
• Trigger thousands of IPIs per second.
• RPMsg processing, not IPI, usually limits actual throughput.

Debugging IPI Issues

Run the following to check the IPI status:

# On Linux, inspect IPI mailbox state
cat /sys/kernel/debug/mailbox/*

# Check interrupt counts
cat /proc/interrupts | grep ipi

Run the following to enable IPI tracing in Linux:

# Enable mailbox subsystem debug
echo 8 > /proc/sys/kernel/printk
echo 'file drivers/mailbox/* +p' > /sys/kernel/debug/dynamic_debug/control

# Check IPI events in dmesg
dmesg -w | grep ipi

Run the following to debug the Zephyr:

/* In prj.conf, enable logging */
CONFIG_PRINTK=y
CONFIG_LOG=y
CONFIG_IPM_LOG_LEVEL_DBG=y

VRing Initialization After IPI Establishment

After IPI communication becomes active, initialize VirtIO vrings for bulk data transfer. Use vrings as circular buffer structures for actual message data transfer. Use IPI for lightweight notification, such as new data available!. Linux initializes the vrings as the host.

Zephyr uses the vrings after Linux sets them up. VRing configuration comes from two sources:

• The Linux device tree defines memory regions, addresses, and sizes.
• The Zephyr resource table defines vring parameters, buffer count, alignment, and notify ID.

Linux combines both sources during remoteproc startup to initialize the vrings.

VRing Definition Sources

The following snippet shows Linux device tree reserved-memory entries that define VRing memory regions and remoteproc memory references.

reserved-memory {
    /* VRing0 memory region */
    vdev0vring0@9860000 {
        phandle = <0x266>;
        no-map;
        reg = <0x0 0x9860000 0x0 0x4000>;  /* 16 KB */
    };

    /* VRing1 memory region */
    vdev0vring1@9864000 {
        phandle = <0x267>;
        no-map;
        reg = <0x0 0x9864000 0x0 0x4000>;  /* 16 KB */
    };
};

remoteproc@eba00000 {
    r52f@0 {
        /* Reference to vring memory regions */
        memory-region = <&rsctbl9800000>,
                        <&vdev0buffer>,
                        <&vdev0vring0>,
                        <&vdev0vring1>,
                        <&ddrboot>;
    };
};

The Linux device tree provides these values:

• Physical addresses: 0x9860000, 0x9864000
• Memory sizes: 16 KB each
• Memory type: reserved and no-map

Zephyr Resource Table: VRing Parameters

/* Generated by Zephyr with CONFIG_OPENAMP_RSC_TABLE=y */

The resource table provides these values:

• Vring data structures, with addresses set later by Linux
• Number of buffers: eight from CONFIG_OPENAMP_RSC_TABLE_NUM_RPMSG_BUFF
• Alignment requirement: 4096 bytes
• Notify IDs, which identify the vring that triggered the IPI

Host Linux VRing Initialization Sequence

The Linux kernel initializes vrings when you run echo start > /sys/class/remoteproc/remoteproc0/state:

Initialization proceeds in these steps:

/* Linux: drivers/remoteproc/remoteproc_core.c */
static int rproc_start(struct rproc *rproc, const struct firmware *fw)
{
    /* 1. Parse resource table from firmware */
    ret = rproc_parse_fw(rproc, fw);

    /* Extracts vdev and vring descriptors */

    /* 2. Allocate and initialize vrings */
    for_each_vring_in_rsc_table(rvring) {
        /* Match resource table DA to DT memory region */
        ret = rproc_alloc_vring(rproc, rvring);

        /* Initialize vring structure in memory */
        vring_init(&rvring->vring, rvring->num, rvring->va, rvring->align);

        /* This sets up descriptor table, avail ring, used ring */
    }

    /* 3. Create VirtIO device */
    ret = rproc_add_virtio_dev(rvdev, rsc_vdev->id);

    /* Registers with virtio bus, creates virtio0 */

    /* 4. Start firmware on remote processor */
    ret = rproc->ops->start(rproc);

    /* R52 released from reset, begins executing */

    /* 5. Wait for remote to signal ready */
    /* Zephyr sets status in resource table */

    /* 6. Kick remote to notify vrings are ready */
    rproc_virtio_notify(rvring, 0);

    /* Sends first IPI */
}

Perform these steps in order:

1. Parse the resource table from the firmware.
2. Extract the VirtIO device and vring descriptors.
3. Match each resource table device address to a Device Tree memory region.
4. Allocate and initialize each vring.
5. Set up the descriptor table, available ring, and used ring.
6. Create the VirtIO device.
7. Start the remote processor firmware.
8. Wait for the remote side to signal readiness.
9. Send the first IPI to notify the remote side that the vrings are ready.

The following is a Linux kernel log showing initialization:

[   10.123] remoteproc remoteproc0: powering up r52f@0
[   10.234] remoteproc remoteproc0: Booting fw image rpmsg_multi_services.elf
[   10.345] remoteproc remoteproc0: vring0: va 0xffff... da 0x9860000 size 0x4000
[   10.456] remoteproc remoteproc0: vring1: va 0xffff... da 0x9864000 size 0x4000
[   10.567] remoteproc remoteproc0: registered virtio0 (type 7)
[   10.678] remoteproc remoteproc0: remote processor r52f@0 is now up
[   10.789] virtio_rpmsg_bus virtio0: rpmsg host is online

Note the following key points:

• Linux allocates kernel virtual addresses for the physical regions.
• Linux initializes vring structures in those regions.
• Linux creates the VirtIO device.
• Linux starts the R52 processor.
• Zephyr does not initialize vrings.
• Zephyr uses the vrings that Linux already set up.

Remote Zephyr VRing Usage

Zephyr accesses the vrings that Linux already initializes.

/* From main_remote.c */
struct rpmsg_device *platform_create_rpmsg_vdev(...)
{
    struct fw_rsc_vdev_vring *vring_rsc;
    struct virtio_device *vdev;

    /* 1. Create VirtIO device pointing to resource table */
    vdev = rproc_virtio_create_vdev(VIRTIO_DEV_DEVICE, VDEV_ID,
                                    rsc_table_to_vdev(rsc_table),
                                    rsc_io, NULL,
                                    mailbox_notify,
                                    NULL);

    /* 2. Wait for Linux to finish initialization */
    rproc_virtio_wait_remote_ready(vdev);

    /* Polls resource table status set by Linux */

    /* 3. Map to existing vring0 (already initialized by Linux) */
    vring_rsc = rsc_table_get_vring0(rsc_table);
    ret = rproc_virtio_init_vring(vdev, 0, vring_rsc->notifyid,
                                  (void *)vring_rsc->da,
                                  rsc_io, vring_rsc->num, vring_rsc->align);

    /* 4. Map to existing vring1 (already initialized by Linux) */
    vring_rsc = rsc_table_get_vring1(rsc_table);
    ret = rproc_virtio_init_vring(vdev, 1, vring_rsc->notifyid,
                                  (void *)vring_rsc->da,
                                  rsc_io, vring_rsc->num, vring_rsc->align);

    /* 5. Create RPMsg device on top of vrings */
    rpmsg_virtio_init_shm_pool(&shpool, NULL, SHM_SIZE);
    ret = rpmsg_init_vdev(&rvdev, vdev, ns_cb, shm_io, &shpool);

    return rpmsg_virtio_get_rpmsg_device(&rvdev);
}

Key Takeaways

• Linux initializes vrings during remoteproc start.
• Zephyr maps to existing vrings after boot.
• The resource table bridges both sides by defining parameters for Linux.

• The Linux Device Tree defines physical memory addresses and sizes.
• The Zephyr Resource Table defines vring parameters such as num, align, and notifyid.
• Linux combines both by matching addresses and applying parameters.

• The resource table device address must match the Linux Device Tree region address.
• Both sides must agree on vring size.
• You need at least 16 KB for 8 buffers with 4096-byte alignment.
• Linux must find the resource table during firmware parsing.

• “Zephyr initializes its own vrings” is false.
• “Linux initializes vrings, and Zephyr uses them” is correct.

Debugging VRing Issues

Verify that Linux initializes the VRings:

# Check whether remoteproc found and allocated vrings
dmesg | grep vring

# Expected:
# remoteproc remoteproc0: vring0: va 0xffff... da 0x9860000 size 0x4000
# remoteproc remoteproc0: vring1: va 0xffff... da 0x9864000 size 0x4000

# Check whether the system created the virtio device
ls /sys/bus/virtio/devices/

# Expected: virtio0

# Check vring status through debugfs, if enabled
cat /sys/kernel/debug/remoteproc/remoteproc0/resource_table

You should see vring entries with the correct addresses.

# Check DT reserved memory regions
cat /proc/iomem | grep vring

09860000-09863fff : vdev0vring0@9860000
09864000-09867fff : vdev0vring1@9864000

The following is debug resource table parsing:

# Enable remoteproc debug
echo 'file remoteproc_core.c +p' > /sys/kernel/debug/dynamic_debug/control
echo 'file virtio_rpmsg_bus.c +p' > /sys/kernel/debug/dynamic_debug/control

# Start firmware
echo start > /sys/class/remoteproc/remoteproc0/state

# Check logs for vring initialization
dmesg | grep -i "vring\|vdev\|resource"