In a virtualized environment, the driver attached to a PF has enough privilege to program and access QDMA registers. For all the lesser privileged functions, certain PFs and all VFs must communicate with privileged drivers using the mailbox mechanism. The communication API must be defined by the driver. The QDMA IP does not define it.
Each function (both PF and VF) has an inbox and an outbox that can fit a message size of 128B. A VF accesses its own mailbox, and a PF accesses its own mailbox and all the functions (PF or VF) associated with that PF.
The QDMA mailbox allows the following access:
- From a VF to the associated PF.
- From a PF to any VF belonging to its own virtual function group (VFG).
- From a PF (typically a driver that does not have access to QDMA registers) to another PF.
VF To PF Messaging
A VF is allowed to post one message to a target PF mailbox until the target
function (PF) accepts it. Before posting the message the source function should make
sure its o_msg_status is cleared, then the VF can
write the message to its Outgoing Message Registers. After finishing message
writing, the VF driver sends msg_send command
through write 0x1 at the control/status register (CSR) address 0x5004. The mailbox
hardware then informs the PF driver by asserting i_msg_status field.
The function driver should enable the periodic polling of the i_msg_status to check the availability of incoming
messages. At a PF side, i_msg_status = 0x1
indicates one or more message is pending for the PF driver to pick up. The cur_src_fn in the Mailbox Status Register gives the
function ID of the first pending message. The PF driver should then set the Mailbox
Target Function Register to the source function ID of the first pending message.
Then access to a PF’s Incoming Message Registers is indirectly, which means the
mailbox hardware will always return the corresponding message bytes sent by the
Target function. Upon finishing the message reading, the PF driver should also send
msg_rcv command through write 0x2 at the CSR
address. The hardware will deassert the o_msg_status at the source function side. The following figure
illustrates the messaging flow from a VF to a PF at both the source and destination
sides.
PF To VF Messaging
The messaging flow from a PF to the VFs that belong to its VFG is slightly different than the VF to PF flow because:
A PF can send messages to multiple destination functions, therefore, it might
receive multiple acknowledgments at the moment when checking the status. As
illustrated in the following figure, a PF driver must set Mailbox Target Function
Register to the destination function ID before doing any message operation; for
example, checking the incoming message status, write message, or send the command.
At the VF side (receiving side), whenever a VF driver get the i_msg_status = 0x1, the VF driver should read its
Incoming Message Registers to pick up the message. Depending on the application, the
VF driver can send the msg_rcv immediately after
reading the message or after the corresponding message being processed.
To avoid one-by-one polling of the status of outgoing messages, the
mailbox hardware provides a set of Acknowledge Status Registers (ASR) for each PF.
Upon the mailbox receiving the msg_rcv command from
a VF, it deasserts the o_msg_status field of the
source PF and it also sets the corresponding bit in the Acknowledge Status
Registers. For a given VF with function ID <N>, acknowledge status is at:
- Acknowledge Status Register address: <N> / 32 + <0x22420 Register Address>
- Acknowledge Status bit location: <N> / 32
The mailbox hardware asserts the ack_status
filed in the Status Register (0x22400) when there is any bit was asserted in the
Acknowledge Status Register (ASR). The PF driver can poll the ack_status before actually reading out the Acknowledge
status registers. The PF driver might detect multiple completions through one
register access. After being processed, the PF driver should also write the value
back to the same register address to clear the status.
Mailbox Interrupts
The mailbox module supports interrupt as the alternative event notification mechanism. Each mailbox has an Interrupt Control Register (at the offset 0x22410 for a PF, or at the offset 0x5010 for a VF). Set 1 to this register to enable the interrupt. Once the interrupt is enabled, the mailbox will send the interrupt to the QDMA given there is any pending event for the mailbox to process, namely, any incoming message pending or any acknowledgment for the outgoing messages. Configure the interrupt vector through the Function Interrupt Vector Register (0x22408 for a PF, or 0x5008 for a VF) according to the driver configuration.
Enabling the interrupt does not change the event logging mechanism, which means the user must check the pending events through reading the Function Status Registers. The first step to respond to an interrupt request is disabling the interrupt. It is possible that the actual number of the pending events is more than the number of the events at the moment when the mailbox is sent the interrupt.
The mailbox will check its event status at the time the interrupt control change from disabled to enabled. If there is any new events that arrived the mailbox between reading the interrupt status and the re-enabling the interrupt, the mailbox will generate a new interrupt request immediately.