The TX Path consists of outgoing RDMA READ, RDMA WRITE transactions, and ACK packets for incoming RDMA SEND/WRITE requests and responses for incoming RDMA READ requests. Based on the SQPIi doorbell, the Send Work Queue requests are processed. The DMA module is configured for data transfers for all outgoing transactions. The TX data flow for RDMA WRITE/SEND is shown in This Figure .
The user application requests the ERNIC IP to transmit an RDMA WRITE/SEND/READ packet by posting a WQE on the SEND Queue for a particular QP and ringing the corresponding SQ Producer Index Doorbell (SQPI DB). The ERNIC processes the WQE and pulls data for RDMA WRITE/SEND commands based on the information provided in the WQE. This data along with relevant headers is pushed out on the link. Once an acknowledgment is received from the remote host, the ERNIC informs the user application of the successful completion of the WQE by posting a CQE, based on the configuration of QPCONFi[5], and by posting the completion count through the side band interface. The CQHEAD for the corresponding QP is also updated. For RDMA READ requests, the RX packet handler intimates the TX path. For this communication, an outstanding read request queue is available for each QP. The depth of the queue is determined by a parameter for incoming request resources. When the outstanding read request queue is full, it is indicated to the RX path. Any further requests to the QP will result in an NAK-Invalid and the QP is moved to fatal state.
Note: The direction of arrows shows the flow of data.
This Figure shows the flow of RDMA READ requests to the remote host from the ERNIC.
This Figure shows the flow for RDMA READ requests to the remote host from the ERNIC. The user application requests the ERNIC IP to transmit an RDMA READ request to the remote host by posting a WQE on the SEND Queue for a particular QP and ringing the corresponding SQ Producer Index Doorbell (SQPI DB). ERNIC processes the WQE and creates the request packet with relevant headers and pushes it out on the link. Once the response data packets are received from the remote host, ERNIC writes the data (after removing the headers) to the local buffer address provided in the WQE. It then informs the user application of the successful completion of the WQE by posting a CQE, based on the configuration of QPCONFi[5], and by posting the completion count through the side band interface. The CQHEAD for the corresponding QP is also updated.