An Apache Arrow format data can be represented in the illustrated figure on the left side. The whole data is seperated into multiple record batches; each batch consists of multiple columns with the same length.
It is worth mentioning that the length of each record batch is a statistic info, unknown while reading/writing each record batch data. Besides, the data width of different data types are different, especially for string, because the length of each string data is variable.
Thus, the Apache Arrow columnar data format cannot be implemented directly on hardware. A straight-forward implementation of the arrow data would be, for each field id, one fixed size DDR buffer is predefined. However, because the number and data type of each field is unknown, DDR space is wasted heavily. To fully utilize the DDR memory on the FPGA, the “data-frame” format is defined and employed, which can be seen in the right side of the preceding figure.
The DDR is split into multiple mem blocks. Each block is 4 MB in size with a 64-bit width. The mem block address and linking info is recored on the meta section of DDR header. In other words, for each column/field, the data is stored in 4M -> 4M -> 4M linkable mem blocks. The length, size, count, etc. info are also saved in the DDR header.
Three types of data are columnar stored differently comparing to the Apache Arrow format, namely, Null, Boolean, and String. For Null and Boolean, because to only 1-bit is required for each data, bitmap[4096][16] and boolbuff[4096][16] (each data 64-bit) is used to save the data, respectively. The following figure illustrates the bitmap layout; each 64-bit data indicates 64 x input data, and the maximum supported number of input data number of 64 x 4096. And the supported maximum field num is 16. The same data storage buffer is employed for Boolbuff.