The TGP kernel design is show in the following figure:
The functions of each module in the figure is:
- preProcess: Delete special characters and use 6-bit encoding characters. The encoding rules are: 0~9 <-> ‘0’~‘9’, 10~35 <-> ‘a/A’~’z/Z’, 36 -> ‘ ‘ & ‘n’.
- twoGram: Split field to terms according to 2-Gram, and one term is two characters.
- insertSort: Sort terms using insertion sort algorithm.
- getIDFTFAddr: Merge the same term, and use term as the address to get term frequency (TF) value and the address of inverted document frequency (IDF) value in block RAM. Both TF vlaue and IDF value are provided by the inverted index.
- weighted Union: The module is a key design, and its detailed information is shown in the following figure.
The function of each module of weighted Union in the figure is:
- distData: According to the order of term, distribute the TF value to the corresponding channel. The core design is to distribute data to each channel cyclically through the write_nb operation and the subsequent mBackPressure module.
- mBackPressure: Cooperate with the distData module to realize back pressure control.
- mergeTree: Combine CH channels into one, and its bottom layer is implemented by mergeSum. The function of mergeSum is to add values with the same ID.
- byValue: Get the first index ID that exceeds the threshold.
For more details, refer to the source code.