Input of predict function are D streams of features, in which D is how many features that predict function process at one cycle. According to prediction model formula, to calculate final result, it has dependency issue. To achieve II = 1 to process input data, prediction comes in three stages: - Stage 1: Compute the sum of D features multiply D weights, and name it as partSum. Later D features’ processing does not depend on former D features. It could achieve II = 1. - Stage 2: Compute the sum of partSum; you allocate a buffer whose length(L) is longer than latency of addition, and add each partSum to different position. 0th partSum will be added to 0th buff, 1st to 1st buff… The L th part Sum will be added to 0th buff. Because L is longer than the addition latency, the 0th buff has already finished the addition of 0th buffer. So Lth partSum’s addition will not suffer a dependency issue,so Stage 2 could Archieve II = 1. - Stage 3: Add L buffs, and get the final result. This part also does not have a dependency issue and could achieve II = 1. Stage 1 to 3 are connected with streams and run dataflow. In this way, you could achieve II = 1 in total.
The correctness of Linear Regression/LASSO Regression/Ridge Regression is verified by comparing results with Spark mllib. The results are identical.