Latency of a buffer-based TDM FIR is predominantly due to the buffering in the input and output buffers. Other factors which affect latency are data and coefficient types and FIR length, though these tend to have a lesser effect.
To minimize the latency, the buffer size should be set to the minimum size that meets the required throughput.