You cannot directly apply this single-channel-mixer in the context of the TDM mixer. In the TDM mixer, samples from different channels arrive in an interleaved manner, while samples in the single-channel case arrive consecutively. This creates a data flow problem. You must convert the interleaved stream into a channel-by-channel stream at the mixer input, then restore the interleaved format at the mixer output.
This operation is sometimes called a corner-turn or 2D transpose, based on the following figure. Samples arrive row-wise, interleaved by channel, one sample at a time. The diagram shows four channels and eight samples for each channel. Feed the single-channel mixer with samples chosen column-wise: first eight samples from channel 0, then eight from channel 1, and so on. The minimum number of samples must be eight to match the vectorization. In practice, the column depth is a multiple of eight.