An engineer proposes replacing one FFN layer with a ban...

The primary concern is that depth-wise convolutions have an arithmetic intensity of approximately k/2, which is well below the hardware ridge point. While the replacement might reduce the total number of FLOPs, those FLOPs are executed on general ALUs at a fraction of the speed of tensor cores. Because the operation is memory-bound, the system will spend most of its time waiting for data from HBM, likely resulting in slower overall performance compared to the compute-bound matmul it replaced.