What is the difference between structured and unstructu...

Unstructured pruning results in sparse weight matrices, which fail to reduce memory bandwidth requirements on hardware optimized for dense matrix operations. In contrast, structured pruning removes entire layers, heads, rows, or columns. This produces smaller, dense tensors that are compatible with standard kernels, allowing for actual latency improvements and reduced memory bandwidth usage during inference.