One-step pruning implements the EagleEye 1 algorithm. It introduces a strong positive correlation between different pruned models and their corresponding fine-tuned accuracy by a simple, yet efficient, evaluation component called adaptive batch normalization. It enables you to get the subnetwork with the highest potential accuracy without actually fine-tuning the models. In short, the one-step pruning method searches for a bunch of subnetworks (i.e., generated pruned models) that meet the required model size, and selects the most promising one. The selected subnetwork is then retrained to recover the accuracy.
The pruning steps are as follows:
- Search for subnetworks that meet the required pruning ratio.
- Select a potential network from a bunch of subnetworks with an evaluation component.
- Fine-tune the pruned model.
Figure 1. One-step Pruning Workflow
Note:
- Bailin Li et al., EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning, arXiv:2007.02491