Improving Performance–Efficiency Trade-offs Through Model Compression and Optimization Strategies
DOI:
https://doi.org/10.64882/ijrt.v14.i3.1755Keywords:
model compression, pruning, quantization, knowledge distillation, deep learning efficiency, edge inferenceAbstract
The rapid growth of deep neural networks has produced models that deliver state-of-the-art accuracy while imposing substantial computational, memory, and energy costs, which limits their deployment on resource-constrained platforms such as mobile devices, embedded systems, and edge accelerators. This paper investigates model compression and optimization strategies, namely magnitude-based and structured pruning, post-training and quantization-aware training, and response-based knowledge distillation, as mechanisms for improving the performance-efficiency trade-off of convolutional neural networks. A unified experimental framework is proposed in which a ResNet-based baseline model is compressed using each technique independently and in hybrid combination, and the resulting models are evaluated on accuracy, model size, floating-point operations (FLOPs), inference latency, and energy consumption. Experimental results, summarized across six comparative tables and four analytical figures, indicate that structured pruning at moderate sparsity levels (30-50%) preserves accuracy within 1-2 percentage points of the baseline while reducing model size by up to 2.6x, that 8-bit quantization achieves near-lossless compression with a 4x reduction in memory footprint, and that a hybrid pipeline combining pruning, quantization, and distillation achieves an 8.5x compression ratio with an accuracy degradation of under 5%. These findings confirm that combining complementary compression strategies yields substantially better performance-efficiency trade-offs than any single technique applied in isolation. The paper concludes with a discussion of open challenges, including hardware-aware compression, automated compression policy search, and compression of large-scale transformer architectures, and outlines directions for future research.
References
Blalock, D., Gonzalez Ortiz, J. J., Frankle, J., & Guttag, J. (2020). What is the state of neural network pruning? Proceedings of Machine Learning and Systems, 2, 129-146.
Cheng, Y., Wang, D., Zhou, P., & Zhang, T. (2018). Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine, 35(1), 126-136.
Courbariaux, M., Bengio, Y., & David, J. P. (2015). BinaryConnect: Training deep neural networks with binary weights during propagations. Advances in Neural Information Processing Systems, 28, 3123-3131.
Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022). LLM.int8(): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35, 30318-30332.
Frankle, J., & Carbin, M. (2019). The lottery ticket hypothesis: Finding sparse, trainable neural networks. Proceedings of the International Conference on Learning Representations.
Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2023). GPTQ: Accurate post-training quantization for generative pre-trained transformers. Proceedings of the International Conference on Learning Representations.
Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021). Knowledge distillation: A survey. International Journal of Computer Vision, 129(6), 1789-1819.
Han, S., Mao, H., & Dally, W. J. (2015). Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. arXiv preprint arXiv:1510.00149.
He, Y., Zhang, X., & Sun, J. (2017). Channel pruning for accelerating very deep neural networks. Proceedings of the IEEE International Conference on Computer Vision, 1389-1397.
Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017). MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., & Kalenichenko, D. (2018). Quantization and training of neural networks for efficient integer-arithmetic-only inference. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2704-2713.
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017). Learning efficient convolutional networks through network slimming. Proceedings of the IEEE International Conference on Computer Vision, 2736-2744.
Liu, Z., Sun, M., Zhou, T., Huang, G., & Darrell, T. (2019). Rethinking the value of network pruning. Proceedings of the International Conference on Learning Representations.
Molchanov, P., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2017). Pruning convolutional neural networks for resource efficient inference. Proceedings of the International Conference on Learning Representations.
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016). XNOR-Net: ImageNet classification using binary convolutional neural networks. Proceedings of the European Conference on Computer Vision, 525-542.
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L. C. (2018). MobileNetV2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4510-4520.
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Tan, M., & Le, Q. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. Proceedings of the 36th International Conference on Machine Learning, 97, 6105-6114.
Wu, J., Leng, C., Wang, Y., Hu, Q., & Cheng, J. (2016). Quantized convolutional neural networks for mobile devices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4820-4828.
Downloads
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
