Information Consistent Pruning (InCoP): How to Efficiently Search for Sparse Networks?

This repository contains the Python code for the Inf-CoP project, which introduces a novel “stopping criterion” for iterative magnitude pruning methods (IMPs) to efficiently search for sparse networks. The project focuses on reducing the number of insignificant nodes in over-parameterized deep neural networks (DNNs) while minimizing the training time required for each pruning iteration.

Link to the arxiv paper.

Pruning Methods

The code includes different pruning methods:

• LTH (Lottery Ticket Hypothesis): Prunes the network based on a specified percentile value.
• SAP (Sparsity-Informed Adaptive Pruning): Prunes the network based on signal activation and correlations.
• CIAP (Correlation Informed Adaptive Pruning): Prunes the network based on correlation information. This method’s name changed in the paper to InCoP-IF (Information Flow based InCoP).
• GIAP (Gradient Informed Adaptive Pruning): Prunes the network based on gradient information. This method’s name changed in the paper to InCoP-GF (Gradient Flow based InCoP).

Components

• Pruner: Prunes the network based on specific pruning criteria determined by the chosen method.
• Activation: Manages activation functions and gradient flows in the network.
• Similarity: Calculates similarity metrics between layers of the network.
• Network: Defines the neural network architecture and training/testing methods.
• Data: Manages data loading and preprocessing for training and testing.
• Plot Tool: Provides functions for plotting and visualizing results.
• Constants: Contains shared constants and configuration settings.

Setup

Requirements

• Python 3.x
• PyTorch
• NumPy
• tqdm
• matplotlib
• scikit-learn

Parameters

• --net_arch=vgg16: Specifies the architecture or model type to be used for the neural network. In this case, the VGG16 architecture will be employed for the network design.
• --net_dataset=MNIST: Indicates the dataset to be used for training and testing the neural network for example MNIST.
• --net_lr.01=: Denotes the learning rate value to be applied during the training process of the neural network. A learning rate of 0.01 will be utilized for updating the model parameters.
• --net_train_epochs=20: Specifies the number of training epochs to be executed during the training phase of the neural network model. The network will undergo training for 20 epochs.
• --net_warmup=10: Indicates the number of warm-up epochs to be conducted at the beginning of the training process. A warm-up period of 10 epochs will be performed prior to full training.
• --prune_method=ciap: Determines the specific pruning method to be applied to the neural network. In this case, the Correlation Information Adaptive Pruning (CIAP) method will be used for network pruning.
• --prune_p=1.0: Represents a parameter value (p) specific to the pruning method, which impacts the pruning process. The value of p is set to 1.0 in this scenario.
• --prune_q=2.0: Represents another parameter value (q) associated with the pruning method, influencing the pruning approach. The value of q is specified as 2.0 for this experiment.
• --exper_imp_total_iter=15: Indicates the total number of iterative pruning iterations to be conducted during the IMP. The network will undergo 15 iterative pruning iterations in the process.
• --exper_num_trial=3: Specifies the number of trial runs or experiments to be carried out for testing and evaluation purposes. In this case, a total of 3 trial runs will be executed as part of the experiment.

These flag descriptions provide clarity on the key parameters and settings used in the InfCoP for training, experiment types, and the number of trial runs.

Experiments

MNIST

One complete set of experiments using the MNIST dataset looks like the following. We can replace the network architecture (--net_arch) and dataset (--net_dataset) to observe the results for various types of networks across different sizes of dataset.

ciap

(p, q) = (1, 2)

sbatch pytorch.slurm ../src/prune.py –net_arch=vgg16 –net_dataset=MNIST –net_lr=.01 –net_train_epochs=20 –net_warmup=10 –prune_method=ciap –prune_p=1.0 –prune_q=2.0 –exper_imp_total_iter=15 –exper_num_trial=3

(p, q) = (.5, 1)

sbatch pytorch.slurm ../src/prune.py –net_arch=vgg16 –net_dataset=MNIST –net_lr=.01 –net_train_epochs=20 –net_warmup=10 –prune_method=ciap –prune_p=0.5 –prune_q=1.0 –exper_imp_total_iter=15 –exper_num_trial=3

giap

(p, q) = (1, 2)

sbatch pytorch.slurm ../src/prune.py –net_arch=vgg16 –net_dataset=MNIST –net_lr=.01 –net_train_epochs=20 –net_warmup=10 –prune_method=giap –prune_p=1.0 –prune_q=2.0 –exper_imp_total_iter=15 –exper_num_trial=3

(p, q) = (.5, 1)

sbatch pytorch.slurm ../src/prune.py –net_arch=vgg16 –net_dataset=MNIST –net_lr=.01 –net_train_epochs=20 –net_warmup=10 –prune_method=giap –prune_p=0.5 –prune_q=1.0 –exper_imp_total_iter=15 –exper_num_trial=3

sap

(p, q) = (1, 2)

sbatch pytorch.slurm ../src/prune.py –net_arch=vgg16 –net_dataset=MNIST –net_lr=.01 –net_train_epochs=20 –net_warmup=10 –exper_type=performance –prune_method=sap –prune_p=1.0 –prune_q=2.0 –exper_imp_total_iter=15 –exper_num_trial=3

(p, q) = (.5, 1)

sbatch pytorch.slurm ../src/prune.py –net_arch=vgg16 –net_dataset=MNIST –net_lr=.01 –net_train_epochs=20 –net_warmup=10 –prune_method=sap –prune_p=0.5 –prune_q=1.0 –exper_imp_total_iter=15 –exper_num_trial=3

lth

sbatch pytorch.slurm ../src/prune.py –net_arch=vgg16 –net_dataset=MNIST –net_lr=.01 –net_train_epochs=20 –net_warmup=10 –prune_method=lth –prune_p=1.0 –prune_q=2.0 –exper_imp_total_iter=15 –exper_num_trial=3