Performing Model Analysis - Performing Model Analysis - 1.1 English

Before conducting model pruning, you need to analyze the model first. The main purpose of this process is to find a suitable pruning strategy, which will be used later to prune the model.

To run model analysis, you need to provide a python script containing the functions that evaluate model performance. Assume your script is eval_model.py and you need to provide the required functions in one of three ways:

A function named model_fn() that returns a python dict of metric ops:

def model_fn():
  tf.logging.set_verbosity(tf.logging.INFO)
  img, labels = get_one_shot_test_data(TEST_BATCH)

  logits = net_fn(img, is_training=False)
  predictions = tf.argmax(logits, 1)
  labels = tf.argmax(labels, 1)
  eval_metric_ops = {
      'accuracy': tf.metrics.accuracy(labels, predictions),
      'recall_5': tf.metrics.recall_at_k(labels, logits, 5)
  }
  return eval_metric_ops

A function named model_fn() that returns an instance of tf.estimator.Estimator and a function named eval_input_fn() that feeds test data to estimator:

def model_fn():
  return tf.estimator.Estimator(
      model_fn=cnn_model_fn, model_dir="./models/train/")

def eval_input_fn():
  return tf.estimator.inputs.numpy_input_fn(
      x={"x": eval_data},
      y=eval_labels,
      num_epochs=1,
      shuffle=False)

A function named evaluate() taking a single parameter as argument that returns the metric score:

def evaluate(checkpoint_path):
  with tf.Graph().as_default():
    net = ConvNet(False)
    net.build(test_only=True)
    score = net.evaluate(checkpoint_path)
    return score

Suppose you use the first way to write the script, the following snippet shows how to call vai_p_tensorflow to perform model analysis.

vai_p_tensorflow \
  --action=ana \
  --input_graph=inference_graph.pbtxt \
  --input_ckpt=model.ckpt \
  --eval_fn_path=eval_model.py \
  --target="recall_5" \
  --max_num_batches=500 \
  --workspace:/tmp \
  --exclude="conv node names that excluded from pruning" \
  --output_nodes="output node names of the network"

The arguments in this command are explained below. See vai_p_tensorflow Usage for a full list of options.

--action: Which action to do.
--input_graph: A GraphDef proto file that represents the inference graph of the network.
--input_ckpt: The path to a checkpoint to use for pruning.
--eval_fn_path: The path to a python script defining an evaluation graph.
--target: The target score that evaluates the performance of the network. If there is more than one score in the network, you should choose one of the most important or your most concerned.
--max_num_batches: The number of batches to run in evaluation phase. This parameter will affect the time it takes to analyze model. In general, the larger this value, the more time required for the analysis, but the more accurate the analysis is. The maximum value of this parameter is size of validation set / batch_size, that is, all data in the validation set will be used for evaluation.
--workspace: Directory for saving output files.
--exclude: Convolution nodes excluded from pruning.
--output_nodes: Output nodes of the inference graph.

Performing Model Analysis - Performing Model Analysis - 1.1 English - UG1333

AI Optimizer User Guide (UG1333)