Model Accuracy Test - 2.0 English

To test the model accuracy on the board, you must take into account the following factors.

• Image dataset
• Model
• Accuracy test program
• Ground truth file of the image dataset
• Accuracy extraction comparison script

ResNet50 Example

Take resnet50 as an example.

1. Get the image dataset and the ground truth file of the dataset.

   You can get the image dataset information from https://github.com/Xilinx/Vitis-AI/tree/master/models/AI-Model-Zoo. resnet50 uses the imagenet dataset.

2. Get the model.

   wget https://www.xilinx.com/bin/public/openDownload?filename=resnet50-zcu102_zcu104_kv260-r2.0.0.tar.gz -O resnet50-zcu102_zcu104_kv260-r2.0.0.tar.gz

3. Copy the model to the board.

   scp resnet50-zcu102_zcu104_kv260-r2.0.0.tar.gz root@IP_OF_BOARD:~/

   It includes resnet50 and resnet50_acc models.

4. Untar it on the board.

   tar -xzvf resnet50-zcu102_zcu104_kv260-r2.0.0.tar.gz -C /usr/share/vitis-ai-library/models

5. Set the path and image name of the image dataset to a file, such as image.list.txt.
   
   
6. Run the accuracy test program on the board.

   cd ~/Vitis-AI/demo/Vitis-AI-Library/samples/classification/
   ./test_accuracy_classification_mt resnet50 image.list.txt resnet50.image.list.result

   Note: The accuracy test loads the resnet50_acc model besides the resnet50 model. The only difference between the two models is the model prototxt file.

   After the accuracy test program running finished, the result file resnet50.image.list.result will be generated.

7. Copy the resnet50.image.list.result to the host.
8. Run the corresponding accuracy extraction comparison script to get the final accuracy.

   python evaluation.py image.list.gt resnet50.image.list.result

   
   
   

   The following is the code of evaluation.py

   
   #!/usr/bin/env python
   
   
   import sys
   # argv[1] must groundtruth
   readfile=open(sys.argv[1], 'r')
   readfile1=open(sys.argv[2], 'r')
   
   
   dic_val={}
   
   
   m = 0
   for line in readfile:
       temp = line.strip('/').split()
       key = temp[0]
       value = int(temp[1])
       dic_val[key] = value
       m = m + 1
   
   
   n = 0
   for line1 in readfile1:
       temp = line1.strip('/').split()
       if temp[0] in dic_val and int(temp[1]) == dic_val[temp[0]]:
           # print int(temp[1]), dic_val[temp[0]]
           n = n + 1
   
   
   #print m
   #print n
   readfile1.close()
   readfile2=open(sys.argv[2], 'r')
   rate = float(n)/float(m)
   print("accuracy of top-5: ", rate)
   
   
   l = 0
   a = 0
   for line2 in readfile2:
       a = a + 1
       if (a%5 != 1) : continue
       temp = line2.strip('/').split()
       if temp[0] in dic_val and int(temp[1]) == dic_val[temp[0]]:
           l = l + 1
   rate1 = float(l)/float(m)
   

   Note: For the accuracy extraction comparison script, refer to https://github.com/Xilinx/Vitis-AI/tree/master/models/AI-Model-Zoo.