What is Instance Segmentation?
Instance segmentation is a computer vision task that gives each object in an image its own class label and pixel mask.
A model returns one mask per object. Two objects of the same class get two separate masks. Mask R-CNN, YOLACT and Mask2Former are common models for this task.
How it differs from other tasks
- Object detection returns a bounding box and a class for each object. It returns no mask.
- Semantic segmentation returns a class for each pixel. It does not separate objects of the same class.
- Instance segmentation combines the two. The Mask R-CNN authors describe it as both a semantic task and a form of detection.
Common models
- Mask R-CNN (2017) adds a mask branch to the box branch of Faster R-CNN. The ResNet-101-FPN version reaches 35.7 mask AP on COCO test-dev at 5 frames per second.
- YOLACT (2019) is a fully convolutional model. It reaches 29.8 mAP on COCO at 33.5 frames per second on one Titan Xp GPU.
- Mask2Former (2021) is one transformer model for panoptic, instance and semantic segmentation. It reaches 50.1 AP on COCO instance segmentation.
These figures come from different papers and different hardware, so they are not a fair ranking. They also describe COCO photographs of everyday objects. They say nothing about cracks or lights on an airfield.
How papers compare models
Papers report average precision (AP), AP50 and AP75. The COCO evaluation code uses 10 intersection-over-union (IoU) thresholds from 0.50 to 0.95 in steps of 0.05. For masks, it computes the IoU between the predicted mask and the reference mask.
Use in airfield pavement inspection
A mask can give the size of one defect when you know the image scale. The Pavement Condition Index (PCI) follows ASTM D5340. The PAVER manuals record asphalt distress in square or linear feet and concrete distress as a count of slabs. For example, a slab with shrinkage cracks counts once, however many cracks it has. A count of masks is therefore not a PCI input by itself. The masks must be converted to the units that the manual uses for each distress type.
Frequently Asked Questions
- What is the difference between instance segmentation and semantic segmentation?
- Semantic segmentation assigns a class to every pixel but does not separate objects of the same class. Instance segmentation also separates them. If an image shows three cracks, semantic segmentation marks all crack pixels as one class, and instance segmentation returns three masks.
- Is instance segmentation the same as object detection?
- No. Object detection returns a bounding box and a class for each object. Instance segmentation returns a pixel mask and a class for each object, so it combines object detection with semantic segmentation.
- Which instance segmentation model is the most accurate?
- No paper compares them fairly here. The published figures use different hardware and different model versions. Mask R-CNN reports 35.7 mask AP at 5 fps, YOLACT reports 29.8 mAP at 33.5 fps, and Mask2Former reports 50.1 AP on COCO. All of these figures are for COCO photographs, not pavement.
References
- he-mask-rcnn
- Mask R-CNN He, Gkioxari, Dollar, Girshick, arXiv 1703.06870 (v1 20 Mar 2017, v3 24 Jan 2018)
- bolya-yolact
- YOLACT: Real-time Instance Segmentation Bolya, Zhou, Xiao, Lee, arXiv 1904.02689 (ICCV 2019)
- cheng-mask2former
- Masked-attention Mask Transformer for Universal Image Segmentation (Mask2Former) Cheng, Misra, Schwing, Kirillov, Girdhar, arXiv 2112.01527 (CVPR 2022)
- pycocotools-cocoeval
- cocoeval.py, COCO API (pycocotools) COCO Consortium
- paver-asphalt
- PAVER Distress Identification Manual for Asphalt Surfaced Airfields US Army Corps of Engineers ERDC-CERL
- paver-concrete
- PAVER Distress Identification Manual for Concrete Surfaced Airfields US Army Corps of Engineers ERDC-CERL
- faa-ac-150-5380-6c
- AC 150/5380-6C, Guidelines and Procedures for Maintenance of Airport Pavements FAA
This is a reference entry. TarmacView provides drone inspection services and doesn't sell the equipment or materials described in this glossary.