The Definitive Guide to Instance Segmentation [+V7 Tutorial]

Ready to streamline AI product deployment right away? Check out:

• V7 Model Training

• V7 Workflows

• V7 Auto Annotation

• V7 Dataset Management

What is Instance Segmentation?

Instance Segmentation is the technique of detecting, segmenting, and classifying every individual object in an image.

We can refer to Instance Segmentation as a combination of semantic segmentation and object detection (detecting all instances of a category in an image) with the additional feature of demarcating separate instances of any particular segment class added to the vanilla segmentation task.

Instance Segmentation produces a richer output format as compared to both object detection and semantic segmentation networks. 

Pro tip: Curious to learn more about neural networks? Check out The Essential Guide to Neural Network Architectures.

Whereas an object detection system coarsely localizes multiple objects with bounding boxes and a semantic segmentation framework produces pixel-level category labels for each category class, Instance Segmentation produces a segment map of each category as well as each instance of a particular class—therefore, providing a more meaningful inference on an image.

Let’s consider an image below with cats and dogs.

Semantic Segmentation can mark out the dog and cats’ pixels, however, there is no indication of how many dogs and cats are there in the image.

With Instance Segmentation, one can find the bounding boxes of each instance (which in this case pertains to a dog and two cats) as well as the object segmentation maps for each instance, thereby knowing the number of instances (cats and a dog) in the image.

Pro tip: Looking to annotate with bounding boxes? Check out 9 Essential Features for a Bounding Box Annotation Tool.

Instance segmentation vs. Semantic Segmentation vs. Panoptic Segmentation

To understand the difference between three kinds of segmentation—Instance Segmentation, Semantic Segmentation, and Panoptic Segmentation, we need to define two different categories of objects—stuff and things.

Stuff defines categories, which cannot be counted such as the sky and the road.

Things are the actual objects in the image—they can be counted by assigning different instance IDs to each one of them.

Semantic Segmentation takes the input image and marks every pixel in the image to category class. Thus all instances of a particular category receive the same label.

In Instance Segmentation, bounding boxes are generated for each instance of multiple categories present along with the object segmentation masks. It treats multiple objects of the same class as distinct instances.

Panoptic segmentation uses an algorithm to understand the difference between amorphous stuff and countable things like cars and persons. Panoptic segmentation uses the same backbone as that of Mask-RCNN used for Instance Segmentation with an extra panoptic head responsible for producing the final outputs by processing the semantic and instance results. It is completely parameter-free and requires no training. 

Therefore, the panoptic head is used to differentiate between things and stuff. 

For pixels with no instance or thing output, it is considered to be a class pertaining to stuff. For pixels both having stuff and thing labels, Softmax is used to determine if it should be labeled as that instance or semantic values. 

Additionally, the authors have also made support for logits for the ‘unknown’ class in order to avoid making wrong predictions. If for any pixel the maximum logit for the ‘thing’ class from the semantic head is larger than the maximum of the logit from the instance head, then it is likely that those pixels correspond to missing instances and are labeled as unknown.

How does Instance Segmentation work?

Instance Segmentation is a challenging task and requires the detection of multiple instances of different objects present in an image along with their per-pixel segmentation mask. 

Instance Segmentation methods can be both R-CNN driven or FCN driven. 

FCNs (Fully Convolutional Networks) have been widely used for Semantic Segmentation. 

Although, convolutional networks being translation-invariant, cannot be used for Instance Segmentation, which requires the detection and segmentation of individual object instances, the same image pixel receives the same responses (thus classification scores) irrespective to it’s relative position in the context. 

In order for Instance Segmentation to work, Semantic Segmentation needs to operate on region level and the same pixel can have different semantics in different regions—this cannot be modeled by a single FCN on a single image. 

In conventional FCNs, a classifier is trained to predict each pixel’s likelihood score of “the pixel belongs to some category”. 

Pro tip: Read Image Classification Explained: An Introduction [+V7 Tutorial].

We use k2 position-sensitive score maps that respond to k x k evenly partitioned cells of the object to introduce translation variant property. Each score now represents the likelihood of ”the pixel belonging to some object instance at a relative position”. 

For example, a pixel corresponding to a person can be a foreground for a person in one semantic position and a background for another person (instance). Now, object detection and segmentation are carried out jointly and simultaneously. 

Mask R-CNN is the state-of-the-art model for Instance Segmentation with three outputs. It has a class label and a bounding box offset, which is similar to that of Faster R-CNN, and a third branch that outputs the object mask requiring extraction of a much finer spatial layout of an object. 

Unlike class labels or box offsets, which inevitably get collapsed into short output vectors by fully connected layers, the predicted masks from each ROI are able to maintain object spatial layout without collapsing into a vector representation that lacks spatial dimensions. 

Models like Faster R-CNN used ROIPool for extracting small feature maps from each ROI and to achieve these quantizations are performed—on a continuous coordinate x by computing for e.g. [x / 16] where 16 is the feature map size and [.] performs rounding operation. 

Pro tip: Ready to train your models? Have a look at Mean Average Precision (mAP) Explained: Everything You Need to Know.

These quantizations introduce misalignments between the ROI and the extracted features. To curb these effects, Mask R-CNN uses the ROI Align layer which avoids quantization (for e.g, x/16 is done instead of [x / 16] and bilinear interpolation is used to compute the exact values of the input features at four regularly sampled locations in each ROI bin. 

An ROI is considered positive if it overlaps enough with the ground truth bounding box, which is analyzed by the mask loss and is only defined for positive ROIs or ones which have enough overlapping with the ground truth. 

In the case of architecture, ResNet has been the choice for backbone architecture in both R-CNN and FCN driven approaches. 

The extracted features are usually taken from the final convolutional layer of ResNet-50. For extracting ROI features, it uses an FPN (Feature Pyramid Network) which uses a top-down architecture with lateral connections to build an in-network feature pyramid from a single–scale input. Anchors of a single scale are assigned to each level, formally anchors are defined to have areas of {322, 642, ..} pixels on {P2, P3, …} respectively. Anchors of multiple aspect ratios are also used at each level. 

Training labels are assigned to the anchors based on Intersection-over-Union (IoU) ratio. A higher positive IoU is given a positive label and vice-versa. After region proposals, Non-Max Suppression is used to remove the bounding boxes whose IoU is less than the threshold. 

Pro tip: Looking for a cost-free way to label your data? Check out The Complete Guide to CVAT—Pros & Cons.

Instance Segmentation applications

Here are some of the most prominent applications of Instance Segmentation.

Self-driving cars

For a self-driving car with complex street scenarios such as a construction site or a very crowded street with a lot of pedestrians, it should have a detailed understanding of its surroundings. 

Such fine-grained results can be achieved by segmenting image content with pixel-level accuracy, This approach can be done by Panoptic Segmentation which is a mix of Semantic Segmentation (differentiating sky, road, pedestrian, and other cars) and Instance Segmentation (differentiating different instances of the same category). 

It can also be used in conjunction with dense (pixel level) distance-to-object estimation methods to help enable high-resolution 3-D depth estimation of a scene. 

Medical scans

There is also a wide variety of applications of Instance Segmentation in the medical domain. 

In histopathologic images which are usually whole slide images containing a large number of nuclei of various shapes surrounded by cytoplasm. Instance Segmentation plays an important role to detect and segment nuclei which can be further processed for the detection of dangerous diseases like cancer. ‍

It is also used for detecting tumors in MRI scans of the brain. Semantic Segmentation is also being widely used in segmenting multiple organs on laparoscopic surgery and segmenting cataract surgery instruments.

Satellite imagery

In satellite imagery, the size of the objects is usually quite small, and performing pixel-wise is not very efficient due to the close placement of the objects relative to the resolution of the image. Therefore, to treat each object as a separate instance, we can use a network architecture performing instance segmentation to achieve a better separation between objects.

Some major areas where instance segmentation is used on satellite images include the detection and counting of cars, ships detection for maritime security, oil discharge control, and sea pollution monitoring, and segmentation of buildings are used to make geospatial analysis.