Segmentation

Segmentation is used to identify objects with great precision. This is done by using a "brush" or Smart Labeling to mark the exact location of an object you want your model to detect. This object type is commonly used for identifying irregular shapes.

Let’s say you want to identify cracks in cement. You could use object detection; however, bounding boxes would include too much background (or space that you do not want to include in your label). During model training, the model will look at every labeled pixel. If you include extra, unimportant pixels, the model might think these pixels are also important. Consider the screenshots below where the user tries to identify a crack with Object Detection and Segmentation. Which would you choose?

Bounding Boxes to Detect Cracks in Cement

Pixel Precision to Detect Cracks in Cement

Labeling Tools for Segmentation

Segmentation offers the following tools to label objects. You can mix and match tools when labeling.

• Smart Labeling: Click an area to auto-detect objects. 
• Polygons: Create a shape to fill the object you want to detect. You can also use this tool to "erase" labeled regions.
• Brushes: "Paint" over the object you want to detect. You can also use this tool to "erase" labeled regions.
• Polylines: Create lines over the object you want to detect.

Label with Smart Labeling

Smart Labeling is a labeling tool powered by the Segment Anything Model (SAM) that helps you label objects in Segmentation projects faster than using traditional methods.

To use Smart Labeling, simply select the tool (or press Q), hover over and click the object you want to select. Smart Labeling automatically detects the shape of the object and adds a gray overlay to it. To confirm that Smart Labeling detected the correct object, press Return (Mac) or Enter (Windows) to confirm.

Smart Labeling

Label an Object with Smart Labeling

To label an object with Smart Labeling:

1. Click the Smart Labeling icon if it isn't selected, or press q.
2. Select the Class you want to label from the drop-down menu.
   Select Smart Labeling and the Class
3. Hover over the object you want to select. Smart Labeling adds a gray overlay over the object it "thinks" you want to select.
   Hover Over the Object
4. Click the object. Smart Labeling adds a green dot where you clicked. This, along with the gray overlay, help you track what regions you've selected. Clicking the object also adds a blue border to your frame, which indicates that you're using Smart Labeling. LandingLens Adds a Green Dot
5. If you want to add another region to the same object, hover over that region. The cursor turns into a + (plus).
   The Cursor Turns into a + (Plus)
6. Click the object. Smart Labeling adds a gray overlay to the object and adds a green dot where you clicked.
   
   LandingLens Adds an Overlay and a Dot
7. You can continue to add regions using this method.
   
   Click All the Regions You Want to Add
8. Once you've labeled all regions of the object, either click Save Labels (Return), or press Return (Mac) or Enter (Windows).
   
   Save the Labels
9. Smart Labeling labels the object with the selected Class.
10. 
    
    The Object Is Labeled

Remove a Region from a Selection

Let's say a region gets accidentally added to your selection. While Smart Labeling is selected, you can hover over that region until the cursor turns into a − (minus). Then click that region. Smart Labeling removes the selection. When you're done, press Return (Mac) or Enter (Windows). 

To remove a region from an object (assumes that Smart Labeling is enabled, and you just labeled an object):

1. Hover over and click the labeled region. Smart Labeling removes the overlay and adds a red dot where you clicked, which helps you track what regions you've removed.
   
   Remove Regions
2. Continue to click selected regions until you've removed all the selections you want.
3. When you're done removing regions, press Return (Mac) or Enter (Windows).

Tips for Labeling Multiple Objects in an Image

Let's say you're developing a model that detects tractors in construction sites. When using Smart Labeling, we recommend that you label each tractor one at a time, instead of labeling them all at once. In other words, label one tractor, and then press Return (Mac) or Enter (Windows). Then label the second tractor, and so on.

Label Each Object Individually

We don't recommend labeling all tractors at once, and then clicking Enter/Return. Doing that makes Smart Labeling think that these areas are the same object, and not the same type of object. If you do this, you might notice additional areas added to the object that aren't actually part of that object.

Label with Polygons

The polygon tool lets you draw an outline of a shape. After the outline has been completed, the shape will be automatically filled. You can also use this tool to erase labeled regions.

To label with polygons:

1. Open the Segmentation project you want to edit.
2. Open the image you want to label.
3. Click the Polygon icon. (You can also press W on your keyboard.)
4. Select a class from the drop-down menu.
   Click the Polygon Tool and Choose a Class
5. Click where you want to start your shape. A blue circle displays, indicating the start of the shape.
6. Click to continue creating your shape.
7. Complete your shape by clicking the blue circle. The shape automatically fills with the color of the class.
   
   Create Shapes with the Polygon Tool

Label with Brushes

The brush tool lets you paint over the object of interest. You can also use this tool to erase labeled regions. 

To label with brushes:

1. Open the Segmentation project you want to edit.
2. Open the image you want to label.
3. Click the Brush icon. (You can also press B on your keyboard.)
4. Select a class from the drop-down menu.
   
   Click the Brush Tool and Choose a Class 
5. Adjust the size of the line width accordingly.
6. Click and drag your cursor to "paint" over the area you want to identify.
7. Release the mouse to complete your edits.
   
   Label with the Brush Tool

Remove Excess Labels with Brushes

To remove excess labels with a Brush:

1. Open the Segmentation project you want to edit.
2. Open the image you want to label.
3. Click the Brush icon. (You can also press B on your keyboard.)
4. Click the Erase icon. (You can also press E on your keyboard.)
   
   Click the Brush Tool and Select the Eraser 
5. Adjust the size of the eraser accordingly.
6. Click and drag your cursor over the area you want to remove.
7. Release the mouse to complete your edits.
   
   Use the Brush to Erase Labels

Label with Polylines

The polyline tool lets you create lines over the object of interest.

To label with polylines:

1. Open the Segmentation project you want to edit.
2. Open the image you want to label.
3. Click the Polyline icon. (You can also press L on your keyboard.)
4. Select the class you want to label from the drop-down menu.
   
   Click the Polyline Tool and Select a Class 
5. Adjust the size of the line width accordingly.
6. Click where you want to start your line.
7. Click to change the direction of your line.
8. Press Enter on your keyboard to complete your edits.
   
   Label with the Polylines