Skip to content

Image Segmentation

Segmentation is a critical step in the image analysis pipeline, particularly when analyzing paintings. After loading and preprocessing raw images (e.g., photographs of paintings), segmentation can help in further isolating specific figures or objects within the image. In case one is interested in the combined group of figures, the process of segmentation can be skipped. The segmentation process can be further divided into three main tasks: object detection, segmentation, and pose estimation.

This tool provides a user-friendly interface to perform advanced image segmentation and analysis tasks. By following these steps, you can effectively isolate and analyze figures within images, which is essential for more in-depth image analysis.

This section explains how to use the image segmentation features in the app, guiding you through the process step by step.

Loading the Model

The app utilizes YOLO (You Only Look Once) as state-of-the-art models for detection, segmentation, and pose estimation. In the context of analyzing paintings, YOLO can quickly and accurately identify and segment figures or objects within an image, allowing for detailed analysis of each isolated component. The first step is to load the appropriate model based on the task you want to perform:

Detection

Description: Detection identifies and locates objects within an image by drawing bounding boxes around them. It categorizes each detected object according to predefined classes. Usefulness: This method is used when you need to quickly identify the presence and position of objects in an image. It's the first step in understanding the overall composition before diving into more detailed segmentation or pose analysis.

Segmentation

Description: Segmentation goes a step further than detection by not only identifying objects but also delineating their precise boundaries. This method isolates objects by creating a mask that covers the entire shape of each detected object, providing a pixel-level understanding. Usefulness: Segmentation is particularly useful when you need to analyze the shape, size, or outline of objects. In art analysis, this allows for detailed comparison of the figures' contours, which is essential for tasks such as examining whether similar templates were used in different paintings.

Pose Estimation

Description: Pose Estimation identifies key points on a figure to understand its orientation and posture. It is especially useful for human figures, where understanding the pose can reveal important details about the subject's activity or emotion in the artwork. Usefulness: This method is vital when the analysis requires understanding the interaction between different parts of a figure, such as in the study of human anatomy in art. Pose estimation provides insights into the alignment and movement within the painting, contributing to the overall analysis of the composition.

The model is loaded based on your selection, and if there is an issue loading the model, an error message will be displayed.

Selecting the Task

You can select the task you want to perform from the options available:

  • Detection: Choose this option to quickly identify the objects within an image.
  • Segmentation: Select this to create masks that delineate the boundaries of objects within the image.
  • Pose Estimation: Use this to detect and analyze key points on the figures, such as joints in a human figure.

Setting Confidence and Display Options

Before running the task, you can set the model's confidence level using a slider. This controls how certain the model needs to be before it considers an object as detected. You can also choose to add bounding boxes to the output image to visualize detected objects clearly.

Running the Task

After selecting the task and setting the options:

  • The app processes the image using the YOLO model.
  • The detected objects are displayed along with any bounding boxes, if selected.
  • If no objects are detected, an error message is shown.

Viewing and Downloading Results

The app displays the processed image, and you can compare the original image with the processed result. If the segmentation or pose estimation task was selected, the isolated objects or keypoints are also shown.

Finally, you can download the processed images (e.g., separate figures) directly from the interface for further use or analysis. Note that, in case figures are overlapping, the segmentation process may not be able to separate them completely. Only figures in front will be segmented completely.