KR20010067778A - Image/video processing method and image/video object segmentation method - Google Patents

Abstract

PURPOSE: A method for treating an image/video and a method for segmenting an image/video object are provided to automatically segment the image/video without intervening by a user. CONSTITUTION: A semantic object is defined in an image. A pixel value of pixels in the defined semantic object is combined with a predetermined label based on a water marking, thereby labelling the semantic object. When the semantic object is defined in the image, an original image is divided into small blocks without overlapping small blocks with each other(202). In addition, the small blocks are grouped by a user in such a manner that the small blocks belonged to the same object are grouped(204). Then, image data including the labeled semantic object is compressed by using a predetermined image compression method(210). The compressed image data is stored in a predetermined data base(212).

Description

Image / video processing method and image / video object segmentation method

The present invention relates to a video / video data processing method, and more particularly, to a video / video data processing method for automatically segmenting an object without user intervention.

Extraction of semantic objects from still images or video is essential for content-based image processing. For example, video object segmentation is required as a preprocessing process in the encoding process in object-based MPEG-4 encoders and the extraction of various MPEG-7 descriptors such as shape descriptors.

However, until now, it is not known how to use a computer to automatically segment objects. In other words, the user's help is required in the process of segmenting semantic objects. Because of the human intervention, content-based video and video processing is not fully automatic, which makes it difficult to implement an online system.

To solve this problem, a method of minimizing or simplifying human involvement in the segmentation of video or video is described by C. Gu and MC Lee in September 1998, "Semiautomatic segmentation and tracking of semantic video objects. Semi-automatic Segmentation and Tracking of Video ", IEEE Transaction On Circuit and System for Video Technology, vol. 8, no. 5, pp. 572-584, "Fast object tracking in digital video" published in August 2000 by D. K. Park, H. S. Yoon, and C. S. Won, IEEE Transaction On Consumer Electronics, vol. 46, no. 3, pp. 785-790, and filed by The Trustees of Columbia University in the city of New York, dated September 24, 1999, entitled "System and method for semantic video object segmentation". PCT Application No. PCT / US99 / 22264.

The object partitioning methods disclosed in the above-mentioned materials or patent documents require the support of the user offline. Moreover, the techniques used in the object partitioning methods are not storage. Therefore, according to the conventional object segmentation method, this requires not only human support, but also the repetitive segmentation process is still performed on an offline basis, and the same semi-automatic object segmentation process whenever a semantic object extraction task is required. There is a problem that this must be repeated.

An object of the present invention is to provide an image processing method for processing an image so that an object can be automatically segmented without user intervention in the image.

Another object of the present invention is to provide an image processing apparatus for performing the image processing method.

Another object of the present invention is to provide a method of segmenting semantic objects from an image processed by the image data processing method as described above.

Another object of the present invention is to provide an image segmentation apparatus which performs the image segmentation method.

Another technical problem to be solved by the present invention is to provide a video processing method for processing a video so that an object can be automatically segmented in the video without user intervention.

Another object of the present invention is to provide a method for segmenting semantic objects from video processed by the video processing method as described above.

1 is a block diagram illustrating an image / video processing apparatus according to an exemplary embodiment of the present invention.

2 is a flowchart illustrating main steps of an image / video processing method according to an embodiment of the present invention performed in the image / video processing apparatus of FIG. 1.

3A is a diagram illustrating an example of an input original image.

3B is a diagram illustrating an example of an image divided into small blocks that do not overlap each other.

3C is a diagram illustrating an example of a labeling result.

3D is a diagram illustrating an example of a watermarked image.

4 is a block diagram illustrating a structure of an image / video segmentation apparatus according to an embodiment of the present invention.

5 is a flowchart illustrating main steps of a video / video segmentation method according to an embodiment of the present invention performed in the video / video segmentation apparatus of FIG. 4.

6A illustrates an example of a watermarked image.

6B is a diagram illustrating an example of an extracted object label.

6C is a diagram illustrating an example of an extracted rough object.

6D is a diagram illustrating an example of a precise semantic object.

In order to achieve the above technical problem, an image processing method according to the present invention is a method of processing an image so that an object can be automatically segmented without user intervention in the image, the method comprising the steps of: (a) defining a semantic object in the image; ; And (b) labeling the semantic object by combining the pixel value of the pixels in the defined semantic object with a predetermined label based on watermarking.

In addition, the step (a), (a-1) dividing the original image into small blocks that do not overlap each other; And (a-2) grouping the small blocks into groups belonging to the same object with the user's intervention.

In addition, after the step (b), (c) compressing the image data including the labeled semantic object using a predetermined image compression method; And (d) storing the compressed image data in a predetermined database.

In addition, the step (a), (a ') with the help of a human using a computer user interface to define a semantic object from the image; preferably includes.

In addition, step (b) may comprise: (b-1) assigning a label to each pixel of the defined semantic object; And (b-2) performing watermarking by hiding the label assigned to the pixel to the gray level value of the corresponding pixel.

In addition, in the step (b-2), the (b-2-1) pixel The gray level of Where L is an integer to make the watermark robust to changes in the Least Significant Bit (LSB), K is the total number of segmented objects in the image, Is a truncated integer value, Is an absolute value, and "mod" represents a modular operation, the watermarked gray level value of Determining by; And (b-2-2) replacing each pixel value with the determined watermarked gray level value.

According to another aspect of the present invention, there is provided an image processing apparatus including: a divider for dividing an object from an image input through user intervention; A labeling unit to label the grouped object with a predetermined label; And a watermarking unit obtaining a watermarked image by combining the original image and the label based on the watermarking.

According to another aspect of the present invention, there is provided a method of segmenting an image, the method comprising: (a) extracting an object label from a watermarked image; (b) obtaining a watermarked gray level from all the pixels in the watermarked image; And (c) extracting the object label from the watermarked gray level value.

According to another aspect of the present invention, there is provided an image segmentation apparatus, including: a label extractor configured to extract an object label from a watermarked image; A coarse object extracting unit which extracts a coarse object using the extracted object labels; And a precision object extractor which obtains a precise semantic object by trimming a rough object.

According to another aspect of the present invention, a video processing method according to the present invention is a method of processing a video so that an object can be automatically segmented without user intervention in a video shot, the method comprising: (a) a first frame of an input video shot; Dividing the into small blocks that do not overlap each other; (b) defining semantic objects by grouping small blocks into groups belonging to the same object with the user's intervention; And (c) assigning the same object labels to the objects corresponding to the manually segmented semantic objects while tracking along the temporal domain of the video shot.

According to another aspect of the present invention, there is provided a video segmentation method comprising: (a) extracting an object label from a first frame of a watermarked video shot; (b) extracting coarse objects from the input watermarked video using the object labels extracted from the first frame while tracking along the temporal domain of the video shot; And (c) obtaining a precise semantic object by trimming a coarse object.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an image / video processing apparatus according to an exemplary embodiment of the present invention. Referring to FIG. 1, an image / video processing apparatus according to an exemplary embodiment of the present invention includes a division unit 102, a labeling unit 104, a watermarking unit 106, a compression unit 108, and a database 110. do. 2 is a flowchart illustrating main steps of an image / video processing method according to an embodiment of the present invention performed in the image / video processing apparatus of FIG. 1. 2 is often referenced below.

The operation of the video / video processing apparatus will be described. The dividing unit 102 divides the object semi-automatically or manually from the image input by the user's intervention. In this embodiment, the block-based semi-automatic object partitioning method is applied to the partitioning. For example, the original image is divided into small blocks that do not overlap each other (step 202). 3A illustrates an example of an input original image, and FIG. 3B illustrates an example of an image divided into small blocks that do not overlap each other. Next, the user's intervention groups the small blocks into groups belonging to the same object (step 204). In the grouping process, the user may make a manipulation such as mouse dragging or clicking by using a user interface provided by a predetermined program in the computer. For convenience of description, the present embodiment describes a case in which only one object and a background exist in the image.

Next, the labeling unit 104 labels the grouped object with a predetermined label (step 206). Figure 3c shows an example of the labeling results. Referring to FIG. 3C, blocks belonging to a grouped object as a group are labeled with binary "1", and blocks belonging to the background are labeled with binary "0". In this way, one bit information may be allocated to all blocks. Each block in the blocks consists of a number of pixels, so that all the pixels in the block are assigned the same one bit of information.

Next, the watermarking unit 106 obtains the watermarked image by combining the original image and the label based on the watermarking (step 208). In the present exemplary embodiment, the watermarking unit 106 performs watermarking by hiding label values to gray level values of corresponding pixels. For example, pixel The gray level of Watermarked gray level Is,

Determined by Replace each pixel value with the determined watermarked gray level value. This makes the object label pixel Gray level value Insert in Here, L is an integer for making the watermark robust to changes in the Least Significant Bit (LSB), and K is the total number of divided objects in the image. Also, Represents a truncated integer value, Denotes an absolute value and "mod" denotes a modular operation. Such a watermarking process can be understood to hide the object label in the middle of the gray level. Thus, it can withstand changes in LSB (Least Significant Bit) due to perturbation from the outside.

The labeling process is performed on all pixels in the image. That is, by identifying whether the corresponding pixel is the last pixel in the image, if not the last pixel, the next pixel is selected and the labeling process is performed. If the last pixel, the object label insertion is terminated. 3D shows an example of a watermarked image. Referring to FIG. 3D, the watermarked image is not distinguished from the original image due to the property of watermarking. When the insertion of the object label is finished, the image in which the object label is inserted is compressed (step 210), and the compressed image is stored in the database 110 (step 212).

According to the image / video processing method as described above, semantic objects are defined in the original image with the aid of a user interface, and object labels are inserted in the original image. The image with the object label inserted is compressed and stored in the database. Images processed according to the image / video processing method as described above and stored in the database can be automatically divided as described later.

Meanwhile, search and retrieval of images based on semantic objects is used in various content-based image and video applications. To this end, segmentation of video and video is essential.

4 is a block diagram illustrating a structure of an image / video segmentation apparatus according to an embodiment of the present invention. Referring to FIG. 4, an image / video segmentation apparatus according to an exemplary embodiment of the present invention includes a stretcher 402, a label extractor 404, a coarse object extractor 406, and a precision object extractor 408. do. 5 is a flowchart illustrating main steps of a video / video segmentation method according to an embodiment of the present invention performed in the video / video segmentation apparatus of FIG. 4. 5 is often referenced below.

The operation of the video / video segmentation apparatus will be described. The decompression unit 402 obtains the watermarked image by decompressing the compressed image loaded from the database 110 (step 502). 6A shows an example of a watermarked image. Next, the label extractor 404 extracts the object label from the watermarked image (step 504). 6B illustrates an example of the extracted object label. Step 504 is accomplished by obtaining a watermarked gray level from all the pixels in the watermarked image and extracting the object label from the watermarked gray level value. That is, in this embodiment, the gray level value watermarked with the label The object label value from of,

Extract according to. E.g, The gray level of the pixels on the image Is 30 and the pixel belongs to an object labeled x (i, j) = 2, that is, k = 2, where K = 3 and L = 4, Get Next, the watermarked gray level is calculated according to equation (1). = 30 + 4 1 = 34. now, Hidden Object Labels silver Becomes Here, it should be noted that even if the watermarked image undergoes an image processing process that may cause various perturbations on the gray level, an accurate object label can be extracted. Gray level disturbances varying from 32 to 35 are acceptable and the correct object labels can be obtained. According to the above method, the object label is hidden in the middle part of the gray level, thereby making it robust to the LSB change. In addition, increasing the integer value L is more robust to changes caused by unintended disturbances such as compression and scaling. However, since L increases, the insertion of an object label is performed at a higher bit level, which may cause a deterioration that can be visually recognized at a specific pixel. Therefore, L must be properly determined.

Next, the coarse object extractor extracts the coarse object using the extracted object labels (step 506). 6C shows an example of the extracted rough object. Next, the precision object extractor obtains a precise semantic object by trimming the coarse object (step 508). In this step, general pixel-based image segmentation methods including a region growing technique may be used. The region growth technique allows for precise division of pixels around the contour. In addition, the noise or distortion of the image due to unintentional image deformation due to compression or the like can be corrected by using a low pass filter or a morphological filter. 6D shows an example of a precise semantic object.

Thus, semantic objects in the image can be automatically extracted. That is, according to the video / video object segmentation method as described above, no human intervention is required. Furthermore, according to the video / video segmentation method as described above, whenever a semantic object extraction task is required, object segmentation is fully automatic without human intervention.

According to the video / video object segmentation method described above, content-based image processing applications or other computer vision problems that use content-based indexing for MPEG-7 and object-based compression such as MPEG-4. We can segment semantic objects completely automatically without human intervention from images imported from databases.

In the above embodiment, for convenience of description, only the image is described as the object of division. However, the method according to the present invention as described above can be applied to a variety of video including a video as will be understood by those skilled in the art.

To explain in more detail, first, the object is segmented semi-automatically or manually within the first frame of the video shot input by the user's intervention. As described with reference to FIG. 2, a block-based semi-automatic object partitioning method may be applied to the partitioning. That is, the first frame of the input video shot is divided into small blocks that do not overlap each other, and then, by user intervention, the small blocks are grouped into groups belonging to the same object. Next, the grouped object is labeled with a predetermined label. Now, while tracking along the temporal domain of the video shot, objects corresponding to the manually segmented semantic objects are given the same object label. Next, the watermarked video is obtained by combining the original video and the label based on the watermarking, and the obtained watermarked video is compressed and stored.

In the segmentation process, first, a watermarked image is obtained by importing compressed video from a database and stretching the imported compressed video. Next, an object label is extracted from the first frame of the watermarked video shot. Next, using the object label extracted from the first frame of the video shot while tracking along the temporal domain of the video shot, the same object label is assigned to the objects corresponding to the manually segmented semantic object. Next, extract the rough object using the extracted object labels, and refine the rough object to obtain a precise semantic object.

The scope of the present invention is not limited to the above embodiments, but may be modified or modified by those skilled in the art within the scope of the present invention as defined by the appended claims. In addition, the image / video processing method and the image / video segmentation method according to the present invention as described above can be written as a program executed in a personal or server-class computer. Program codes and code segments constituting the program can be easily inferred by computer programmers in the art. The program may also be stored in a computer readable recording medium. The recording medium includes a magnetic recording medium, an optical recording medium, and a propagation medium.

As described above, the present invention provides a method for content-based image processing applications that use content-based indexing for MPEG-7 and object-based compression, such as MPEG-4, from databases in other computer vision problems. From imported images, semantic objects can be segmented completely automatically without human intervention.

Claims (22)

In a method of processing an image so that the object can be automatically segmented without user intervention in the image, (a) defining a semantic object in the image; And (b) labeling the semantic object by combining the pixel value of the pixels in the defined semantic object with a predetermined label based on watermarking. According to claim 1, wherein the step (a), (a-1) dividing the original image into small blocks that do not overlap each other; And (a-2) grouping small blocks into groups belonging to the same object with the user's intervention. The method of claim 1, wherein after step (b), (c) compressing the image data including the labeled semantic object using a predetermined image compression method; And (d) storing the compressed image data in a predetermined database. According to claim 1, wherein the step (a), (a ') defining a semantic object from an image using a user interface of a computer with the help of a human; According to claim 1, wherein step (b), (b-1) assigning a label to each pixel of the defined semantic object; And (b-2) performing watermarking by hiding a label assigned to a pixel to gray level values of the corresponding pixels. The method of claim 5, wherein step (b-2) (b-2-1) pixels The gray level of Where L is an integer to make the watermark robust to changes in the Least Significant Bit (LSB), K is the total number of divided objects in the image, Is a truncated integer value, Is an absolute value, and "mod" represents a modular operation, the watermarked gray level value of Determining by; And (b-2-2) replacing each pixel value with the determined watermarked gray level value. An apparatus for processing an image to automatically segment an object in the image without user intervention, A divider dividing an object from an image input by a user's intervention; A labeling unit to label the grouped object with a predetermined label; And And a watermarking unit configured to obtain a watermarked image by combining the original image and the label based on the watermarking. The method of claim 7, wherein the divider, And dividing the original image into small blocks that do not overlap each other, and grouping the small blocks into groups belonging to the same object by user intervention. The method of claim 7, wherein the watermarking unit, pixel The gray level of Where L is an integer to make the watermark robust to changes in the Least Significant Bit (LSB), K is the total number of segmented objects in the image, Is a truncated integer value, Is an absolute value, and "mod" represents a modular operation, The gray level of Watermarked gray level of Determining each pixel value and replacing each pixel value with the determined watermarked gray level value. The method of claim 7, wherein A compression unit compressing an image in which an object label is inserted; And And a database for storing the compressed image. The image processing apparatus further comprises; (a) extracting an object label from the watermarked image; (b) obtaining a watermarked gray level from all the pixels in the watermarked image; And and (c) extracting the object label from the watermarked gray level value. The method of claim 11, wherein before step (a), Obtaining a watermarked image by decompressing a compressed image loaded from a database. The method of claim 11, wherein after step (c), (d) extracting the coarse object using the extracted object labels; And (e) obtaining a precise semantic object by trimming a coarse object. The method of claim 13, wherein step (e) An image segmentation method based on a region growing technique. The method of claim 11, wherein step (c) comprises: Gray level value watermarked with label The object label value from of, And extracting according to the image segmentation method. A label extractor which extracts an object label from a watermarked image; A coarse object extracting unit which extracts a coarse object using the extracted object labels; And And a precision object extracting unit obtaining a precise semantic object by trimming a coarse object. The method of claim 16, And a decompressing unit which obtains a watermarked image by decompressing a compressed image loaded from a database. In a method of processing video so that the object can be automatically segmented without user intervention within the video shot, (a) dividing the first frame of the input video shot into smaller blocks that do not overlap each other; (b) defining semantic objects by grouping small blocks into groups belonging to the same object with the user's intervention; And and (c) assigning the same object labels to the objects corresponding to the manually segmented semantic objects while tracking along the temporal domain of the video shot. The method of claim 18, wherein after step (c), (d) obtaining the watermarked video by combining the original video and the label based on the watermarking. The method of claim 19, wherein after step (d), (e) compressing the watermarked video using a predetermined image compression method; And (d) storing the compressed video in a predetermined database. (a) extracting an object label from the first frame of the watermarked video shot; (b) extracting coarse objects from the input watermarked video using the object labels extracted from the first frame while tracking along the temporal domain of the video shot; And (c) obtaining a precise semantic object by trimming a coarse object. The method of claim 21, wherein before step (a), Obtaining a watermarked image by importing the compressed video from the database and decompressing the loaded compressed video.