KR20110047814A - Apparatus and method for dividing moving picture data in moving picture system - Google Patents

Abstract

PURPOSE: A video data partition apparatus and method thereof in a video system are provided to partition video data by using color information, depth information, and motion information in a video system. CONSTITUTION: A storing unit(110) stores a plurality of video data that is photographed by a plurality of image photographing device. A first extracting unit(121) extracts color information from video data for each frame. A first partition unit(131) partitions frames of video data and extracts depth information from video data. A second extracting unit(122) extracts motion information by suing color information. A second partition unit(132) partitions video data by using motion information and depth information.

Description

Apparatus and method for dividing moving picture data in moving picture system}

The present invention relates to a moving picture system, and more particularly, to an apparatus and method for dividing moving picture data using color information, depth information, and motion information in a moving picture system.

The video is composed of a plurality of frames, and is a video that makes the user feel as if the objects included in the frame is moving by playing the plurality of frames in quick succession. The quality of such video data is determined by the number of frames per second. In other words, the video date is a video that is quickly skipped as the plurality of frames change momentarily little by little, the smoothness of the movement is determined by how many frames are inserted and moved in one second.

In general, since the user wants to view more high quality video data, the user's needs are satisfied by increasing the number of frames per second of the video data. However, when the number of frames per second of video data increases, the size of the video data increases, which takes a long time to transmit the video data to the video receiver. For this reason, a method of compressing a video or a method of segmenting a video has been studied. In addition, there is a method of dividing the image into uniform and similar area units based on various features of the image as the video segmentation method, which is one of the most important elements in image analysis and computer vision work for medical image processing or robot driving. For this reason, video segmentation methods have been studied in the field of stereo matching and computer vision, and much research has been done.

The video segmentation method, which has been studied previously, segmented the video using sub-features such as color, edge, and texture of the video. However, since it is difficult to express all the information of the video only by the color, edge, and texture of the video, there is a disadvantage in that the image cannot be divided by objects. Recently, in order to solve the disadvantage that the objects included in the video cannot be divided, a lot of researches have recently been conducted to segment the video using the combination of sub-features.

However, the currently proposed method of segmenting an image cannot be segmented for each object included in the image when the color and texture of the object included in the video are similar. Accordingly, there is a need for a method of segmenting an image that can segment an image while solving such a problem.

Accordingly, an object of the present invention is to provide a video data segmentation apparatus and method in a video system.

Another object of the present invention is to provide an apparatus and method for dividing moving image data using color information, depth information, and motion information in a moving image system.

Another object of the present invention is to provide an apparatus and method for dividing moving image data by extracting color information, depth information, and motion information of moving image data for each frame of the moving image in a moving image system.

In addition, another object of the present invention is to provide a video data segmentation apparatus and method for maintaining the continuity of the segmented video data by updating the color information, depth information, motion information of the video data extracted from the video in the video system.

In order to solve the above objects, a video data segmentation apparatus according to an embodiment of the present invention includes a storage unit for synchronizing and storing a plurality of video data shot by a plurality of video photographing apparatuses frame by frame; A first extracting unit extracting color information for each frame from the video data; A first separator extracting depth information for each frame from the video data and separating the frames of the video data into a foreground area and a background area, respectively; A second extracting unit extracting motion information for each frame by using the foreground area, the background area, and the color information; And a second separator configured to segment the video data using the color information, the depth information, and the motion information.

In accordance with another aspect of the present invention, there is provided a video data segmentation method comprising: synchronizing and storing a plurality of video data photographed by a plurality of video photographing apparatuses frame by frame; Extracting color information and depth information for each frame from the video data; Dividing the frames of the moving image data into a foreground area and a background area by using the depth information; Extracting motion information for each frame by using the foreground area, the background area, and the color information; And dividing the moving image data by using the color information, the depth information, and the motion information.

According to the present invention, the video data may be extracted by color information, depth information, and motion information of the video data for each frame of the video. In addition, the present invention can maintain the continuity of the divided video data by updating the color information, depth information, and motion information of the video data extracted from the video in the video system. The present invention can be used to extract stereo or multi-view depth information to maintain continuity of depth data for each frame.

The above objects, features, and advantages will be more clearly understood from the following detailed description with reference to the accompanying drawings, and accordingly, those skilled in the art to which the present invention pertains may have the technical idea of the present invention. It will be easy to implement. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail.

The present invention proposes a video data segmentation apparatus and method in a video system. Here, an embodiment of the present invention to be described later proposes an apparatus and method for segmenting video data using color information, depth information, and motion information in a video system. In addition, an embodiment of the present invention proposes a video data segmentation apparatus and method for maintaining the continuity of segmented video data by updating color information, depth information, and motion information of video data extracted from a video in a video system. Next, an apparatus for dividing video data in a video system according to an exemplary embodiment of the present invention will be described in more detail with reference to FIG. 1.

1 is a diagram schematically illustrating a structure of an apparatus for dividing moving image data in a moving image system according to an exemplary embodiment of the present invention.

Referring to FIG. 1, the video data splitting apparatus according to the present invention includes a storage unit 110, an extraction unit 120, a separation unit 130, a calculation unit 140, an adjustment unit 150, and a recording unit 160. do. The storage unit 110 includes a first storage unit 111 and a second storage unit 112, and the extraction unit 120 includes a first extraction unit 121 and a second extraction unit 122, The separator 130 includes a first separator 131 and a second separator 132.

Although the video data is not specifically illustrated in FIG. 1, the video data is video data photographed by a plurality of video recording apparatuses of various types, for example, a multi-view video camera or a stereo video camera and a depth camera. Is received by the video data splitting apparatus. The multi-view video camera refers to a camera that provides a user with color video captured at various viewpoints through at least one camera. In addition, a stereo video camera means a camera that simultaneously records images of two left and right views on a single image. In addition, the depth video data refers to data captured and received by the depth camera.

The first storage 111 receives video data photographed by a plurality of cameras, and stores frames of the video data in synchronization. The second storage unit 112 synchronizes and stores the depth video data captured by the depth camera.

The first extractor 121 receives the video data stored in the first storage 111 and, as the video data is composed of a plurality of frames, separates the color image for each frame from the video data. The color images separated by the frames are separated by colors, and grouped by colors to extract color information of the video data. Here, the first extractor 121 may not only extract color information of the video data using an average moving algorithm, but also extract color information using various types of algorithms other than the average moving algorithm. can do. In addition, the average shift algorithm is a method of extracting color information by separating the colors included in all frames from the moving image data by similar colors.

The first separator 131 receives the depth video data from the second storage unit 112, extracts depth information for each frame from the depth video data, and then uses the extracted depth information to extract depth video data. Saves it into the foreground area and background area. That is, the first separator 131 extracts depth image information from the received depth video data.

Here, the depth video data refers to an image including depth information among various types of images included in the video data, for example, a gray image. The depth information refers to information indicating a distance from a predetermined viewpoint to an object shown in the frame within the frame of the depth video data. Accordingly, the first separator 131 extracts the depth information that can represent the distance of the object represented in the frame as a pixel value from a predetermined viewpoint.

For example, in more detail, as the value of the pixel indicated by the first separator 131 is closer to 0, the color is closer to black, and the closer to this black color is the object farther from the predetermined point of view. do. In addition, as the value of the pixel represented by the first separator 131 is closer to 255, the color is closer to white, and the closer to white, the closer the object is from a predetermined viewpoint.

As described above, the first separator 131 extracts depth information for each frame from the depth video data, and then separates the depth video data into a foreground area and a background area for each frame by using the extracted depth information. Here, the first separator 131 may not only separate the depth video data into the foreground region and the background region by frame through the K-average grouping algorithm using the extracted depth information, but also the K-average grouping. In addition to the algorithm, various types of algorithms may separate the depth video data into a foreground area and a background area for each frame. Here, the K-average grouping algorithm may include an algorithm for splitting at least one or more pixels within a frame of the depth video data by including data in a group close to a reference point according to the K selection. it means.

The first separator 131 sets positions of arbitrary K groups and K groups for each frame of the depth video data, and calculates positions up to K groups with respect to the depth video data. In addition, the first separator 131 divides the depth video data into a foreground area and a background area by frames by including the corresponding area in the group closest to the calculated position.

The second extractor 122 extracts motion information for each frame from the video data by using the foreground area, the background area, and the color information. In other words, when the color information of the previous frame exists, the second extractor 122 compares the color information of the previous frame with the color information of the current frame by using the foreground area and the background area. The motion information is extracted through the comparison result.

The second separator 132 divides the moving image data using the color information, the depth information, and the motion information. In other words, the second separator 132 uses color information for each of the frames, for example, previous frames, for the current frame, for example, in the foreground area and the background area separated from the first separator 131. Split video data.

Here, the second separator 132 applies a moving image data segmentation algorithm using color information to the foreground region and the background region separated by the first separator 131. The moving picture data segmentation algorithm may be a mean shift algorithm proposed by Comaniciu or the like, or a grab cut algorithm proposed by Yuri boycov or the like. The average moving algorithm is an algorithm that extends an algorithm for dividing static video data using color of video data, and refers to an algorithm capable of segmenting video data. Here, the average moving algorithm is an algorithm for dividing dynamic video data into an algorithm capable of dividing video data without being affected by the shape change of an object.

The second separator 132 transmits the splitting result of the moving picture data for the previous frame and the splitting result of the moving picture data for the current frame to the calculator 140.

The calculator 140 determines a depth search range to be used for the depth estimation to perform depth estimation on the foreground area and the background area for each frame of the video data divided by the second separator 132. For example, the calculator 140 determines a depth search range to be used for the depth estimation by using the split result of the video data of the current frame. In this case, the calculator 140 selects reference depth information to be used for determining the depth search range from the depth information of the previous frame and the depth information of the current frame, and here, the depth information of the current frame is provided for convenience of description. It is assumed that the selection is made with reference depth information. In addition, the calculator 140 selects the reference depth information as described above, updates the selected depth information that satisfies the following condition 1 to condition 3 to each frame of the divided video. Calculate the final reference depth information where the depth information is stored.

Specifically, the calculator 140 searches for an optimal motion region for each frame, for example, a previous frame, that satisfies all of the following conditions 1 to 3, and for this purpose, the calculator 140 searches for a current frame. Sliding each frame, for example, previous frames, searches for an optimal movement region for each frame.

Condition 1. Pixel ratio of S 'overlapping S> critical ratio and

Condition 2. Color Difference between Pixels in S and S 'in Overlapping Regions of S and S' <Critical Color Difference and

Condition 3. Depth difference between pixels in S and S 'in the overlapping area of S and S' <critical depth difference

In the above conditions, S means a previous frame and S 'means a current frame. Also, the depth difference is a difference value between the depth information of the previous frame and the depth information of the current frame. The overlapping area means an area where the previous frame and the current frame correspond to each other. Therefore, the calculation unit 140 calculates average depth information by averaging depth information of pixels included in a previous frame, and stores the calculated average depth information in the reference depth information of the current frame as described above to obtain final reference depth information. Calculate The calculation unit that calculates the reference depth information and the final reference depth information in this way determines the depth search range to be used for the depth estimation, as shown in Equation 1 below. Next, the first processor 211 searches for depth information with high similarity in the depth search range set as shown in Equation 1 based on the selected reference depth information.

Reference Depth (x, y)-Error × Reference Depth (x, y) <Depth Search Range <Reference Depth (x, y) + Error × Reference Depth (x, y)

Referring to Equation 1, the reference depth refers to the final reference depth information. In addition, the depth search range is determined as shown in Equation (1). That is, the calculator 140 searches for depth information with high similarity using a similarity algorithm in the depth search range. The similarity algorithm may be a sum of squared distance (SSD), a sum of absolute (SAD), a normalized correlation coefficient (NCC), a mean absolute difference (MAD), or the like. Next, a process of determining depth information having high similarity within the depth search range using a MAD algorithm will be described.

Hereinafter, a process of calculating the similarity using the mean absolute difference will be described with reference to Equation 2.

Referring to Equation 2, yi and yj represent color values for each frame of the divided video data. Since moving image data is captured by a plurality of photographing apparatuses, multiple viewpoints exist for one image. The multiple views include a first view area, for example, a left image and a second view area, for example, a right area. SL represents an area of the left image, and SR represents an area of the right image. A (S) represents the size of the divided video data. That is, the calculation unit 140 determines depth information having the smallest similarity difference in the depth search range by using Equation 2, and uses the final criterion represented by the determined minimum depth information in Equation 1 below. Update to depth information.

However, an occlusion area may occur when a boundary line between the object existing in the foreground area and the background area or a boundary line between two objects having different depth information exists. Since mismatches may occur due to the occlusion area, the calculator 140 adjusts the final depth information according to the mismatches generated. Then, the calculator 140 selects one of the final depth information of the current frame, the depth information of the previous frame, and the initial depth information to finally determine the depth information. That is, depth information having the smallest difference value is compared by comparing an average absolute difference between reference points obtained from each of depth information of the previous frame, final depth information of the current frame, and initial depth information, and corresponding points in other viewpoint video data. Will be chosen.

Then, the adjusting unit 150 finally projects the selected reference depth information to another viewpoint camera except the reference viewpoint camera. Since the reference depth information is depth information of the reference viewpoint camera, the reference depth information is projected to obtain depth information of another viewpoint camera except for the reference viewpoint camera. In this case, an occlusion area may occur due to a boundary line between objects in the moving image data or a boundary line between two objects having different depth information. In addition, a hole is generated due to the shielding area. The adjusting unit 150 compensates for the hole generated as the shielding area with depth information of pixels around the hole. Then, the recording unit 160 records the final depth information of each viewpoint video data photographed by the reference viewpoint and the other viewpoint camera as digital video data. Next, a process of dividing the video data in the video system according to an exemplary embodiment of the present invention will be described in more detail with reference to FIG. 3.

2 is a diagram schematically illustrating a process of dividing video data in a video system according to an exemplary embodiment of the present invention.

Referring to FIG. 2, in operation 210, the video data splitting apparatus receives video data photographed by a plurality of cameras and depth video data photographed by a depth camera, and synchronizes the video data and depth video data in synchronization. do. Here, the video data is video data photographed by a plurality of video recording apparatuses of various types, for example, a multi-view video camera or a stereo video camera and a depth camera, and received from the video recording apparatuses by the video data splitting apparatus. do. The multi-view video camera refers to a camera that provides a user with moving images captured at various viewpoints through at least one camera. The stereo video camera refers to a camera that simultaneously provides images of two left and right views in one image. In addition, the depth video data refers to data captured and received by the depth camera.

In operation 220, the video data splitting apparatus separates the color image from the video data for each frame, separates the color image separated for each frame for each color, and groups the color image to color information of the video data. Extract Here, the moving picture data dividing apparatus may not only extract color information of the moving picture data using an average moving algorithm, but also extract color information using various types of algorithms other than the average moving algorithm. . In addition, the average shift algorithm is a method of extracting color information by separating the colors included in all frames from the moving image data by similar colors.

In operation 230, the moving image data dividing apparatus separates the foreground and background regions using initial depth information of the depth image. That is, the image includes depth information, for example, a gray image, from among various types of images included in the video data. The depth information refers to information indicating a distance from a predetermined viewpoint to an object shown in the frame within the frame of the depth video data. Accordingly, the first separator 131 extracts the depth information that can represent the distance of the object represented in the frame as a pixel value from a predetermined viewpoint.

For example, in more detail, the closer the value of the pixel indicated by the moving picture data dividing device to zero is the color closer to black, the closer to this black means the object farther from the predetermined viewpoint. In addition, as the value of the pixel represented by the moving picture data dividing apparatus is closer to 255, the color is closer to white, and the closer to white, the closer the object is from a predetermined viewpoint.

As described above, the video data dividing apparatus extracts depth information for each frame from the depth video data, and then separates the depth video data into a foreground area and a background area for each frame by using the extracted depth information. Here, the first separator 131 may not only separate the depth video data into the foreground region and the background region by frame, but also through the K-average grouping algorithm using the extracted depth information. In addition to the average grouping algorithm, various types of algorithms may separate the depth video data into a foreground area and a background area for each frame. Here, the K-average grouping algorithm may include an algorithm for splitting at least one or more pixels within a frame of the depth video data by including data in a group close to a reference point according to the K selection. it means. The video data splitting apparatus sets positions of arbitrary K groups and K groups for each frame of the depth video data, and calculates positions up to K groups for the depth video data. The video data splitting apparatus includes the corresponding area in the group closest to the calculated position to separate the depth video data into the foreground area and the background area for each frame.

In operation 240, the moving image data dividing apparatus extracts motion information for each frame from the moving image data using the foreground area, the background area, and the color information. In other words, when there is color information about a previous frame, the video data dividing apparatus compares the color information of the previous frame and the color information of the current frame using the foreground area and the background area, and then compares the color information. The motion information is extracted through the result.

In operation 250, the video data dividing apparatus divides the video data using the color information, the depth information, and the motion information. In other words, the video data dividing apparatus divides the video data by using the color information in the foreground area and the background area of each reference frame, for example, the previous frame, which are separated from each other.

Here, the video data segmentation apparatus applies a video data segmentation algorithm using color information to the separated foreground and background regions, respectively. The moving picture data segmentation algorithm may be a mean shift algorithm proposed by Comaniciu or the like, or a grab cut algorithm proposed by Yuri boycov or the like. The average moving algorithm is an algorithm that extends an algorithm for dividing static video data using color of video data, and refers to an algorithm capable of segmenting video data. Here, the average moving algorithm is an algorithm for dividing dynamic video data into an algorithm capable of dividing video data without being affected by the shape change of an object. Next, the process of extracting the variation and depth using the apparatus for dividing the video data in the video system according to the embodiment of the present invention will be described in more detail with reference to FIG. 3.

FIG. 3 is a diagram schematically illustrating a process of extracting disparity and depth using an apparatus for dividing video data in a video system according to the present invention.

Referring to FIG. 3, first, in operation 310, when the segmentation information and the depth information of the video data of the previous frame exist, the video data segmentation apparatus selects a reference depth. On the other hand, if the segmentation information and depth information of the video data for the previous frame does not exist, the video data segmentation apparatus calculates the area unit depth in step 330. In more detail, the moving picture data dividing apparatus determines a depth search range to be used for the depth estimation for performing depth estimation on the foreground area and the background area for each frame of the divided moving picture data. For example, the moving picture data dividing apparatus determines a depth search range to be used for the depth estimation using the split result of the moving picture data for the current frame. In this case, the video data splitting apparatus selects reference depth information to be used for determining a depth search range from depth information of a previous frame and depth information of a current frame, and here, the depth information of the current frame is referred to for convenience of description. It is assumed that the selection is made with depth information. In addition, the video data segmentation apparatus selects the reference depth information as described above, and then updates the depth information satisfying all of the conditions 1 to 3 to the selected reference depth information and then applies it to each frame of the divided video data. Calculate the final reference depth information where the depth information is stored. Here, since the conditions 1 to 3 have been described in detail above, detailed description thereof will be omitted.

In operation 340, the moving image data dividing apparatus searches for depth information having high similarity in the depth search range set as shown in Equation 1 based on the calculated final reference depth information. Here, since the description of Equation 1 has been described above in detail, a detailed description thereof will be omitted. That is, the video data splitting apparatus searches depth information having a high similarity level using a similarity algorithm in the depth search range. The similarity algorithm may be a sum of squared distance (SSD), a sum of absolute (SAD), a normalized correlation coefficient (NCC), a mean absolute difference (MAD), or the like. In operation 340, the moving image data dividing apparatus determines depth information having high similarity within the depth search range using a MAD algorithm.

In operation 350, the moving image data dividing apparatus selects a depth having the highest similarity among the initial depth, the depth of a region, and the depth of a pixel of the depth camera. In operation 360, the moving picture data dividing apparatus adjusts the final depth information existing at the boundary between the object existing in the foreground area and the background area or at the boundary between two objects having different depth information. .

In operation 370, the moving image data dividing apparatus finally projects the selected reference depth information to another viewpoint camera except for the reference viewpoint camera. Since the reference depth information is depth information of the reference viewpoint camera, the reference depth information is projected to obtain depth information of another viewpoint camera except for the reference viewpoint camera. In this case, an occlusion area may occur due to a boundary line between objects in the moving image data or a boundary line between two objects having different depth information. In addition, a hole is generated due to the shielding area. In operation 380, the moving image data dividing apparatus compensates the hole for the depth information of the pixels around the hole. In operation 390, the moving image data dividing apparatus records final depth information of each moving image data captured by the reference viewpoint and the other viewpoint camera as a digital moving image.

Meanwhile, in the detailed description of the present invention, specific embodiments have been described, but various modifications may be made without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined not only by the scope of the following claims, but also by the equivalents of the claims.

1 is a view schematically showing the structure of an apparatus for dividing video data in a video system according to an embodiment of the present invention;

2 is a diagram schematically illustrating a process of dividing video data in a video system according to an exemplary embodiment of the present invention;

3 is a diagram schematically illustrating a process of extracting variation and depth using an apparatus for dividing video data in a video system according to the present invention;

Claims (10)

In the video data splitting apparatus, A storage unit which synchronizes and stores the plurality of video data captured by the plurality of video photographing apparatuses frame by frame; A first extracting unit extracting color information for each frame from the video data; A first separator extracting depth information for each frame from the video data and separating the frames of the video data into a foreground area and a background area, respectively; A second extracting unit extracting motion information for each frame by using the foreground area, the background area, and the color information; And A second separator configured to segment the video data using the color information, the depth information, and the motion information Video data segmentation apparatus comprising a. The method of claim 1, wherein the first extraction unit, And extracting the color information by dividing the color image for each frame from the video data for each color and grouping the color images. The method of claim 1, wherein the first separation unit, And extracting the depth information by calculating a distance from a predetermined viewpoint to an object as a pixel value in the gray image for each frame in the moving image data. The method of claim 1, wherein the first extraction unit, The moving picture data segmentation apparatus may include extracting motion information by comparing color information and depth information of each frame from the moving picture data through the foreground area and the background area. The method of claim 1, wherein the second separation unit, And dividing the moving image data into the foreground region and the background region by using a moving image dividing algorithm based on the color information. The method of claim 1, And a calculator configured to determine a reference frame among the frames of the video data and to average depth information for each frame by using an overlapping area between the reference frame and other frames. The method of claim 6, wherein the calculation unit, The depth information search range for each frame is set using the averaged depth information, and the similarity degree of the depth information is calculated in the set search range. The method of claim 7, wherein the calculation unit, The extracted depth information is updated using the calculated similarity, and the updated depth information is adjusted by mismatching corresponding to an occlusion area between the foreground area and the background area to adjust the depth of the divided video data. Moving picture data segmentation apparatus characterized by calculating information. In the video data segmentation method, A plurality of video data shot by a plurality of video recording devices Extracting color information and depth information for each frame; Dividing the frames of the moving image data into a foreground area and a background area by using the depth information; Extracting motion information for each frame by using the foreground area, the background area, and the color information; Dividing the moving image data using the color information, the depth information, and the motion information; And  Averaging depth information for each frame by using overlapping regions between the frames of the moving image data, and calculating depth information of the divided moving image data using the averaged depth information; Video data segmentation method comprising a. The method of claim 9, wherein the extracting the color information and the depth information comprises: Extracting the color information by separating the color image of each frame from the video data into respective colors and grouping them; And extracting the depth information from the moving image data by calculating a distance from a predetermined viewpoint to an object as a pixel value in each frame gray image.

Images