WO2012050185A1

WO2012050185A1 - Video processing device, method of video processing and video processing program

Info

Publication number: WO2012050185A1
Application number: PCT/JP2011/073639
Authority: WO
Inventors: 健史筑波; 正宏塩井
Original assignee: シャープ株式会社
Priority date: 2010-10-14
Filing date: 2011-10-14
Publication date: 2012-04-19
Also published as: JP5036084B2; JP2012085233A

Abstract

A foreground region correction unit is a video processing device which extracts foreground region information representing an image of a foreground from a video representing video information, wherein the foreground image represented by the foreground region information is corrected during a first time using the foreground region information and video information during a second time.

Description

Video processing apparatus, video processing method, and video processing program

The present invention relates to a video processing device, a video processing method, and a video processing program.
This application claims priority based on Japanese Patent Application No. 2010-231928 filed in Japan on October 14, 2010, the contents of which are incorporated herein by reference.

In recent years, terminals with an imaging function such as a digital video camera, a digital still camera, and a mobile phone are rapidly spreading. In addition, there is an apparatus that generates a new video by processing and processing video shot by these cameras. For example, a specific image area in a video can be extracted, and the extracted image area can be used as a part for video processing, or a video can be extracted for each individual image area, and the extracted video can be managed in the original video. It is known to use for search.
Specifically, the following methods (1) and (2) are known as methods for extracting a desired image region from a video.

(1) Foreground Image Region Extraction Method Based on Color Information As foreground image region extraction methods based on color information, for example, chroma key processing and techniques described in

Non-Patent Documents

1, 2, and 3 are known.
The chroma key process is a process of extracting a desired object (foreground area) by capturing an object with a certain color (for example, blue) as a background and removing a background portion of the certain color from the captured image. With this process, the video is separated into a video in the foreground image area and a video in the background image area.
In Non-Patent Document 1, for a gray image, a user represents a marker representing a foreground area in a desired image area (foreground image area) and a marker representing a background area in another area (background image area) in advance. In addition, a technique for extracting the foreground image region by graph cuts based on the markers assigned to the foreground image region and the background image region is described. Patent Document 2 describes a technique for extracting a foreground image region by applying graph cuts to a color image.
Non-Patent Document 3 discloses a map (trimap) using three markers: a foreground image region, a background image region, and an unknown image region (an image region that has not been determined to belong to either the foreground image region or the background image region). In other words, a technique for extracting the foreground image area by estimating the mixture ratio α (mat) of the pixels in the foreground image area and the background image area in the unknown area is created.

(2) Foreground Image Region Extraction Method Based on Color Information and Depth Information As foreground image region extraction methods based on color information and depth information, for example, techniques described in

Patent Documents

1 and 2 are known.
In Patent Document 1, a trimap is created based on an image (distance image) of depth information from a camera to a subject, and a mixture of pixels in a foreground image region and a background image region in an unknown region using color information. A technique for estimating the rate α and extracting the foreground image region is described.
Patent Document 2 describes a technique for extracting a foreground image area by roughly extracting a foreground image area from depth information from a camera to a subject, and then recursively repeating an area division and integration method based on color information. Yes.

JP 2010-16546 A JP 2009-276294 A

However, in the conventional chroma key processing, when an object (foreground area) includes a constant color or a similar color of the background, it is determined that the area is a background image area, that is, the object is reliably extracted. There was a disadvantage that it was not possible. In addition, the chroma key process has a drawback that when the background is not a constant color, a non-constant color portion of the background image area is extracted as an object, that is, the object cannot be reliably extracted.
In the prior arts described in Non-Patent Document 1-3 and

Patent Documents

1 and 2, when the color distribution of the object and the background area is similar, or when the object and the background area have a similar pattern (texture) The boundary of the region cannot be specified, and a defective portion is generated in the object. In this case, the background area may be erroneously extracted as an object. That is, the conventional technique has a drawback that the object cannot be reliably extracted. When there is a missing part or an erroneously extracted part of the background image area in the target object, in the case of a moving image, the extracted shape of the target object becomes discontinuous in the time direction, so that flicker or Flickering will occur.

The present invention has been made in view of the above points, and provides a video processing apparatus, a video processing method, and a video processing program capable of reliably extracting an image of an object.

(1) The present invention has been made to solve the above problems, and one aspect of the present invention is a video processing apparatus that extracts foreground area information indicating a foreground image from a video indicated by video information. A video processing apparatus comprising: a foreground area correction unit that corrects the foreground image indicated by the foreground area information at the first time using the foreground area information and the video information at the second time.

(2) In addition, according to one aspect of the present invention, in the video processing device, the foreground area correction unit converts the foreground image indicated by the foreground area information at the first time into the foreground area information at the plurality of second times. And correction using video information.

(3) Further, according to one aspect of the present invention, in the video processing device, the video indicated by the video information includes an image of a predetermined object, and the foreground area correction unit performs the first time. Based on the video information, the target image area information indicating the image of the object, and the video information, the foreground area information, and the target image area information at the second time, from the second time to the first time. Based on the movement amount calculated by the movement amount calculation unit and the foreground image based on the movement amount calculation unit that calculates the movement amount that the foreground image has moved in between, the portion in the video at the first time is A foreground region probability map generation unit that calculates a probability of being a foreground image, and foreground region information at the first time is extracted based on the probability calculated by the foreground image probability map generation unit, and the extracted foreground region information indicates Correct foreground images Characterized in that it comprises a Tadashibu, the.

(4) According to another aspect of the present invention, there is provided a video processing method in a video processing apparatus that extracts foreground area information indicating a foreground image from a video indicated by the video information, wherein the foreground area correction unit performs the first time. A foreground area correcting step of correcting the foreground image indicated by the foreground area information in the second time using the foreground area information and the video information at the second time.

(5) According to another aspect of the present invention, in the video processing method described above, the video indicated by the video information includes an image of a predetermined object, and the foreground area correction step includes a movement amount calculation unit The second time based on the video information and the target image area information indicating the image of the object at the first time, and the video information, foreground area information and the target image area information at the second time. A movement amount calculating step for calculating a movement amount by which the foreground image has moved during a first time from a first time, and a movement amount calculated by the foreground region probability map generation unit in the movement amount calculation step and the foreground image Based on the foreground region probability map generation step for calculating the probability that the portion in the video at the first time is the foreground image, and the correction unit calculates the probability based on the probability calculated in the foreground image probability map generation step. Of extracting the foreground area information at the time, a correction step of extracting the foreground area information to correct the foreground image indicated by, and having a.

(6) According to another aspect of the present invention, a foreground image indicated by the foreground area information at the first time is transmitted to a computer of a video processing apparatus that extracts foreground area information indicating a foreground image from an image indicated by the image information. , A video processing program for executing a foreground area correction procedure for correction using foreground area information and video information in a second time period.

(7) According to another aspect of the present invention, in the video processing program described above, the video indicated by the video information includes an image of a predetermined object, and the foreground region correction procedure is performed at the first time. Based on the video information, the target image area information indicating the image of the object, and the video information, the foreground area information, and the target image area information at the second time, from the second time to the first time. Based on the movement amount calculation procedure for calculating the movement amount by which the foreground image has moved in between, the movement amount calculated by the movement amount calculation procedure and the foreground image, the portion in the video at the first time is the foreground. A foreground region probability map generation procedure for calculating a probability of being an image of the foreground, foreground region information at a first time is extracted based on the probability calculated by the foreground image probability map generation procedure, and the foreground region information indicated by the extracted foreground region information is Picture A video processing program for correcting procedure, to the execution of correcting.

According to the present invention, the foreground image area or the background image area can be reliably extracted.

It is a block diagram showing the structure of the video processing apparatus 1 which concerns on embodiment of this invention. It is the schematic which shows an example of the detection process of the user designation ROI information which concerns on this embodiment. It is a flowchart which shows an example of operation | movement of the video processing apparatus which concerns on this embodiment. It is a schematic block diagram which shows the structure of the object extraction part which concerns on this embodiment. It is a schematic block diagram which shows the structure of the clustering part which concerns on this embodiment. It is the schematic which shows an example of the process result of the clustering part which concerns on this embodiment. It is a flowchart showing an example of operation | movement of the feature-value calculation part which concerns on this embodiment. It is the schematic showing an example of the labeling (area | region information) which concerns on this embodiment. It is the schematic showing an example of the expression by the unweighted undirected graph and the adjacency matrix about the connection relation between the areas. It is the schematic which shows an example of the acquisition method of the perimeter of the area | region which concerns on this embodiment, and the surrounding rectangle of an area | region. It is a flowchart which shows an example of operation | movement of the foreground area | region extraction part which concerns on this embodiment. It is a schematic block diagram which shows the structure of the foreground area | region correction | amendment part which concerns on this embodiment. It is explanatory drawing explaining the template matching which concerns on this embodiment. It is explanatory drawing explaining the spiral search which concerns on this embodiment. It is a flowchart which shows an example of operation | movement of the foreground area correction part which concerns on this embodiment. It is a flowchart which shows an example of operation | movement of the buffer part which concerns on this embodiment. It is a flowchart which shows an example of operation | movement of the object extraction part which concerns on this embodiment. It is the schematic which shows an example of the depth information which concerns on this embodiment. It is the schematic which shows an example of the foreground area | region probability map P which concerns on this embodiment. It is explanatory drawing of an example of a foreground image area | region which concerns on this embodiment. It is the schematic which shows another example of the detection process of the user designation ROI information which concerns on this embodiment. It is the schematic which shows another example of the detection process of the user designation ROI information which concerns on this embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of a video processing apparatus 1 according to an embodiment of the present invention. In this figure, a video processing apparatus 1 includes a video information acquisition unit 10, a depth information acquisition unit 11, a video information reproduction unit 12, a ROI (Region of Interest; target image region) acquisition unit 13, a video display unit 14, and an object extraction unit. 15 and a mask information recording unit 16.

The video information acquisition unit 10 acquires video information (t). Here, the video information (t) is video information of a moving image and is a function of time t (elapsed time from the start time of the moving image). However, the video information of the present invention is not limited to this, and may be video information of a plurality of still images. For example, the video information (t) may be a moving image or a still image including images that are continuous or adjacent in time with the imaging device fixed, or images captured from consecutive or adjacent positions at the same time. (In the latter case, video information is a function of position). The video information acquisition unit 10 may acquire the video information (t) from the imaging device, or may acquire the video information (t) recorded in advance in a recording unit or an external recording device. .
The video information acquisition unit 10 outputs the acquired video information (t) to the video information reproduction unit 12, the ROI acquisition unit 13, and the object extraction unit 15.

The depth information acquisition unit 11 acquires depth information (t) of the video information (t) acquired by the video information acquisition unit 10. Here, the depth information (t) is information representing the distance from the imaging device to the imaging object for each pixel of the video information (t).
The depth information acquisition unit 11 outputs the acquired depth information (t) to the object extraction unit 15.

The video information reproduction unit 12 generates a video signal for controlling the output of each pixel at each time t of the video display unit 14 based on the video information (t) input from the video information acquisition unit 10. The video information reproduction unit 12 displays the video on the video display unit 14 by outputting the generated video signal to the video display unit 14. That is, the video information reproduction unit 12 reproduces the video of the video information (t) and causes the video display unit 14 to display the reproduced video.
Here, the video information reproducing unit 12 superimposes the image of the object extracted by the object extracting unit 15 on the video of the video information (t) based on the mask information (t) recorded by the mask information recording unit 16. indicate. That is, the mask information (t) is associated with the video information (t) at time t. When the mask information recording unit 16 does not record the mask information (t), the video of the video information (t) is reproduced as it is.

The video display unit 14 is a touch panel type display. The video display unit 14 displays the video of the video information (t) by controlling the output based on the video signal input from the video information reproducing unit 12. When the user touches the display, the video display unit 14 converts information indicating the touched position into information indicating the position of the video information (ts) in the image (ts) at a certain time ts.
The video display unit 14 specifies the ROI in the image (ts) displayed on the display while the user touches the display, whereby the ROI position information (referred to as user-specified ROI information (ts)), that is, the position of the ROI. And information indicating the shape (circumscribed shape) are detected. The details of the process for detecting the user-designated ROI information (ts) in the video display unit 14 will be described later.
The video display unit 14 outputs the detected user-specified ROI information (ts) to the ROI acquisition unit 13.

The ROI acquisition unit 13 is based on the image within the range of the user-specified ROI information (ts) input from the video display unit 14, and the image of the video information (t) at each time t other than the time ts (the image of each frame). ; Hereinafter referred to as processed image (t)), an image that matches or is similar to the image of the user specified ROI information (ts) is detected. Thereafter, the ROI acquisition unit 13 extracts information indicating the position and shape of the matching or similar image as ROI information (t).

Specifically, the ROI acquisition unit 13 calculates and records feature points (referred to as ROI feature points (ts)) from an image within the range of the user-specified ROI information (ts). A feature point is a characteristic point in an image, for example, a point that is extracted as a part or a vertex of a subject based on a change in color or luminance between pixels. Is not limited to this. The ROI acquisition unit 13 calculates an image feature point (t) of the processed image (t) at each time t.
The ROI acquisition unit 13 performs matching between the image feature point (t) and the ROI feature point (ts). Specifically, the ROI acquisition unit 13 sequentially multiplies the ROI feature point (ts) by a transformation matrix, thereby moving (including rotating) the ROI feature point (ts) and enlarging / reducing the feature. The number of points that match (referred to as the number of feature points) is calculated. When the ROI acquisition unit 13 determines that the number of feature points is equal to or greater than a predetermined threshold, the ROI acquisition unit 13 records the conversion matrix at that time. The ROI acquisition unit 13 sets the position information obtained by multiplying the user-specified ROI information (ts) by the transformation matrix as ROI information (t). That is, the ROI acquisition unit 13 determines which part in the image (t) the image within the range of the user-specified ROI information (ts) matches. If the images match, the ROI information (ts) The position information corresponding to is ROI information (t). The ROI acquisition unit 13 outputs the extracted ROI information (t) (including user-specified ROI information (ts)) to the video display unit 14 and the object extraction unit 15. The ROI acquisition unit 13 stores the extracted ROI information (t) in the ROI information storage unit 1583.

The object extraction unit 15 receives video information (t) from the video information acquisition unit 10, receives depth information (t) from the depth information acquisition unit 11, and receives ROI information (t) from the ROI acquisition unit 13. . The object extraction unit 15 generates mask information (t) at each time t using the input video information (t), depth information (t), and ROI information (t). Details of processing performed by the object extraction unit 15 will be described later.
The object extraction unit 15 records the extracted mask information (t) in the mask information recording unit 16.

<Detection processing of user-specified ROI information (ts)>
The details of the detection process of the user-specified ROI information (ts) performed by the video display unit 14 will be described below.
FIG. 2 is a schematic diagram illustrating an example of detection processing for user-specified ROI information (ts) according to the present embodiment. In FIG. 2, a square with a symbol A is a touch panel display (video display unit 14). A square with a symbol B is an image (ts) at a time ts (in FIG. 2, ts = 0.133 seconds) indicated by the time information. What attached | subjected the code | symbol O represents the image of the target object (a person in FIG. 2) which a user wants to extract. Moreover, what attached | subjected the code | symbol U represents a user's hand.

FIG. 2 is a diagram when a rectangular (quadrangle) selection tool is used, and shows that the user has surrounded the target image to be extracted with the rectangular selection tool. In this figure, the position information of the frame (the circumscribed rectangle of the object O) labeled with the symbol r1 is user-specified ROI information (ts). Specifically, in the case of FIG. 2, the user-specified ROI information (ts) is recorded, for example, as data in Table 1 below.

In Table 1, the user-specified ROI information (ts) includes time ts (or a number (frame number) attached to a video frame), and whether or not there is an extraction target image (target image) in a circumscribed rectangle. Presence / absence flag (referred to as extraction target flag), circumscribed rectangle starting point position (x0, y0) (coordinate of point P1 in FIG. 2), circumscribed rectangle width (length indicated by reference sign W1 in FIG. 2), circumscribed rectangle Is represented by the vertical width (the length represented by the symbol L1 in FIG. 2).

<Operation of Video Processing Device 1>
Hereinafter, the operation of the video processing apparatus 1 will be described.
FIG. 3 is a flowchart showing an example of the operation of the video processing apparatus 1 according to the present embodiment.

(Step S11) The video information acquisition unit 10 acquires video information (t), and outputs the acquired video information (t) to the video information playback unit 12, the ROI acquisition unit 13, and the object extraction unit 15. The depth information acquisition unit 11 acquires depth information (t), and outputs the acquired depth information (t) to the object extraction unit 15. Thereafter, the process proceeds to step S12.

(Step S12) The video information reproducing unit 12 reproduces the video of the video information (t) input in step S11, and causes the video display unit 14 to display the reproduced video. Thereafter, the process proceeds to step S13.
(Step S13) The user pauses the reproduction of the video reproduced in step S12 at a certain time ts and designates the ROI. The video display unit 14 detects user-specified ROI information (ts) for the ROI specified by the user, and outputs it to the ROI acquisition unit 13. Then, it progresses to step S14.

(Step S14) The ROI acquisition unit 13 extracts the ROI information (t) at each time t based on the user-specified ROI information (ts) detected in Step S13. The ROI acquisition unit 13 outputs the extracted ROI information (t) to the video display unit 14 and the object extraction unit 15. The video display unit 14 displays the circumscribed shape indicated by the ROI information (t) (the circumscribed rectangle in the case of Table 1 and the circumscribed circle in the case of Table 2) superimposed on the image of the video information (t) at that position. To do. Then, it progresses to step S2.

(Step S2) The object extraction unit 15 performs object extraction using the video information (t) and depth information (t) acquired in step S11, and the ROI information (t) extracted in step S14, and mask information (T) is generated. The object extraction unit 15 records the generated mask information (t) in the mask information recording unit 16. Thereafter, the process proceeds to step S15.
(Step S15) Based on the mask information (t) recorded by the mask information recording unit 16, the video information reproducing unit 12 superimposes the object image extracted by the object extracting unit 15 on the video of the video information (t). To display.

In steps S14, S2, and S15, the video processing device 1 may perform processing related to all input video information (t) at time t, or video information (t in a range specified by the user) ) (T1 ≦ t ≦ t2) may be performed.
In the above operation, the video processing apparatus 1 detects the user-specified ROI information (ts) for the ROI specified by the user, so that the ROI can be reliably extracted as compared with the case where the ROI is automatically extracted. it can. In the video processing apparatus 1, the video display unit 14 displays the circumscribed shape indicated by the ROI information (t) so as to be superimposed on the video of the video information (t), so that the user has detected a desired ROI. I can understand that.

<About processing performed by the object extraction unit 15>
Hereinafter, the process performed by the object extraction unit 15 will be described in detail.
FIG. 4 is a schematic block diagram illustrating the configuration of the object extraction unit 15 according to the present embodiment. In this figure, the object extraction unit 15 includes filter units 151a and 151b, a distribution model estimation unit 152, a clustering unit 153, a feature amount calculation unit 154, a foreground region extraction unit 155, a foreground region correction unit 156, a mask information generation unit 157, And a buffer unit 158. The buffer unit 158 includes a video information storage unit 1581, a foreground region information storage unit 1582, an ROI information storage unit 1583, an ROI depth distribution information storage unit 1584, and a corrected foreground region information storage unit 1585. Note that parallelograms denoted by reference signs I1, D1, R1, and M indicate information, and are video information (t), depth information (t), ROI information (t), and mask information (t), respectively.

The filter unit 151a removes noise from the input video information (t) and performs a smoothing process. Specifically, the filter unit 151a uses a smoothing filter (hereinafter also referred to as an edge holding smoothing filter) that holds an edge (contour) for each color component with respect to the processed image (t) at each time t. Do. Here, the filter unit 151a uses a bilateral filter represented by the following equation (1) as a smoothing filter.

However, the input image is f (x, y), the output image is g (x, y), W is the window size to which filtering is applied, σ ₁ is a parameter for controlling the weighting coefficient related to the inter-pixel distance (standard deviation of Gaussian distribution) ), Σ ₂ represents a parameter (standard deviation of Gaussian distribution) for controlling a weighting coefficient relating to a difference in pixel values.
The filter unit 151 a outputs the video information (t) smoothed by the smoothing filter to the clustering unit 153 and the feature amount calculation unit 154.

The filter unit 151b removes noise from the input depth information (t) and performs a smoothing process. Specifically, the filter unit 151b performs an edge holding smoothing filter. Thereby, the filter part 151b removes the noise in the horizontal direction generated by occlusion (shielding).
The filter unit 151 b outputs the depth information (t) smoothed by the smoothing filter to the feature amount calculation unit 154 and the distribution model estimation unit 152.

Based on the smoothed depth information (t) input from the filter unit 151b and the input ROI information (t) for each processed image unit, the distribution model estimation unit 152 calculates the depth distribution model in the ROI. Estimate the parameters. Specifically, the distribution model estimation unit 152 acquires the parameters of the distribution model by the maximum likelihood estimation method using the following equation (2), that is, GMM (Gaussian Mixture Model) expressed by a mixed model of Gaussian distribution. . Hereinafter, the acquired parameter is referred to as ROI depth distribution information (t).

In Expression (2), P (x) represents the probability that the vector x appears. w _i represents a weighting factor of a Gaussian distribution of class i, μ _i represents an average vector of class i, Σ _i represents a covariance matrix of class i, and D represents the number of dimensions of vector x. N (x | μ _i , Σ _i ) represents a Gaussian distribution of class i, and is expressed using a mean vector μ _i and a covariance matrix Σ _i . The distribution model estimation unit 152 obtains each parameter of the distribution model using an EM (Expectation-Maximization) algorithm. That is, the distribution model estimation unit 152 determines the depth distribution of the extraction target region in the ROI as the distribution of the class having the maximum weighting coefficient w _i . That is, the depth distribution is calculated by assuming that the area occupied by the area in the ROI is large.
The distribution model estimation unit 152 outputs the estimated ROI depth distribution information (t) to the foreground region extraction unit 155 and the buffer unit 158.

The clustering unit 153 performs clustering on the smoothed video information (t) input from the filter unit 151a for each processed image (t), thereby processing the processed image (t) into a plurality of regions (superpixels). (Also called).
For example, the clustering unit 153 performs clustering in the feature amount space. The clustering by the feature amount space means that each pixel of the image space is mapped to the feature amount space (for example, color, edge, motion vector), and the K-means method, the Mean-Shift method, or the K nearest neighbor in the feature amount space. Clustering is performed by a technique such as a search method (approximate K nearest neighbor search method). That is, the clustering unit 153 divides the processed image (t) into a set (region; class) of pixels having similar feature quantities (feature quantity values are within a predetermined range).
The clustering unit 153 replaces the pixel value in the original image space for the pixels in the class with the pixel value (for example, the average value) that is the representative value of each region after the clustering process in the feature amount space is completed. The clustering unit 153 assigns a label for identifying the region to each pixel in each region, and outputs region information (t).
Details of the clustering unit 153 will be described below.

FIG. 5 is a schematic block diagram illustrating the configuration of the clustering unit 153 according to the present embodiment. In this figure, the clustering unit 153 includes a feature amount detection unit 1531, a seed generation unit 1532, a region growth unit 1533, and a region integration unit 1534. The clustering unit 153 performs clustering using feature values based on edges and colors, but the present invention is not limited to this feature value.

The feature amount detection unit 1531 receives the smoothed video information (t). The feature amount detection unit 1531 calculates the feature amount of the pixel (x, y) (x and y are coordinates representing the position of the pixel in the image) in each processed image (t). Specifically, the feature amount detection unit 1531 applies a differential operator for each color component (for example, RGB (Red (red), Green (green), Blue (blue))), and each color in the x direction and the y direction. The gradient of the component i (gradient) G _i (x, y | t) = (ΔG _ix (t), ΔG _iy (t)) (i = 1, 2, 3) is calculated. For example, i = 1, 2, and 3 are an R component, a G component, and a B component, respectively.
The feature amount detection unit 1531 calculates the edge strength E ₂ (x, y | t) by performing the calculation of the following equation (3) for each pixel of the coordinates (x, y).

Here, TH_E (x, y | t) is a predetermined threshold for the coordinates (x, y) at time (t). Equation (3) indicates that E ₂ (x, y | t) = 0 when E ₁ (x, y | t) is smaller than TH_E (x, y | t), and If E ₁ (x, y | t) is equal to or greater than TH_E (x, y | t), it indicates that E ₂ (x, y | t) = E ₁ (x, y | t). Note that the feature amount detection unit 1531 may adjust the threshold value TH_E (x, y | t) in pixel units, block units, region units, and image units. As a result, it is possible to detect a feature amount suitable for the image characteristics.
The feature amount detection unit 1531 outputs the calculated edge strength E ₂ (x, y | t) to the seed generation unit 1532.

The seed generation unit 1532 generates seed information for generating a super pixel using the edge intensity E ₂ (x, y | t) input from the feature amount detection unit 1531. Specifically, the seed generation unit 1532 calculates seed information S (x, y | t) using the following equation (4).

That is, the seed generation unit 1532 has a minimum value of the edge strength E ₂ (x, y | t) within the range of the window size W ₁ × W _{2 with} the coordinates (x, y) as the center (Local Minima). ), The seed information S (x, y | t) is set to “1”, and otherwise, the seed information S (x, y | t) is set to “0”. Here, W ₁ represents the size of the window in the x direction, and W ₂ represents the size of the window in the y direction.
The seed generation unit 1532 outputs the generated seed information S (x, y | t) to the region growth unit 1533.

The region growing unit 1533 applies the region growing method based on the seed information S (x, y | t) input from the seed generating unit 1532 to generate the super pixel group R ₁ (t). Specifically, region growing unit 1533, the edge strength of a pixel (x1, y1) _E 2 | and (x1, y1 t), pixels near the pixel (x, y) of the edge intensities _E 2 (x , Y | t) and the pixel (x, y) whose difference is equal to or less than a predetermined value is set to the same region as the original pixel (x1, y1), thereby expanding the region. Here, the region growing unit 1533 starts this processing from a pixel whose seed information S (x, y | t) is “1”. That is, the region growing unit 1533 gradually grows a region having almost the same feature value from the pixel having the seed information S (x, y | t) of “1”.
The region growing unit 1533 sets the expanded region as the super pixel group R ₁ (t), and outputs information indicating the super pixel group R ₁ (t) to the region integrating unit 1534.

The region integration unit 1534 performs region integration processing for integrating a super pixel group having a small area area into another region from the super pixel group R ₁ (t) indicated by the information input from the region growth unit 1533. Specifically, the region integration unit 1534 uses one point of each superpixel group R ₁ (t) as a vertex, and is represented by a connection relationship between the vertices and an edge (weight) between the connected vertices. A direction graph is calculated. Here, for an edge (weight) between vertices, for example, a distance in a color space between each vertex and a representative color of each corresponding superpixel is used.
The region integration unit 1534 performs region integration on the weighted undirected graph so as to form a minimum spanning tree (MST) using a greedy method, and generates a super-cell group R ₂ (t). For each superpixel, the region integration unit 1534 assigns a label for identifying the superpixel to all the pixels in the superpixel, and outputs the labeling result as region information (t).

FIG. 6 is a schematic diagram illustrating an example of a processing result of the clustering unit 153 according to the present embodiment. This figure is an image of a bear doll riding a train toy. In addition, bear toys are also moving as train toys move along the tracks.
In this figure, FIG. 6A by reference numeral 6A is an image representing the edge strength E ₂ obtained from the input image. Further, FIG. 6B with numeral 6B is an image representing the seed information S obtained from the edge intensity E _2. Further, FIG. 6C attached with reference numeral 6C is an image showing the super pixel group R ₁ (t), and FIG. 6D attached with reference numeral 6D is an image showing the super pixel group R ₂ (t).
In FIG. 6A, a bright (white) portion represents a region having a high edge strength, and a dark (black) portion represents a region having a low edge strength. In FIG. 6B, a bright (white) portion represents a seed, and a dark (black) portion represents a portion that is determined by a region growth method to which region (class) belongs.
Comparing FIG. 6C and FIG. 6D, there are a large number of superpixels (regions) having a small area in the superpixel group R ₁ (t) in FIG. 6C, but small in the superpixel group R ₂ (t) in FIG. 6D. The number of superpixels (areas) having an area is decreasing. As described above, the video processing apparatus 1 can obtain a more accurate clustering result by performing the clustering process with fewer superpixels (regions) having a small area.

Returning to FIG. 4, the feature amount calculation unit 154 receives the region information (t) from the clustering unit 153, the smoothed video information (t) from the filter unit 151a, and the depth information (t) from the filter unit 151b. ) And ROI information (t). The feature amount calculation unit 154 calculates a feature amount for each region (label) based on the input region information (t), video information (t), depth information (t), and ROI information (t). Specifically, the feature amount calculation unit 154 calculates the following feature amounts (1) to (7). Thereafter, feature amount information (t) indicating the calculated feature amount is output to the foreground region extraction unit 155.
(1) Adjacent relationship between regions (2) Distance between the centroid of the ROI region and the centroid of each region (hereinafter referred to as the centroid distance)
(3) Average value, median, variance, and standard deviation of each color component (4) Average value, median, variance, and standard deviation of depth (5) Area area (6) Area circumference (7) circumscribed rectangle of area

<Operation of Feature Quantity Calculation Unit 154>
An example of the feature value calculation method (1) to (7) will be described with reference to FIGS. FIG. 7 is a flowchart illustrating an example of the operation of the feature amount calculation unit 154. FIG. 8 is a diagram illustrating an example of labeling (region information) in an 8 × 8 pixel block for explanation. FIG. 9 is a diagram illustrating an example of a connection relation between regions in FIG. 8 expressed by an unweighted undirected graph and an adjacency matrix. FIG. 10 is a diagram illustrating an example of the method of obtaining the perimeter of the region and the circumscribed rectangle of the region, taking the label 3 in FIG. 8A as an example.

(Step S154-01) The feature amount calculation unit 154 receives the region information (t) from the clustering unit 153, the video information (t) after smoothing from the filter unit 151a, and the depth information (t) after smoothing from the filter unit 151b. , And ROI information (t). Thereafter, the process proceeds to step S154-02.

(Step S154-02) The feature amount calculation unit 154 sums the pixel coordinate values for the pixels in the ROI region based on the ROI information (t). Subsequently, the feature amount calculation unit 154 divides the total value by the number of pixels in the ROI area, and uses the calculation result as the center of gravity of the ROI area. Thereafter, the process proceeds to step S154-03.

(Step S154-03) Based on the region information (t), the feature amount calculation unit 154 scans the processing target image line by line from the origin (raster scan), and coordinates and the number of pixels of all pixels belonging to each label The position (start point) where the pixel belonging to each label first appears and the adjacency relationship between the labels (regions) are obtained. Further, the position (start point) at which the pixel belonging to each acquired label first appears is stored as the start point of contour tracking when acquiring the area perimeter in step S154-08. Thereafter, the process proceeds to step S154-04. Here, an example of a method for obtaining the detection result of the start position of the label and the adjacent relationship between the labels will be described with reference to FIGS. For example, it is assumed that there is a labeling result (region information) in an 8 × 8 pixel block shown in FIG. In the case of FIG. 8A, when scanning is performed line by line from the origin of the 8 × 8 pixel block (upper left in FIG. 8A), the reference symbol Li (i = 1, 2,. .., 5) are detected as the starting point of each label. In addition, the connection relationship between the regions is acquired by comparing the label of the pixel with the label of the pixel adjacent to the pixel and sequentially detecting different labels in each pixel. For example, in the case of a pixel of interest (reference numeral Q) shown in FIG. 9A with reference numeral 9A, the label of the pixel is 1, and a pixel adjacent to the pixel (adjacent pixel: reference numeral Ni (i = 1, 2,... .., 8)), the different labels are 3, 4. As a result, it can be understood that the label 1 is adjacent to the

labels

3 and 4. By performing the above processing for all the pixels, the adjacency relationship between the labels regarding the region information in FIG. 8A can be finally expressed as an unweighted undirected graph shown in FIG. In FIG. 9B, each node number corresponds to each label number, and an edge between nodes represents a connection relationship. The structure of the graph of FIG. 9B is expressed by the adjacency matrix shown in FIG. In the adjacency matrix of FIG. 9C, “1” is assigned as a value when there is an edge between nodes, and “0” is assigned as a value when there is no edge between nodes.

(Step S154-04) The feature amount calculation unit 154 sums the coordinate values for the pixels belonging to each label Li (0 ≦ i <MaxLabel). Here, MaxLabel represents the total number of labels for identifying superpixels (regions) acquired from the region information (t). Subsequently, the feature amount calculation unit 154 divides the total value by the number of pixels belonging to the label Li, and uses the result as the center of gravity of the label Li. Then, the distance (center of gravity distance) between the center of gravity of the ROI area obtained in step S154-02 and the center of gravity of the label Li is calculated. Thereafter, the process proceeds to step S154-05.

(Step S154-05) The feature amount calculation unit 154 calculates the average value, median value, variance value, and standard value for each color component of each label Li (0 ≦ i <MaxLabel) from the smoothed video information (t). Calculate the deviation. Thereafter, the process proceeds to step S154-06.

(Step S154-06) The feature amount calculation unit 154 calculates the average value, median value, variance value, and standard deviation of the depth of each label Li (0 ≦ i <MaxLabel) from the smoothed depth information (t). calculate. Thereafter, the process proceeds to step S154-07.

(Step S154-07) The feature amount calculation unit 154 sets the total number of pixels belonging to the label Li as a region area. Thereafter, the process proceeds to step S154-08.

(Step S154-08) The feature amount calculation unit 154 calculates the area perimeter of the label Li. Thereafter, the process proceeds to step S154-09. Here, an example of a method for calculating the area perimeter will be described using the label 3 as an example, with reference to FIG. The area peripheral length is a movement amount that goes around the area clockwise (or counterclockwise) from the start point of the label in FIG. 10A. 8 when connected components of the coupling, and C ₁ number to track moving up and down and right and left, there are a C ₂ to track moving obliquely, region perimeter (Perimeter) has the formula (5) or by the formula (6) Calculated.

(Step S154-09) The feature amount calculation unit 154 calculates the minimum rectangle (circumscribed rectangle) circumscribing the area indicated by the label Li. Thereafter, the process proceeds to step S154-10. Here, an example of a circumscribed rectangle acquisition method will be described with reference to FIG. In FIG. 10B, the coordinates of the vertices of the four points constituting the enclosing rectangle of the label 3 (code P1, P2, P3, P4), (represented as _{X L)} X coordinate of the pixel at the left edge in the region, at the far right X coordinate of the pixel (represented as _{X R),} Y coordinates of the pixels in the upper end _(Y T), and with the Y coordinate of the pixel (represented as _{Y B)} at the lower end,
_{_{_{_{P1: = (X L, Y}}}} T), P2: = (X R, Y T), P3: = (X R, Y B), P4: = (X L, Y B), and is expressed. Further, X _L , X _R , Y _{T and} Y _B are calculated by the equation (7). In Equation (7), the symbol Li represents a label number, the symbol R _Li represents a set of pixels belonging to the label Li, and the symbols x _j and y _j represent the pixels j belonging to the set R _Li , respectively. Represents the x coordinate and y coordinate.

(Step S154-10) When the calculation of the feature values of all the labels is completed (Yes in Step S154-10), the feature value calculation unit 154 obtains the feature value information (t) including the feature value of each label (region). The data is output to the foreground area extraction unit 155. If there is an unprocessed label for which the feature amount is calculated (No in step S154-10), the process returns to step S154-04 to calculate the feature amount of the next label.

In this way, the feature quantities (1) to (7) can be calculated.
In the present embodiment, the operation of the feature amount calculation unit 154 has been described in the order of steps S154-01 to S154-10. However, the present invention is not limited to this, and can be changed within a range in which the present invention can be implemented. . In this embodiment, an adjacency matrix is used as an example of a data structure representing a connection relationship between regions, but the present invention is not limited to this, and an adjacency list may be used. The feature amount calculation unit 154 calculates the feature amount using RGB as the color space of the image. However, the present invention is not limited to this, and YCbCr (YUV), CIE L * a * b * (Elster, Aster) , Biester), CIE L * u * v * (Elster, Yuster, Baster) or other color space.

The foreground region extraction unit 155 receives the feature amount information (t) from the feature amount calculation unit 154, the ROI depth distribution information (t), and the ROI information (t) from the distribution model estimation unit 152. Further, the foreground area extraction unit 155 reads the video information (t) from the video information storage unit 158 of the buffer unit 158. Here, the read video information (t) is information stored by the video information acquisition unit 10 and not subjected to the smoothing process, but is not limited thereto, and the video information subjected to the smoothing process is not limited thereto. (T) may be used.
The foreground region extraction unit 155 extracts the foreground image region (t) to be extracted from the video information (t) based on the ROI information (t), the feature amount information (t), and the ROI depth distribution information (t). To do. The foreground area extraction unit 155 stores the foreground area information (t) indicating the extracted foreground image area (t) in the foreground area information storage unit 1582. The foreground area extraction unit 155 also obtains information indicating the foreground image area (t0-k) at the time t0-k (k = 1, 2,..., K; k indicates the time of the frame). After storing in the storage unit 1582, information indicating the time t0 is output to the foreground region correction unit 156.

Details of the processing performed by the foreground area extraction unit 155 will be described below.
FIG. 11 is a flowchart showing an example of the operation of the foreground area extraction unit 155 according to the present embodiment.
(Step S155-01) The foreground area extraction unit 155 sets parameters for search conditions for searching for a basic foreground area, which is a core area for extracting the foreground area. Specifically, the foreground region extraction unit 155 sets predetermined values as the lower limit value and the upper limit value of each feature amount information (t). Foreground region extraction unit 155, for example, lower limit value (referred to as minimum area) of region area, upper limit value (referred to as maximum area), lower limit value of region peripheral length (referred to as minimum peripheral length), maximum value (referred to as maximum peripheral length), As an initial value of the upper limit value of the center-of-gravity distance, the maximum value (maximum distance) of the radius of the circle inscribed in the circumscribed rectangle of the ROI area is set as a parameter. By setting the search conditions for the basic foreground region in this way, it is possible to detect a region having a center of gravity and a large area within the ROI. In addition, it is possible to prevent erroneous detection of a region belonging to the background region as a basic foreground region. Thereafter, the process proceeds to step S155-02.
(Step S155-02) The foreground region extraction unit 155 selects a region having the minimum centroid distance from among the regions that satisfies the lower limit value of the centroid distance or less than the upper limit value. Thereafter, the process proceeds to step S155-03.

(Step S155-03) The foreground region extraction unit 155 determines whether or not the feature amount information (t) of the region selected in step S155-02 is a value between the lower limit value and the upper limit value set in step S155-01. Determine whether. When it is determined that the feature amount information (t) is a value between the lower limit value and the upper limit value (Yes), that is, it is determined that the region selected in step S155-02 is the basic foreground region, and step S155- Proceed to 05. On the other hand, when it is determined that the feature amount information (t) is not a value between the lower limit value and the upper limit value (No), the process proceeds to step S155-04.
(Step S155-04) The foreground region extraction unit 155 subtracts a predetermined value from the lower limit value of each feature amount information (t), or adds a predetermined value to the upper limit value, so that the feature amount information (T) Each lower limit value and upper limit value are updated. Thereafter, the process proceeds to step S155-02.

(Step S155-05) The foreground region extraction unit 155 calculates the average value related to the depth among the feature amount information (t) of the basic foreground region determined in Step S155-03 and each region having a center of gravity within the ROI, or The median values are compared, and it is determined whether or not the difference between the pieces of feature amount information (t) is within a predetermined threshold value (or less than a predetermined threshold value). The foreground area extraction unit 155 integrates the area for which the difference in the feature amount information (t) is determined to be within a predetermined threshold and the basic foreground area, and determines the integrated area as the foreground area.
Here, the foreground region extraction unit 155 determines a threshold value of the difference between the feature amount information (t) based on the ROI depth distribution information (t) acquired from the distribution model estimation unit 152. Specifically, the foreground region extraction unit 155 calculates a threshold value (TH_D1) of the difference between the feature amount information (t) using the following equation (8).

Here, α_1 is a predetermined scaling constant. Σ_1 is a standard deviation when the depth distribution of the foreground region is assumed to be a Gaussian distribution.

Returning to FIG. 4, the foreground area correction unit 156 sets the foreground image area (t0) at the time t0 (first time) indicated by the information input from the foreground area extraction unit 155 to the time t0− Correction is performed based on the foreground image area (t0-k) of k (second time) (k = 1, 2,..., K). The foreground area correction unit 156 repeats this correction sequentially from t0 to time t0. The foreground area correction unit 156 stores information (corrected foreground image area information) indicating the corrected foreground image area (t0) in the foreground area information storage unit 1585 of the buffer unit 158. Further, the foreground area correction unit 156 outputs information (corrected foreground image area information) indicating the corrected foreground image area (t0) to the mask information generation unit 157.
Hereinafter, the correction performed by the foreground area correction unit 156 will be described in detail.

FIG. 12 is a schematic block diagram showing the configuration of the foreground area correction unit 156 according to the present embodiment. In this figure, the foreground region correction unit 156 includes a movement amount calculation unit 1561, a foreground region probability map generation unit 1562, a foreground region determination unit 1563, and a boundary region correction unit 1564.

Information indicating the time t0 is input from the foreground region extraction unit 155 to the movement amount calculation unit 1561. The movement amount calculation unit 1561 includes information from time t0 to time t0-K (video information (t0-k), video information (t0), foreground area information (t0-k), foreground area information (t0), ROI information ( t0-k), ROI information (t0), (k = 1, 2,..., K)) is read from the buffer unit 158.
Based on the read video information (t0-k), foreground area information (t0-k), and ROI information (t0-k), the movement amount calculation unit 1561 calculates the foreground area indicated by the foreground area information (t0-k). For the image (t0-k), a movement amount (t0, t0-k) (also referred to as a motion vector) obtained by subtracting the position in the processed image (t0-k) from the position in the processed image (t0) is calculated. That is, the movement amount (t0, t0-k) represents the movement amount that the foreground area image (t0-k) has moved from time t0-k to time t0. Specifically, the movement amount calculation unit 1561 calculates the movement amount (t0, t0-k) by performing template matching (also referred to as motion search) processing shown in FIG.

FIG. 13 is an explanatory diagram for explaining template matching according to the present embodiment. In this figure, the horizontal axis is time t, the vertical axis is the y coordinate, and the direction perpendicular to the horizontal axis and the vertical axis is the x coordinate. Also, the image with the symbol Ik represents the processed image at time (t0-k). An image region denoted by reference symbol Ok represents a foreground image region at time (t0-k) and a circumscribed rectangle surrounding the foreground image region (object). In addition, the coordinates with the reference symbol Ak represent the coordinates of the starting point position of the circumscribed rectangle surrounding the foreground area indicated by the reference symbol Ok. The image with the symbol Mk is an image indicated by the mask information (t0-k) in the circumscribed rectangle at time (t0-k). Here, the mask information (t0-k) is information for identifying the foreground image region (white portion in FIG. 13) and the background image region (black portion in FIG. 13) within the circumscribed rectangle. The foreground image region (t0-k) indicated by t0-k) and the other region in the circumscribed rectangle are set as the background image region (t0-k). A symbol Vk is a vector from the coordinate Ak to the coordinate A0. This vector represents the amount of movement (t0, t0-k) of the foreground image area (t0-k).

The movement amount calculation unit 1561 uses the foreground image region Ok as a template, moves the template on the processed image (t0) (may be rotated or enlarged / reduced), and has the highest similarity to the template ( Detected area). The movement amount calculation unit 1561 calculates a difference in coordinates between the detected estimated area and the foreground image area Ok as a movement amount (t0, t0-k).

Specifically, the movement amount calculation unit 1561 calculates coordinates (x0, y0) (referred to as initial search coordinates) of the center of gravity of the ROI area indicated by the ROI information (t0). The movement amount calculation unit 1561 performs a spiral search around the initial search coordinates (x0, y0) to detect the estimated region and calculates the movement amount (t0, t0-k). Here, the spiral search is a coordinate in which the range of the foreground image area (t0) is high (in this case, the search initial coordinates) so as to gradually expand the range in a spiral order as shown in FIG. It is a technique for moving and searching for an estimated area. When the movement amount calculation unit 1561 extracts a movement amount having a similarity higher than a predetermined value, the spiral search may end there. Thereby, the movement amount calculation unit 1561 can reduce the calculation amount.
The movement amount calculation unit 1561 calculates the similarity R _SAD using the following formula (9) using the coordinates selected in the spiral order (referred to as selected coordinates) as the center of gravity, and determines the region having the smallest value as the estimation region.

However, M × N (W1 × L1 in the example of FIG. 2) represents the size of the template, (i, j) represents the coordinates of the pixels in the template, and T (i, j | t0−k) is The pixel value at the position of coordinates (i, j) is represented. Further, (dx, dy) is a value (offset value) obtained by subtracting the center of gravity of the ROI area indicated by the ROI information (t0-k) from the selected coordinates, and I (i + dx, j + dy | t0) is in the processed image (t0). The pixel value at the coordinates (i + dx, j + dy) is represented. Equation (9) indicates that the absolute value is calculated by the Manhattan distance (L ₁ -distance, L ₁ -norm) and the sum of i and j is taken.
When the color space is RGB, I (x, y | t0) is the value of each color component in the RGB space, r (x, y | t0), g (x, y | t0), b (x, y | T0) is expressed by the following equation (10).

The movement amount calculation unit 1561 outputs the calculated movement amount (t0, t0-k) to the foreground region probability map generation unit 1562.
The foreground area probability map generation unit 1562 includes K movement amounts (t0, t0-k) (k = 1, 2,... K), foreground area information (t0-k), and the foreground area extraction unit 155. Using the acquired foreground area information (t0) at time t0, a probability P (x, y | t0) representing the probability that each coordinate (x, y) on the processed image (t0) is included in the foreground image area is obtained. calculate. Specifically, the foreground area probability map generation unit 1562 calculates the probability P (x, y | t0) using the following equation (11).

Here, w _k represents a weighting factor, and the sum of w _k from k = 0 to K is 1. Dx _k and dy _k represent the x component and the y component of the movement amount (t, t ± k), respectively. M (x, y | t0-k) is “1” when the pixel at the coordinates (x, y) of the processed image (t0-k) is the foreground image area (t0-k). When the area is not the area (t0-k) (the background image area), the value is “0”. Note that the weighting factor w _k may be set according to the time distance from t0, for example, as shown in Expression (12). That is, for the foreground area information at a time away from time t0, the value of the weighting factor is set small.

Note that a set of probabilities P (x, y | t0) for all coordinates (x, y) of the processed image (t0) is referred to as a foreground region probability map P (t0). The foreground area probability map P (t0) is expressed by the following equation (13).

Here, W represents the number of pixels in the horizontal direction of the processed image (t0), and H represents the number of pixels in the vertical direction of the processed image (t0).

The foreground area probability map generation unit 1562 uses the following expression (14) for the calculated foreground area probability map P (t0), and the pixel at the coordinates (x, y) of the processed image (t0) M (x, y | t0) (foreground area information) indicating whether or not

The foreground area probability map generation unit 1562 sets M (x, y | t0) to “1” (foreground image area) and P when P (x, y | t0) is larger than the threshold Th (x, y | t0). When (x, y | t) is equal to or smaller than the threshold Th (x, y | t0), M (x, y | t0) is set to “0” (background image region). Here, the threshold value Th (x, y | t0) takes a value of 0 to 1, and is represented by the following equation (15), for example.

Here, N _F is the number of frames (number of images) used to generate the foreground region probability map (N _F = K in this embodiment), and N _F0 is (1 ≦ N _F0 <N _F −1). It is a predetermined value satisfying
The foreground area probability map generation unit 1562 outputs the calculated foreground area information M (x, y | t0) to the boundary line correction unit 1564.

The boundary line correction unit 1564 performs a contour correction process along the contour of the foreground image region indicated by the foreground region information M (x, y | t0) input from the foreground region probability map generation unit 1562. Specifically, a smoothed outline (smooth outline) is obtained by performing opening and closing of the morphological image processing.

<Operation of Foreground Area Correction Unit 156>
Hereinafter, the details of the operation of the foreground area correction processing will be described.
FIG. 15 is a flowchart showing an example of the operation of the foreground area correction process according to the present embodiment.

(Step S207-01) The movement amount calculation unit 1561 includes information from time t0 to time t0-K (video information (t0-k), video information (t0), foreground area information (t0-k), foreground area information ( t0), ROI information (t0-k), and ROI information (t0)) are read from the buffer unit 158. The foreground area probability map generation unit 1562 reads the foreground area information (t0-k) from time t0 to time t0-K from the buffer unit 158. Thereafter, the process proceeds to step S207-02.

(Step S207-02) The movement amount calculation unit 1561 calculates the movement amount (t0, t0-k) of the foreground image area (t0-k) based on the information read out in step S207-01. Thereafter, the process proceeds to step S207-03.
(Step S207-03) The movement amount calculation unit 1561 determines whether or not the movement amount (t0, t0-k) from time t0-1 to time t0-K has been calculated (whether there is an unprocessed buffer). To do. If it is determined that the movement amount (t0, t0-k) from time t0-1 to time t0-K has been calculated (Yes), the process proceeds to step S207-04. On the other hand, if it is determined that there is a time t0-k for which the movement amount (t0, t0-k) is not calculated (Yes), the value of k is changed, and the process returns to step S207-02.

(Step S207-04) The foreground region probability map generation unit 1562 calculates the movement amount (t0, t0-k) (k = 1, 2,... K) calculated in Step S207-02, and Step S207-01. The foreground area probability map P (t0) is calculated using the foreground area information (t0-k) read out in step (b). Thereafter, the process proceeds to step S207-05.
(Step S207-05) The foreground area probability map generation unit 1562 uses the expression (13) for the foreground area probability map P (t0) calculated in step S207-04 to calculate the foreground area information M (x, y | T0) is calculated. That is, the foreground area probability map generation unit 1562 extracts the foreground image area as an area having foreground area information M (x, y | t0) = 1. Thereafter, the process proceeds to step S208-01.
(Step S208-01) The boundary line correction unit 1564 performs outline correction processing along the outline of the foreground image area indicated by the foreground area information M (x, y | t0). Thereafter, the operation is terminated.

Returning to FIG. 4, the mask information generation unit 157 generates a mask representing the corrected foreground image region (t) indicated by the information input from the foreground region correction unit 156. Note that the mask information generation unit 157 may generate a mask representing a background area that is an area other than the foreground image area. The mask information generation unit 157 outputs the generated mask information (t).

Returning to FIG. 4, the details of the operation of the buffer unit 158 will be described.
After completion of the foreground area extraction process of the video at time (t0), the buffer unit 158 performs various data (video information (t), information indicating the foreground image area (t), depth at the time (t) that satisfies the following condition A: Information (t), ROI information (t), etc. are discarded, and various data (video information (t0), information indicating foreground image area (t0), depth information (t0), ROI information (t0) at time (t0) ) And the like.

<Condition A>
(1) Various data at time t (2) Time (t0) and time (t0-) where the time distance between time (t0) and time (t0-k) is the longest (regardless of past or future time) k) Various data at time t with the smallest similarity of shape feature parameters (for example, moment invariants) to k)

FIG. 16 is a flowchart illustrating an example of the operation of the buffer unit 158 according to the present embodiment.
(Step S158-01) The buffer unit 158 searches for an empty buffer for storing various information at the time (t0). Thereafter, the process proceeds to step S158-02.
(Step S158-02) The buffer unit 158 determines whether or not there is an empty buffer as a result of the search in step S158. If it is determined that there is an empty buffer (Yes), the process proceeds to step S158-05. On the other hand, if it is determined that there is no empty buffer (No), the process proceeds to step S158-03.

(Step S158-03) The buffer unit 158 selects a buffer (referred to as a target buffer) in which information satisfying the condition A is stored. Thereafter, the process proceeds to step S158-04.
(Step S158-04) The buffer unit 158 discards the various data stored in the target buffer selected in Step S158-03, thereby emptying the target buffer (clearing the storage area). Thereafter, the process proceeds to step S158-05.
(Step S158-05) The buffer unit 158 stores various data at time (t) in the target buffer, and ends the buffer update control.

<Operation of Object Extraction Unit 15>
An operation performed by the object extraction unit 15 will be described.
FIG. 17 is a flowchart illustrating an example of the operation of the object extraction unit 15 according to the present embodiment.

(Step S201) The object extraction unit 15 reads various data (video information (t), depth information (t), ROI information (t)). Specifically, video information (t) is input to the filter unit 151a and the buffer unit 158, depth information (t) is input to the filter unit 151b, and ROI information (t) is input to the distribution model estimation unit 152 and the buffer unit 158. . Thereafter, the process proceeds to step S202.
(Step S202) The object extraction unit 15 determines whether there is an extraction target image by determining whether the extraction target flag included in the ROI information (t) indicates presence or absence. If it is determined that there is an extraction target image (Yes), the process proceeds to step S203. On the other hand, when it determines with there being no extraction object image (No), it progresses to step S210.

(Step S203) The filter unit 151a removes noise from the video information (t) input in step S201 and performs a smoothing process. The filter unit 151b removes noise from the depth information (t) input in step S201 and performs a smoothing process. Thereafter, the process proceeds to step S204.
(Step S204) The distribution model estimation unit 152, based on the depth information (t) smoothed in Step S203 and the ROI information (t) input in Step S201, the ROI depth distribution information (t) in the ROI. Is estimated. Thereafter, the process proceeds to step S205.

(Step S205) The clustering unit 153 divides the processed image (t) into superpixels by performing clustering on the video information (t) smoothed in step S203. The clustering unit 153 performs labeling for each super pixel to generate region information (t). Thereafter, the process proceeds to step S206.
(Step S206) The feature amount calculation unit 154 includes the region information (t) generated in step S205, the video information (t) smoothed in step S203, the depth information (t) smoothed, and the ROI information. Based on (t), a feature value for each region (label) is calculated. Thereafter, the process proceeds to step S207.

(Step S207) The foreground region extraction unit 155 applies the ROI information (t) input in step S201, the ROI depth distribution information (t) estimated in step S204, and the feature amount information (t) calculated in step S206. Based on the video information (t), the foreground image area (t) to be extracted is extracted (foreground area extraction processing). The foreground area extraction unit 155 generates foreground area information (t) indicating the extracted foreground image area (t). Thereafter, the process proceeds to step S208.
(Step S208) The foreground area correction unit 156, for the foreground area (t) extracted in step S207, sets the foreground image area (t0) at time t0 close to the time t0-k (k = 1, 2). ,..., K) based on the foreground image area (t0-k) (foreground area correction processing).

(Step S209) The mask information generation unit 157 generates mask information indicating a mask representing the foreground image region (t0) corrected in step S208. Thereafter, the process proceeds to step S210.
(Step S210) The mask information generation unit 157 stores the mask information generated in step S209 in the mask information storage unit 16.

FIG. 18 is a schematic diagram illustrating an example of depth information (t) according to the present embodiment. In this figure, images D1, D2, and D3 indicate depth information (t1-2), depth information (t1-1), and depth information (t1), respectively. In the images D1, D2, and D3, portions having the same color indicate the same depth. Further, FIG. 18 shows that the bright (light) image portion of the color has a smaller depth (positioned forward) than the dark (dark) image portion of the color.
Note that the depth information (t) in FIG. 18 is obtained by acquiring the amount of deviation of the parallax from the right-eye camera based on the left-eye camera with respect to the video shot by the stereo camera. Also, in the depth information D1 to D3, the left part of the image surrounded by the chain line (the part attached with the symbols U1 to U3) has a difference in parallax because the video seen from the left eye camera and the video seen from the right eye camera are different. This is an indefinite area where the amount cannot be determined.

FIG. 19 is a schematic diagram illustrating an example of the foreground area probability map P (t0) according to the present embodiment. In this figure, images P1, P2, and P3 respectively show a foreground area probability map P (t1-2), a foreground area probability map P (t1-1), and a foreground area probability map P (t1). This foreground area probability map P (t) is calculated based on the depth information (t) in FIG. In this figure, it is shown that a bright (light) color image portion is more likely to have a smaller depth (positioned in front) than a dark (dark) color image portion.

FIG. 20 is an explanatory diagram of an example of the foreground image area (t) according to the present embodiment. In this figure, images M1a, M2a, and M3a are respectively a foreground image area (t1-2), a foreground image area (t1-1), and a foreground image area (after correction by the foreground area correction unit 156 according to the present embodiment). t1). Images M1b, M2b, and M3b are a foreground image area (t1-2), a foreground image area (t1-1), and a foreground image area (t1), respectively, according to the prior art.

In FIG. 20, in the images M1b, M2b, and M3b according to the conventional technique, a missing portion and an erroneous extraction portion are generated in the portions denoted by the symbols E1 to E6. In addition, when the images M1b, M2b, and M3b are played back in time, the foreground image area has a missing portion or an erroneously extracted portion for each image, thereby causing flicker and flicker due to discontinuity of the extracted shape (contour). Arise.
On the other hand, in the images M1a, M2a, and M3a according to the present embodiment, the shape of the foreground image area is smoothed (stabilized), and compared with the images M1b, M2b, and M3b, the foreground image area has a missing portion and an erroneous extraction portion. Occurrence has been reduced. Thereby, in this embodiment, even when the images M1a, M2a, and M3a are reproduced along the time, it is possible to suppress the occurrence of flicker and flicker due to the discontinuity of the extracted shape.

Thus, according to the present embodiment, the foreground area correction unit 156 converts the foreground image area (t0) indicated by the foreground area information (t0) at time t0 to the foreground area information (t0-k) at time t0-k. And video information (t0-k). Thereby, the video processing apparatus 1 can suppress the occurrence of flicker and flicker due to the discontinuity of the extraction shape, and can reliably extract the image of the object.

Further, according to the present embodiment, the movement amount calculation unit 1561 performs the video information (t0) and the ROI information (t0) at the time t0, and the video information (t0-k) and the foreground area information (at the time t0-k). The foreground region probability map generation unit 1562 that calculates the amount of movement of the foreground image region (t0-k) from time t0-k to time t0 based on the t0-k) and ROI information Based on the movement amount calculated by the unit 1561 and the foreground image area (t0-k), a foreground area probability map P (t0) in which each coordinate in the video at time t0 is the foreground image area is calculated. The boundary region correction unit 1564 extracts the foreground region information (t0) at time t0 based on the foreground region probability map P (t0) calculated by the foreground image probability map generation unit 1562, and the extracted foreground region information (t0) The foreground image area (t0) shown is corrected. Thereby, the video processing apparatus 1 can suppress the occurrence of flicker and flicker due to the discontinuity of the extraction shape, and can reliably extract the image of the object.

In addition, according to the present embodiment, the filter unit 151a uses the bilateral filter of Expression (1), so that each image of the video information (t) is converted into a skeleton component in which an edge component is held, noise, and a pattern. Can be separated into texture components.

Further, according to the present embodiment, the filter unit 151b removes the noise of the depth information (t) and smoothes it. Thereby, in the video processing apparatus 1, the estimation accuracy of the mixed model of the depth distribution model in the distribution model estimation unit 152 can be improved. As a result, in the video processing device 1, the foreground area extraction unit 105 can accurately determine the threshold value relating to the depth used for the integration process of the basic foreground area and each area in the ROI.

Also, according to the present embodiment, in the video processing device 1, the clustering unit 153 performs clustering on the skeleton component image after the edge holding smoothing filter. As a result, the video processing apparatus 1 can obtain a superpixel group that is stable (robust) against noise and texture. Note that the super pixel represents a meaningful area having a certain large area.

Further, according to the present embodiment, the video processing apparatus 1 can obtain the depth distribution model of the foreground region with higher accuracy by obtaining the depth distribution model in the ROI using the ROI information by the mixed model. As a result, the foreground area extraction unit 155 can accurately determine the threshold value relating to the depth used for the integration process of the basic foreground area and each area in the ROI.

In addition, according to the present embodiment, the foreground region correction unit 156 corrects a missing portion of the extraction target image region at time (t0) or an erroneous extraction portion of the extraction target image region at time (t0) to extract the extracted image shape in the time direction ( Flickering and flickering due to discontinuity of (contour) can be suppressed.

In the present embodiment, the video processing apparatus 1 performs smoothing processing by removing noise in the video information (t). Thereby, in the video processing apparatus 1, it can suppress that the super pixel group with a small area | region area resulting from noise and a texture component arises.

In the present embodiment, the movement amount calculation unit 1561 uses a motion search such as a spiral search, so that the amount of calculation required to obtain the movement amount (t0, t0-k) can be reduced. Further, the movement amount calculation unit 1561 may use only the white portion (foreground region) on the mask Mk when calculating the similarity (see FIG. 13). Thereby, in the video processing apparatus 1, it is possible to prevent a search error of the movement amount and to omit unnecessary calculation, compared to a case where template matching is performed including the background region. In this embodiment, the example in which the amount of movement of the foreground area from the past time (t−k) (k = 1 to K) to the time (t) has been described, but the future time (t + k) (k = 1 to K), the amount of movement from time (t) may be obtained.

In addition, in the detection process of the user-specified ROI information (ts) in the above embodiment, the case where a rectangular (quadrangle) selection tool is used has been described. However, the present invention is not limited to this, and other selection tools may be used. For example, a selection tool shown in FIGS. 21 and 22 may be used.

FIG. 21 is a schematic diagram illustrating another example of the user-specified ROI information (ts) detection process according to the present embodiment. FIG. 21 is a diagram when an ellipse (circle) selection tool is used, and shows that the user has surrounded the target image to be extracted with the ellipse selection tool.
In this figure, the position information of the frame (reference circumscribed circle of the object O; the circumscribed circle includes an ellipse) with the symbol r2 is user-specified ROI information (ts). Specifically, in the case of FIG. 21, the user-specified ROI information (ts) is recorded as data in Table 2 below, for example.

In Table 2, user-specified ROI information (ts) includes time ts (or frame number), presence / absence flag (extraction target flag) indicating whether or not an extraction target image exists in the circumscribed circle, and the center of the circumscribed circle The position (x0, y0) (the coordinates of the point P2 in FIG. 21), the minor axis direction of the circumscribed circle (the direction of the vector labeled with the symbol D21 in FIG. 21), and the short side length (the length represented by the symbol W2 in FIG. ), The major axis direction of the circumscribed circle (the direction of the vector labeled with D22 in FIG. 21) and the length of the long side (the length represented by the symbol L2 in FIG. 21).

FIG. 22 is a schematic diagram showing another example of detection processing of user-specified ROI information (ts) according to the present embodiment. FIG. 22 is a diagram when a freehand selection tool is used, and represents that the user has surrounded the target image to be extracted. In this figure, the position information of the frame (the circumscribed shape of the object O) denoted by reference numeral r3 is user-specified ROI information (ts). Specifically, in the case of FIG. 22, the user-specified ROI information (ts) is recorded, for example, as data in Table 3 below.

In Table 3, the user-specified ROI information (ts) includes time ts (or may be a frame number), a presence / absence flag (extraction target flag) indicating whether or not an extraction target image exists in the circumscribed shape, and the starting point of the circumscribed shape Position (x0, y0) (the position of the point at which the user started to input the circumscribed shape) (the coordinates of the point P3 in FIG. 22), on the edge of the circumscribed shape clockwise from the starting point position (or may be counterclockwise) It is represented by a chain code that represents a point. The chain code is a numerical value of a position of a point B adjacent to a certain point A, and a numerical value of a position of a point C (a point other than the point A) adjacent to the adjacent point B. , And so on, and a line is represented by the combination of those numerical values.

Further, in the above embodiment, the ROI information (t) obtained from the ROI acquisition unit 13 is used for each processing image unit, and the shape of the ROI surrounding the image area to be extracted is superimposed on the processing image (t). Then, the user may be notified that the image area to be extracted is selected. Further, the frame number and time information of the currently displayed processed image (t) may be presented to the user.

In the above embodiment, the case where the foreground image area (t0-k) at the previous time t0-k is used for the foreground image area (t0) at the time t0 with the foreground area correction unit 156 has been described. The invention is not limited to this. For example, only the foreground image region (t0 + k) at time t0 + k after a certain time t0 may be used, or only the foreground image region (t0 ± k) at time t0 ± k before and after a certain time t0 may be used. . Moreover, k = 1 may be sufficient.

In the above embodiment, the depth information (t) may not be one piece of information for one pixel of the video information (t), but one piece of information for a plurality of adjacent pixels. Also good. That is, the resolution represented by the depth information (t) may be different from the resolution of the video information (t).
Further, the depth information (t) is calculated by, for example, stereo matching in which a subject is imaged by a plurality of adjacent imaging devices, and a displacement such as the position of the subject is detected from a plurality of captured video information to calculate a depth. Information. However, the depth information (t) is not limited to information calculated by a passive stereo method such as stereo matching, but an active three-dimensional measuring instrument (range finder) using light such as TOF (Time-Of-Flight) method. It may be the information acquired by.

In the above embodiment, the case where the video display unit 14 is a touch panel type display has been described. However, the present invention is not limited to this, and other input means may be used, and the video processing apparatus 1 may display video. In addition to the unit 14, an input unit (for example, a pointing device such as a mouse) may be provided.

In the above embodiment, the ROI acquisition unit 13 may extract the ROI information (t) using, for example, any of the following methods (1) to (5).
(1) Harris corner detector
(2) FAST corner detection
(3) Small Universe Segment Assessing Nucleus (corner detector)
(4) SURF (speed up robust features)
(5) SIFT (Scale-invariant Feature Transform)

In the above-described embodiment, the case where the ROI acquisition unit 13 extracts the ROI information (t) using the feature points has been described. However, the present invention is not limited to this, for example, the distribution of the color information of the user-specified area Based on the above, ROI information (t) may be extracted by a particle filter or Mean-shift. For example, the ROI acquisition unit 13 may extract the ROI information (t) using a known motion search.

In the above embodiment, the case where the filter units 151a and 151b use bilateral filters has been described. However, the present invention is not limited to this, and the filter units 151a and 151b may be other filters, for example, TV (Total Variation) filters. , K-nearest neighbor averaging filter, median filter, low pass filter may be used only for the flat part with small edge strength.
Moreover, in the said embodiment, the filter parts 151a and 151b may perform an edge smoothing filter recursively.
In the above-described embodiment, the video processing device 1 may perform the edge holding smoothing filter processing on the video information (t) and the depth information (t) before inputting them to the object extraction unit 15.

In the above-described embodiment, the case where the Gaussian distribution is used as the distribution model used by the distribution model estimation unit 152 for the mixed model has been described. However, the present invention is not limited to this, and for example, an exponential distribution family (Laplace distribution, beta distribution) , Bernoulli distribution, etc.) may be used. In addition, the distribution model estimation unit 152 may set the number of classes Kc used for the mixed model to a predetermined value, or may determine a value as in the following example.
The distribution model estimation unit 152 sets a predetermined class number Kc ′ as the class number Kc, and performs clustering by the K-means method. Thereafter, when there is a class Ci and a class Cj where the distance between classes satisfies a predetermined threshold value or less, the distribution model estimation unit 152 performs a process of merging the class Ci and the class Cj into a new class Ck ′. Do. The distribution model estimation unit 152 determines the number of classes Kc (≦ Kc ′) by repeating this process until the number of classes converges to a constant value.
Note that the method used by the distribution model estimation unit 152 to estimate the depth distribution model is not limited to a parametric estimation method such as a mixed model, and may be a non-parametric estimation method such as a Mean-shift method.

In the above embodiment, the case where the clustering unit 153 performs clustering in the feature amount space has been described. However, the present invention is not limited to this, and clustering in the image space may be performed. Clustering in the image space is a method for performing region division based on the similarity between pixels or pixel groups (regions) constituting the region in the original image space without mapping to the feature amount space. is there. For example, the clustering unit 153 may perform clustering in the image space using the following method.
(A) Pixel Combining Method For example, the clustering unit 153 represents the connection relationship between pixels as a weighted undirected graph, and performs region integration based on the strength of the edge representing the connection relationship so that the vertices form a global minimum tree.
(B) Region growth method (also referred to as region growing method)
(C) Region division integration method (also referred to as Split & Merge method)
(D) A method combining any one of (a), (b), and (c) Note that the clustering unit 153 performs labeling after the clustering processing in the image space, and region information (label information) indicating the labeling result ( t).

In the above embodiment, the movement amount calculation unit 1561 calculates the similarity R _SAD (Equation (9)) (SAD (Sum of Absolute Difference)), and determines the region having the smallest value as the estimation region. The present invention is not limited to this. For example, the estimation region may be determined using other methods shown in (1) to (3).

(1) SSD (Sum of Squared Difference)
The movement amount calculation unit 1561 calculates the absolute value of the difference between the corresponding pixel values between the images using the Euclidean distance (L ₂ −distance, L ₂ −norm), and the sum R _SDD (the following equation (16)) The region having the smallest value of is determined as the estimated region.

Here, Expression (16) indicates that the absolute value is calculated by the Euclidean distance (L ₂ -distance, L ₂ -norm), and the sum of i and j is taken.

(2) NCC (Normalized Cross-Correlation)
Also called normalized cross-correlation. The movement amount calculation unit 1561 determines an area where the _RNCC value of the following equation (17) is closest to 1 as an estimated area.

(3) CCC (Cross-Correlation Coefficient)
Also called cross-correlation coefficient. The movement amount calculation unit 1561 determines an area where the value of R _{CCC in} the following equation (18) is closest to 1 as an estimated area. In addition, the eye bar (“-” (bar) on “I” (eye)) and tea bar (“-” (bar) on “T” (tea)) in formula (18) are respectively This represents an average vector of pixel values in the indicated area.

Note that the operation amount of the movement amount calculation unit 1561 increases in the order of the equations (9), (16), (17), and (18). Note that the movement amount calculation unit 1561 may calculate the movement amount using a hierarchical search method (also referred to as multi-resolution method or coarse-to-fine search method) instead of the spiral search. Good.

In the above-described embodiment, the clustering unit 153 may perform clustering on the images in the ROI based on the ROI information (t) obtained from the ROI acquisition unit 13. Thereby, the clustering unit 153 can reduce the arithmetic operation. Further, the clustering unit 153 may perform clustering on an area wider than the target image area based on the ROI information. As a result, the clustering unit 153 can improve the accuracy of clustering as compared to the case where clustering is performed on images in the ROI.

In the above-described embodiment, the region integration unit 1534 determines whether or not a part of the region exceeds the ROI boundary with respect to the region having the center of gravity in the ROI when determining region integration. You may determine using a rectangle. Thereby, the video processing apparatus 1 can reduce erroneous extraction of the background area as the foreground area. In addition, the area integration unit 1534 may determine the area integration using the adjacent relationship between the areas instead of using the feature amount of the basic foreground area. For example, the region integration unit 1534 may determine the region integration with the region adjacent to the region that has already been determined to be the foreground region, using the feature amount of the region that has already been determined to be the foreground region. As a result, the video processing apparatus 1 can extract the foreground region with higher accuracy.

A part of the video processing apparatus 1 in the above-described embodiment may be realized by a computer. In that case, the program for realizing the control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by a computer system and executed. Here, the “computer system” is a computer system built in the video processing apparatus 1 and includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM or a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” is a medium that dynamically holds a program for a short time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, In such a case, a volatile memory inside a computer system serving as a server or a client may be included and a program that holds a program for a certain period of time. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.
Further, a part or all of the video processing device 1 in the above-described embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the video processing apparatus 1 may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Further, in the case where an integrated circuit technology that replaces LSI appears due to progress in semiconductor technology, an integrated circuit based on the technology may be used.

As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to that described above, and various design changes and the like can be made without departing from the scope of the present invention. It is possible to

The present invention is suitable for use in a video processing apparatus.

DESCRIPTION OF SYMBOLS 1 ... Video processing apparatus, 10 ... Video information acquisition part, 11 ... Depth information acquisition part, 12 ... Video information reproduction part, 13 ... ROI acquisition part, 14 ... Video display part 15 ... Object extraction unit, 16 ... Mask information recording unit, 151a, 151b ... Filter unit, 152 ... Distribution model estimation unit, 153 ... Clustering unit, 154 ... Feature amount calculation , 155... Foreground region extraction unit, 156... Foreground region correction unit, 157... Mask information generation unit, 158... Buffer unit, 1531. Generating unit, 1533 ... area growing unit, 1534 ... area integrating unit, 1561 ... movement amount calculating unit, 1562 ... foreground area probability map generating unit, 1563 ... corrected foreground area determining part, 1564 ···boundary Frequency correction unit

Claims

A video processing apparatus that extracts foreground region information indicating a foreground image from a video indicated by video information,
A video processing apparatus comprising: a foreground area correction unit that corrects a foreground image indicated by foreground area information at a first time by using foreground area information and video information at a second time.
2. The video according to claim 1, wherein the foreground area correction unit corrects the foreground image indicated by the foreground area information at the first time by using the foreground area information and the video information at a plurality of second times. Processing equipment.
The video indicated by the video information includes an image of a predetermined object,
The foreground area correction unit
Based on the video information and the target image area information indicating the image of the object at the first time, and the video information, the foreground area information and the target image area information at the second time, A movement amount calculation unit for calculating a movement amount by which the foreground image has moved during one time;
A foreground region probability map generation unit that calculates a probability that a portion in the video at the first time is a foreground image based on the movement amount calculated by the movement amount calculation unit and the foreground image;
A correction unit that extracts foreground region information at a first time based on the probability calculated by the foreground image probability map generation unit, and corrects a foreground image indicated by the extracted foreground region information;
The video processing apparatus according to claim 1, further comprising:
A video processing method in a video processing apparatus for extracting foreground area information indicating a foreground image from a video indicated by video information,
The foreground area correction unit includes a foreground area correction step for correcting the foreground image indicated by the foreground area information at the first time by using the foreground area information and the video information at the second time. Method.
The video indicated by the video information includes an image of a predetermined object,
The foreground area correction step
Based on the video information and the target image area information indicating the image of the object in the first time, and the video information, foreground area information and target image area information in the second time, the movement amount calculation unit, A moving amount calculating step for calculating a moving amount by which the foreground image has moved from a second time to a first time;
A foreground area probability map generation unit calculates a probability that a portion in the video at the first time is a foreground image based on the movement amount calculated in the movement amount calculation step and the foreground image. A map generation step;
A correction unit that extracts the foreground area information at the first time based on the probability calculated in the foreground image probability map generation step and corrects the foreground image indicated by the extracted foreground area information;
The video processing apparatus according to claim 4, further comprising:
In the computer of the video processing apparatus that extracts the foreground area information indicating the foreground image from the video indicated by the video information,
A video processing program for executing a foreground area correction procedure for correcting a foreground image indicated by foreground area information at a first time using foreground area information and video information at a second time.
The video indicated by the video information includes an image of a predetermined object,
The foreground area correction procedure is
Based on the video information and the target image area information indicating the image of the object at the first time, and the video information, the foreground area information and the target image area information at the second time, A moving amount calculation procedure for calculating a moving amount by which the foreground image has moved during one time;
A foreground region probability map generation procedure for calculating a probability that a portion in the video at the first time is a foreground image based on the movement amount calculated in the movement amount calculation procedure and the foreground image;
A correction procedure for extracting the foreground area information at the first time based on the probability calculated by the foreground image probability map generation procedure and correcting the foreground image indicated by the extracted foreground area information;
Video processing program for executing