JP6272071B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for processing an image.

There is a technique for extracting a subject area from a captured video and synthesizing the video. A typical method is shown below.
There is a method (hereinafter referred to as a first method) in which a camera is fixed and a background image is taken first. Specifically, a place where a predetermined subject (for example, a person, an animal, etc.) is not included is photographed in advance with a camera, and this is used as a background video. Thereafter, it is assumed that a predetermined subject has moved in front of the camera, and this is taken as a captured image. In the first method, a difference between the captured image and the background image is obtained, and an area having the difference is extracted as a non-rigid body area (an area including a predetermined subject).

  There is also a method based on chroma key synthesis (hereinafter referred to as a second method) (see, for example, Patent Document 1). In chroma key composition, first, a screen of a specific color is arranged as a background, and a predetermined subject is photographed together with this background. In the second method, a background color or other color is determined based on the hue, saturation, luminance, and the like of each pixel based on the captured video, and a non-rigid body region (a predetermined subject is included) Area).

  There is also a method using graph cut (hereinafter referred to as a third method). In the third method, after the subject is photographed, the user roughly cuts out the target area, which is set as Seed, given a predetermined constraint condition, and optimized by graph cut. The predetermined constraint conditions are, for example, 1) a label boundary is placed at a portion where the color difference between adjacent pixels is large, and 2) a pixel having a color similar to the designated foreground / background Seed Make it a background, etc.

JP 2004-7770 A

When the first method is used, it is necessary to fix the camera, which makes it difficult to apply the camera work freely.
Further, when the second method is used, it is necessary to limit the background to a special one, which makes it difficult to apply when it is desired to use a landscape or the like as the background. In particular, many of the past videos stored in the video library are not shot with the chroma key, and it is difficult to use the second method.
Further, in the third method, it is necessary for the user himself / herself to specify the target area, which requires a lot of manual labor.

  In this way, in video production, in order to cut out a subject area to be synthesized from a shot video and perform video synthesis, chroma key synthesis (second method) and complicated manual work (third method) by post-processing are performed. Has been used.

  An object of the present invention is to provide an image processing device, an image processing method, and a program for separating a target portion (non-rigid body portion) without using manual work. As a result, the present invention enables a person who does not have specialized knowledge of video processing to perform desired video production without much effort.

  An image processing apparatus according to the present invention includes a feature point detection unit that detects a feature point from an image of each frame in content composed of a plurality of frames, and a feature amount of each feature point detected by the feature point detection unit A corresponding point search unit for searching for corresponding points between images in each frame, and in the image of the target frame, n adjacent feature points are connected and divided into a plurality of polygonal shapes, and the target A polygon shape dividing unit that projects the plurality of polygonal shapes with respect to an image of a frame before and / or after a frame image, and an image of the target frame and an image of the target frame Or, in the image of the subsequent frame, based on the similarity between the pixels, the image is divided into a plurality of polygons by a region dividing unit that divides each image into a plurality of regions, and the polygon dividing unit An extraction unit that compares an image of an elephant frame with an image of the target frame divided into a plurality of regions by the region dividing unit, and extracts each divided region corresponding to each polygonal position as an overlapping region And a corresponding region corresponding to each overlapping region extracted by the extraction unit from the image of the frame before or / and after the image of the target frame based on the corresponding point searched by the corresponding point search unit The identifying unit for identifying the region, each overlapping region extracted by the extracting unit, and each corresponding region identified by the identifying unit are used to calculate a change amount, and the calculated change amount has a predetermined threshold value. If it exceeds, a determination unit that determines that the overlapping region includes a non-rigid body, and an overlap that is determined by the determination unit to include a non-rigid body based on the region divided by the region dividing unit Non-rigid from region Min is configured to include a separation unit for separating the.

  According to such a configuration, the image processing apparatus performs feature point extraction, corresponding point search, and the like for each frame image, while performing region segmentation for each frame image to obtain these results. Based on the image of each frame, the target part (non-rigid part) is separated, so the target part (non-rigid part) can be separated without manual work, and has expertise in video processing Even a person who is not able to perform desired video production without paying a great deal of labor.

  In the image processing apparatus, the corresponding points between the images of each frame searched by the corresponding point search unit are used to optimize the three-dimensional position of each feature point and the camera that captured the content by optimization through bundle adjustment. Estimating the camera posture, and based on the three-dimensional position of each feature point and the camera posture, the feature point is back-projected into a three-dimensional space and re-projected onto the image plane, and the distance between the feature point and the re-projected point Is calculated as a reprojection error, and a calculation unit that identifies feature points that are separated by a predetermined size or more is provided, and the corresponding point search unit uses the calculation unit based on each feature point detected by the feature point detection unit. A configuration may be adopted in which the identified feature points are excluded and corresponding points between images of each frame are searched based on the feature values of the remaining feature points.

  According to this configuration, the image processing apparatus back-projects the feature points into the three-dimensional space based on the three-dimensional position of each feature point and the camera posture, and re-projects the feature points onto the image plane. Since the distance between points is calculated as a reprojection error, and feature points that are more than a predetermined size are excluded, the miscorrespondence is reduced and the target part (non-rigid part) is accurately extracted from each frame image. Can be separated.

  In the image processing apparatus, after separating the non-rigid body part from each overlapping area by the separating unit, the non-rigid body part is used as a foreground area, the part other than the foreground area is used as a background area, and the boundary area between the foreground area and the background area is used. In other words, a configuration may be provided that includes a transparency calculation unit that creates a tri-map from the boundary and calculates the transparency of the boundary portion of the foreground area based on the continuity of the foreground area and the background area.

  According to such a configuration, the image processing apparatus calculates the transparency of the boundary portion of the foreground area (non-rigid body portion) by the transparency calculation unit, and thus can perform natural soft edge processing on the boundary portion of the non-rigid body portion. it can. For example, when such a non-rigid body part is combined with another image, the boundary part of the non-rigid body part is well blended with the other image, and a composite image without a sense of incongruity can be created.

  In the image processing apparatus, a separation region in which a non-rigid portion is separated from each overlapping region by the separation unit, and a corresponding region corresponding to the separation region in an image of a frame before or / and after the target frame image And comparing the corresponding regions corresponding to the separation regions in the image of the frame before and after the image of the target frame with the first similarity determination unit that determines the similarity The similarity is determined to be lower than a predetermined value from the determination result by the second similarity determination unit and the first similarity determination unit, and the similarity is higher than the predetermined value by the second similarity determination unit Is determined, the corresponding region having the higher similarity among the corresponding regions corresponding to the separation region in the image of the frame before or after the target frame image is cut out and stretched to the separation region. It may be configured to include a processing unit to attach.

  According to such a configuration, the image processing apparatus supplements the portion where the non-rigid body part is separated from the image with the elements of the same region in the image of the other frame, so that it is possible to suitably create an image of only the rigid body part.

  An image processing method according to the present invention includes a feature point detection step of detecting feature points from an image of each frame in content composed of a plurality of frames, and a feature amount of each feature point detected by the feature point detection step A corresponding point search step for searching for corresponding points between images in each frame, and in the target frame image, n adjacent feature points are connected and divided into a plurality of polygonal shapes, and the target A polygonal shape dividing step of projecting the plurality of polygonal shapes with respect to an image of a frame before and / or after an image of a target frame, and an image of the target frame and an image of the target frame Alternatively, in the image of the subsequent frame, based on the similarity between pixels, a region dividing step of dividing each image into a plurality of regions, and a plurality of polygonal shapes by the polygonal shape dividing step The divided image of the target frame is compared with the image of the target frame divided into a plurality of regions by the region dividing step, and each divided region corresponding to each polygonal position is extracted as an overlapping region. Corresponding to each overlapping region extracted in the extraction step from the image of the frame before or / and after the target frame image based on the corresponding point searched by the corresponding point search step. A specific step for identifying the corresponding region to be compared, each overlapping region extracted in the extraction step, and each corresponding region identified in the specific step are calculated to calculate a change amount, and the calculated change amount is predetermined. If the threshold is exceeded, the determination step determines that the overlapping region includes a non-rigid body, and the determination step includes a non-rigid body based on the region divided by the region division step. It is configured to include a separation step of separating the determined non-rigid portion from the overlapping region with.

  According to such a configuration, the image processing method performs feature point extraction, corresponding point search, and the like for each frame image, while performing region segmentation for each frame image to obtain these results. Based on the image of each frame, the target part (non-rigid part) is separated, so the target part (non-rigid part) can be separated without manual work, and has expertise in video processing Even a person who is not able to perform desired video production without paying a great deal of labor.

  The program according to the present invention is based on a feature point detection step for detecting a feature point from an image of each frame in content composed of a plurality of frames, and a feature amount of each feature point detected by the feature point detection step. The corresponding point search step for searching for the corresponding points between the images of each frame, and in the image of the target frame, n feature points adjacent to each other are connected and divided into a plurality of polygonal shapes to be the target. A polygonal shape dividing step of projecting the plurality of polygonal shapes with respect to an image of a frame before or / and after a frame image; and an image of the target frame and an image of the target frame before or / And in the image of the subsequent frame, based on the similarity between pixels, the image is divided into a plurality of polygons by a region dividing step for dividing each image into a plurality of regions, and the polygon shape dividing step. The target frame image is compared with the target frame image divided into a plurality of regions in the region dividing step, and each divided region corresponding to each polygonal position is extracted as an overlapping region. Corresponding to each overlapping region extracted in the extraction step from the image of the frame before or / and after the target frame image based on the extraction step and the corresponding points searched by the corresponding point search step A specific step of identifying a corresponding region, each overlapping region extracted in the extraction step, and each corresponding region specified in the specific step are compared to calculate a change amount, and the calculated change amount is a predetermined amount When the threshold value is exceeded, the determination step determines that the overlapping region includes a non-rigid body, and the determination step includes a non-rigid body based on the region divided by the region division step. It is intended for executing a separation step of separating the non-rigid portion from the determined overlapping regions in the computer.

  According to such a configuration, the program performs feature point extraction, corresponding point search, and the like for each frame image, while performing region segmentation for each frame image and based on these results. Since the target part (non-rigid part) is separated from the image of each frame, the target part (non-rigid part) can be separated without manual work, and the person does not have expertise in video processing. However, the desired video production can be performed without much effort.

  According to the present invention, a target portion (non-rigid portion) can be separated without manual operation.

It is a figure which shows the structure of an image processing apparatus. It is a flowchart with which it uses for description about operation | movement of an image processing apparatus. It is a figure where it uses for description about a SLIC technique. It is a figure with which it uses for description about the procedure which produces a rigid body image | video and a non-rigid body image | video.

  When a global motion is canceled in consideration of the camera posture in a video image taken by a video camera, a change appears in video information when a non-rigid body or an animal body is observed at a specific position and time direction. On the other hand, the buildings in the background do not change.

  In this embodiment, focusing on this point, using the result of feature point extraction and corresponding point search, the camera posture and the three-dimensional position information of the feature point are calculated by optimization, and reprojection is performed in this process. A configuration is proposed in which outliers at corresponding points are excluded using an error. Furthermore, in the present embodiment, the image including the moving object is also included in the video shot while moving the camera body using the area division information obtained by dividing the image in the area where the color and brightness are similar. We propose a configuration that determines and separates rigid regions. Hereinafter, the configuration and operation of the image processing apparatus 1 according to the present embodiment will be described in detail.

  As shown in FIG. 1, the image processing apparatus 1 includes a feature point detection unit 11, a corresponding point search unit 12, a polygonal shape division unit 13, a region division unit 14, an extraction unit 15, a specification unit 16, A determination unit 17 and a separation unit 18 are provided. Note that the feature point detection unit 11 and the corresponding point search unit 12 may be configured as the camera posture estimation unit 101. Further, the polygonal shape dividing unit 13, the extracting unit 15, the specifying unit 16, the determining unit 17, and the separating unit 18 may be configured as the non-rigid body region extracting unit 102.

  The feature point detection unit 11 detects a feature point from an image of each frame in content composed of a plurality of frames. Note that the content is data captured by a video camera or the like, and is recorded in the content DB 2, for example.

  The corresponding point search unit 12 searches for corresponding points between images of each frame based on the feature amount of each feature point detected by the feature point detection unit 11.

  The polygonal shape dividing unit 13 connects n feature points adjacent to each other in an image of a target frame and divides the feature points into a plurality of polygonal shapes, and an image of a frame before or / and after the image of the target frame. In contrast, a plurality of polygonal shapes are projected. The polygonal shape dividing unit 13 divides, for example, three adjacent feature points into a plurality of triangular shapes.

  The region dividing unit 14 divides each image into a plurality of regions based on the similarity between pixels in the image of the target frame and the image of the frame before and / or after the image of the target frame. Note that the region dividing unit 14 is supplied with an image of each frame from the content DB 2.

  The extraction unit 15 compares the image of the target frame divided into a plurality of polygonal shapes by the polygonal shape dividing unit 13 and the image of the target frame divided into a plurality of regions by the region dividing unit 14, and determines each polygon. Each divided region corresponding to the position of the shape is extracted as an overlapping region.

  Based on the corresponding points searched by the corresponding point search unit 12, the specifying unit 16 corresponds to each overlapping region extracted by the extraction unit 15 from the image of the frame before or / and after the target frame image. Identify the corresponding area.

  When the determination unit 17 compares each overlapping area extracted by the extraction unit 15 with each corresponding region specified by the specifying unit 16 to calculate a change amount, and the calculated change amount exceeds a predetermined threshold value Determines that the overlapping region includes a non-rigid body.

  The separation unit 18 separates the non-rigid portion from the overlapping region determined by the determination unit 17 as containing a non-rigid body based on the region divided by the region dividing unit 14.

  With this configuration, the image processing apparatus 1 performs feature point extraction, corresponding point search, and the like on each frame image, while performing region division on each frame image, Based on these results, the target part (non-rigid part) is separated from the image of each frame, so that the target part (non-rigid part) can be separated without manual work. Even a person who does not have specialized knowledge can perform desired video production without much effort.

As shown in FIG. 1, the image processing apparatus 1 may include a calculation unit 19. The calculation unit 19 may be configured as the camera posture estimation unit 101.
The calculation unit 19 uses the corresponding points between the images of each frame searched by the corresponding point search unit 12 and optimizes the three-dimensional position of each feature point and the camera of the camera that captured the content by optimization by bundle adjustment. Estimate the posture, back-project the feature point to the three-dimensional space based on the 3D position of each feature point and the camera posture, re-project it on the image plane, and re-project the distance between the feature point and the re-projection point It is calculated as an error, and a feature point that is more than a predetermined size is specified.
The corresponding point search unit 12 excludes the feature points specified by the calculation unit 19 from the feature points detected by the feature point detection unit 11, and based on the feature quantities of the remaining feature points, Search for corresponding points.

  According to this configuration, the image processing apparatus 1 back-projects the feature points into the three-dimensional space based on the three-dimensional position of each feature point and the camera posture, and re-projects the feature points onto the image plane. The distance between the projection points is calculated as a reprojection error, and feature points that are more than the specified size are excluded, so that miscorrespondence is reduced and the target part (non-rigid part) from each frame image with high accuracy Can be separated.

As shown in FIG. 1, the image processing apparatus 1 may be configured to include a transparency calculation unit 20.
After the separation unit 18 separates the non-rigid body part from each overlapping area, the transparency calculation unit 20 sets the non-rigid body part as the foreground area, the part other than the foreground area as the background area, and the boundary area between the foreground area and the background area. A trimap is created, and the transparency of the boundary portion of the foreground area is calculated based on the continuity between the foreground area and the background area.

  According to this configuration, the image processing apparatus 1 calculates the transparency of the boundary portion of the foreground area (non-rigid body portion) by the transparency calculation unit 20, and thus performs natural soft edge processing on the boundary portion of the non-rigid body portion. be able to. For example, when such a non-rigid body part is combined with another image, the boundary part of the non-rigid body part is well blended with the other image, and a composite image without a sense of incongruity can be created.

  As illustrated in FIG. 1, the image processing apparatus 1 may include a first similarity determination unit 21, a second similarity determination unit 22, and a processing unit 23. The first similarity determination unit 21, the second similarity determination unit 22, and the processing unit 23 may be configured as the rigid body image generation unit 103.

  The first similarity determination unit 21 corresponds to the separation region in which the non-rigid portion is separated from each overlapping region by the separation unit 18 and the separation region in the image of the frame before and / or after the target frame image. The similarity is determined by comparing with the area.

  The second similarity determination unit 22 determines the similarity by comparing corresponding areas corresponding to the separation areas in the image of the frame before and after the image of the target frame.

  When the processing unit 23 determines that the similarity is lower than a predetermined value from the determination result by the first similarity determination unit 21 and the second similarity determination unit 22 determines that the similarity is higher than the predetermined value, Of the corresponding regions corresponding to the separation region in the image of the frame before or after the target frame image, the corresponding region having the higher similarity is cut out and pasted to the separation region.

  According to such a configuration, the image processing apparatus 1 supplements the portion where the non-rigid body part is separated from the image with the elements of the same region in the image of the other frame, so that an image of only the rigid body part can be preferably created.

As described above, the video (content composed of a plurality of frames) acquired from the video camera is stored in the content DB 2.
The camera posture estimation unit 101 performs feature point extraction, corresponding point search, camera posture estimation, and feature point three-dimensional position estimation on the video stored in the content DB 2. At the same time, the area dividing unit 14 performs area division on the video.

  The non-rigid region extraction unit 102 extracts non-rigid regions (regions including non-rigid bodies including moving objects) using these results. The rigid image generation unit 103 generates an image of only the rigid region from which the non-rigid region is removed. Further, the transmittance calculation unit 20 estimates the transmittance at the boundary between the non-rigid region and the rigid region.

  In this way, the image processing apparatus 1 can create an image composed only of the non-rigid region and an image composed only of the rigid region.

  Here, the flow of processing by the image processing apparatus 1 will be described with reference to the flowchart shown in FIG. Steps ST1 to ST3 are the camera posture estimation step ST101, steps ST4 to ST8 are the rigid / non-rigid region determination step ST102, and steps ST9 to ST10 are the post process step ST103.

In step ST1, the camera posture estimation unit 101 extracts and describes feature points.
In step ST2, the camera posture estimation unit 101 searches for corresponding points.
In step ST3, the camera posture estimation unit 101 performs optimization (3D coordinate and camera parameter output).

In step ST4, the area dividing unit 14 sets an initial value. In this embodiment, the initial values of the division intervals are S = n / i, n = 64, and i = 1.
In step ST5, the area dividing unit 14 performs area division by a predetermined method. Here, the predetermined method is, for example, spatio-temporal clustering by the SLIC method.

In step ST6, the non-rigid region extraction unit 102 extracts an overlapping region.
In step ST7, the non-rigid region extraction unit 102 evaluates the overlapping region.
In step ST8, the area dividing unit 14 determines whether the number of divisions is the maximum. If the number of divisions is the maximum (Yes), the process proceeds to step ST9. If the number of divisions is not the maximum (No), the process returns to step ST5.

In step ST9, the rigid body image generation unit 103 generates a rigid body image by the hole filling process.
In step ST10, the transparency calculation unit 20 creates a transparency map of the non-rigid region.

<About Camera Posture Estimation Step ST101>
In the camera posture estimation step ST101, feature points are detected and described by the SIFT method for each time image of the moving image. Next, corresponding points on the images at different times corresponding to the feature points on each image are determined using the distance of the feature vector as an index, and a correspondence relationship is obtained. Next, the three-dimensional position of the feature point and the camera posture are estimated by optimization through bundle adjustment (adjustment).

Here, the required information is
The horizontal and vertical positions of the feature points on the image at each time and the (x, y, z) position in the three-dimensional space;
Image center (c _x , c _y ), focal length (f), lens distortion (k1, k2), which are internal parameters of the camera
Camera external parameters (x, y, z, roll, pitch, yaw),
It is.

  Here, in the present embodiment, it is not necessary to estimate the three-dimensional positions of camera parameters and feature points, and the processing of this part is not limited as long as a correspondence relationship with few erroneous correspondences can be obtained.

  For example, a feature point may be extracted and a correspondence relationship may be obtained from the Euclidean distance of those feature vectors, or a correspondence relationship may be obtained in both directions of a frame, and if the same correspondence relationship is not obtained, it is excluded as an outlier. You may apply a mishandling suppression process.

  Reducing erroneous correspondence is advantageous for improving the accuracy of this embodiment. Therefore, in this embodiment, outliers are suppressed by a process of obtaining the three-dimensional positions of camera parameters and feature points by optimization, that is, by using geometric hints.

<Regarding the rigid / non-rigid region determination step ST102>
Next, basic space-time division will be described. As shown in FIG. 3, the SLIC method arranges seed points at a fixed interval (S) in the image A to be processed, and pixel information of the seed position (or the pixel of the pixel with the same updated label). The average value of information) is compared with each pixel information in the range of 2S from the seed, labeling is repeated based on the similarity, and region division is performed. That is, 2S is a range for evaluating each pixel.

  Since the SLIC method is not a method that considers the movement of the subject, in this embodiment, the movement amount of the same cluster region is used between images in the time direction, and is used for comparison of pixel information between pixels and update of the seed position. In addition, a device has been devised in which the regions are correctly connected even to fast-moving subjects.

Specifically, when the feature distance of pixel information between pixels is calculated by SLIC, it is conventionally handled in the color space of the pixel information, for example, the L ^* a ^* b ^* color system, and the position in space-time A six-dimensional feature including (x, y, z) is added, and the distance is used as a criterion for similarity.

また、類似度の評価値Ｄは、下記式で表せる。評価値Ｄは、小さいほど、類似性があることになる。

For example, if the pixel at the seed position is P and the pixel to be compared is Q, each feature vector can be expressed by the following equation.

Moreover, the evaluation value D of similarity can be expressed by the following formula. The smaller the evaluation value D, the more similar.

  Here, S is the distance (pixel) between the arranged seeds, m represents the complexity when the area is divided by SLIC, and the proximity in space becomes more important as m is increased. In this embodiment, m = S.

  Here, in this embodiment, for pixels having the same label between images at each time, evaluation is given to Px and Py so as to cancel the movement amount of the two-dimensional centroid position at each time, and the three-dimensional centroid position is evaluated. Update.

  As a result, even when a fast-moving subject is included in the image, the same subject is likely to be included in the same cluster, and the accuracy of separation of the rigid region and the non-rigid region, which is the object of this embodiment, is improved. Can do.

  However, the present embodiment does not limit the detection method of the subject movement or how to reflect it in the evaluation formula. For example, the optical flow of each pixel is not the amount of centroid movement of the same cluster of each frame. Therefore, the accuracy can be improved by a method such as considering the average movement amount or considering the movement amount of each pixel.

<Rigid / non-rigid area determination (extraction of overlapping areas)>
Rigid / non-rigid areas are determined using a triangular patch consisting of feature points at a certain time using the result of segmentation, 3D position information of feature points determined by camera parameters, and camera posture information. Do.

Here, it is assumed that the division processing result in the camera posture, the three-dimensional position of the feature point, and the spatio-temporal region is obtained.
As shown in FIG. 4, when attention is paid to an image at a certain time (frame) t (image x _{1 in} FIG. 4), the three-dimensional position of the feature point is projected onto the image using the camera posture. Find the coordinates. A two-dimensional coordinate of each feature point can be connected to an adjacent feature point to form a triangular patch. In the present embodiment, the triangular patches are formed using the Delaunay triangulation method, but the present invention is not limited to this method.

Next, similar projection is performed on the images at the previous and subsequent times (frames) (image y ₁ and image z _{1 in} FIG. 4), and the region corresponding to each triangle created at time t is specified. Note that the image y ₁ is an image at a time (frame) t−1 earlier in time than the time t. Image z _1, rather than time t is temporally subsequent time (frame) t + 1 of the image.

Next, correspondence between the region division result and the triangular region is taken, and a region where the region division result overlaps the triangle is extracted.
Here, although the time t is the same two-dimensional coordinate space, the images of the times before and after the time are different in the location of the overlapping region. Thus, assuming that the same triangular area is the same subject area, each pixel in the triangle at time t can be coordinate-transformed into the triangular area at the previous and subsequent times by affine transformation. That is, it becomes clear to which coordinate each pixel of the overlapping area other than the time t corresponds on the image at a different time.

In the example shown in FIG. 4, the region corresponding to the triangular region a ₁ in the image x ₁ at time t is the triangular region b ₁ in the image y ₁ at time t− ₁ , and the image z at time t + 1. in _1, a triangular area _{c 1.}

  Here, it is possible to determine a rigid body and a non-rigid body by evaluating the change amount of the feature amount of the image information with respect to the temporal change of the overlapping region. That is, the change is small in the rigid body, and the change in the non-rigid body is large because the movement of the subject is different from the global movement of the image.

In the example illustrated in FIG. 4, it can be determined that the triangular area a ₁ in the image x _{1 at} the time t includes a non-rigid body based on the same triangular area at the previous and subsequent times. Therefore, the non-rigid region extraction unit 102 extracts a non-rigid region (P _{1 in} FIG. 4) from the triangular region a ₁ .

<Rigid body / non-rigid body area determination (overlapping area evaluation)>
Next, a method for evaluating the overlapping area will be described. Strictly speaking, an error occurs because the subject included in the triangle is not necessarily a plane even if the apex position of the triangle area matches the time change.

For example, when a rigid / non-rigid determination is performed on overlapping regions with different times based on the luminance difference of each pixel, the determination result becomes unstable.
For this reason, in this embodiment, even if some positional errors are included, the overlapping areas are stably compared on the basis of the pattern and color characteristics.

  More specifically, the evaluation is performed by using the HoG (Histogram of Oriented Gradient) and the color similarity for the overlapping region, and a threshold process is performed on the obtained similarity. For HoG, a histogram is created from the direction and degree of the luminance gradient, and this is used as the feature amount.

The color similarity uses a color histogram intersection excluding a luminance component. In this embodiment, the color space is the L ^* a ^* b ^* color system, but is not limited to this. However, not the RGB color system so as not to be affected by illumination, but a color system with independent luminance and color components such as HSV is preferable.

  Usually, the color histogram intersection creates a histogram bin (BIN) and a respective color histogram obtained by quantizing each color component. This is the sum of the minimum values in each bin, and the value obtained by dividing the sum by the sum of one histogram is taken as the similarity.

Here, the L ^* a ^* b ^* color system is adopted as the color space, and the value obtained by the same calculation except for L (brightness) is set as the similarity in this embodiment.
As a result, the similarity evaluation value is less likely to be affected by illumination changes. In this embodiment, L is excluded, but weighting with L, a, and b is also possible according to the degree of illumination change.

<Rigid body / non-rigid area determination (determination process)>
Whether the body is rigid or non-rigid is determined based on the evaluation result of the overlapping region. One of the evaluation parameters is the color histogram intersection, which is more similar to 1.0.

  Another evaluation parameter, HoG, is a gradient histogram. If a color histogram intersection is also calculated for this, the similarity is closer to 1.0.

  It is also possible to evaluate the distance between feature vectors using a histogram as a feature vector. In addition, it is possible to evaluate the calculated evaluation parameters by learning with a non-linear SVM, or weight each of the evaluation parameters, but in this embodiment, both of them are calculated empirically from experiments. Multiplication was performed, and an evaluation value within ± 0.1 with respect to 1.0 was determined as a rigid body.

<Rigid body / non-rigid area determination (detailed by loop)>
In order to make the rigid / non-rigid region information obtained in the above processing more detailed, the number of divisions in the region division processing based on the SLIC method is set to be larger, and re-division is performed.
Here, regarding the boundary between the rigid and non-rigid regions, a judgment error is likely to occur, and the rigid / non-rigid regions may not be properly separated at the previous seed interval. Expand one region in the region direction.

  Since it is not necessary to determine again the area determined to be a rigid body, no seed is set in that area. On the other hand, for a region determined to be a non-rigid body, a seed having coordinates set at twice the seed interval is given to the previous process.

  In addition, when the seed position is updated by calculating the gravity center position of the SLIC method, a constraint condition is provided so that the gravity center position does not move to the rigid body region. In this embodiment, the image information of the rigid region is set to (0, 0, 0) black for R, G, and B. However, the present invention is not limited to this. For example, the image is moved to the rigid region in the update in the SLIC. In such a case, a method may be used in which the center-of-gravity position update of the X, Y, and Z coordinates is not performed and the value obtained by the previous center-of-gravity position update is retained at the time of the update.

  In the present embodiment, by repeating this, finer area division is performed. In the present embodiment, the setting is repeated until the seed interval is every two pixels (that is, a state where one pixel is spaced between the seeds in the horizontal direction and the vertical direction). Since there is a correlation between the number of times and the processing time, the repetition condition may be changed depending on the application.

<About post-process step ST103>
In the above-described rigid / non-rigid region division processing, each boundary region is a so-called hard edge. In this state, it can be used for video processing. However, in movie production and television program production, it is common to prepare and use video material with a soft edge so that a sense of incongruity does not occur during video composition. In addition, in order to use the rigid body region for image processing, the removal of the non-rigid body region is a hole-open state, and it is necessary to compensate with some image information. Hereinafter, these processes will be described.

<Hill filling of rigid body images that do not include non-rigid bodies>
Compare the non-rigid region at a specific time with the same region at different times. At this time, the evaluation is performed by the above-described overlapping region evaluation method. Thus, if the contents are the same, the evaluation result is evaluated with a high degree of similarity. Therefore, it is statistically determined whether or not the region is a rigid region, and video information determined to be a rigid region is compensated by performing affine transformation on a non-rigid region at a specific time.

  During this compensation process, discontinuity may occur in the low frequency components and the pattern near the boundary of each video information, which may cause discomfort. Therefore, in this embodiment, the uncomfortable feeling is reduced by setting each pixel in the boundary area as an average value with the adjacent pixels.

There are no particular limitations on how to deal with this sense of incongruity. For example, the color is similar to that of the pasted image, that is, the rigid region of the hole, and the texture (gradient of the image) is the pasted image, that is, another image. Natural image synthesis is realized by using a rigid region image corresponding to a non-rigid region of time. It is also possible to use the methods published in the following papers.
P. Perez, M.M. Gangnet and A.M. Blake: “Poisson image editing”, Proc. SIGGRAPH, 22 (3), pp. 313-318 (2003)

In the example shown in FIG. 4, an image x _{2 obtained} by extracting a non-rigid body region (P _{1 in} FIG. 4) from the image x _{1 at} time t is an extracted part of another image (image y _{2 in} FIG. 4). And the image z ₂ ) are used for filling processing. Image _{y 2} is the time t-1 of the image, the image _{z 2} is a time t + 1 of the image.
The area corresponding to the triangular area a ₂ in the image x _{2 at} the time t is the triangular area b ₂ in the image y ₂ at the time t−1, and the triangular area c _{2 in} the image z ₂ at the time t + 1. It is.

A portion P ₂ corresponding to the non-rigid region P ₁ is cut out from the triangular region b _2, and a portion P ₃ corresponding to the non-rigid region P ₂ is cut out from the triangular region c ₂ . Then, the cut-out part P ₂ and the part P ₃ are combined with the corresponding part of the triangular area a ₂ . In this way, a non-rigid image is created.

<Transparency map generation of non-rigid body area (α plane creation)>
When video synthesis is performed, the nature of the boundary region is a point to be considered in order to synthesize the background image and the foreground image without a sense of incongruity. Usually, the unnatural feeling is reduced by blending the background and foreground boundary pixels into a soft edge instead of a hard edge. Specifically, in the case of an image to be a foreground, for example, an RGB image, an image plane (α plane) representing a distribution of transparency is further added, and video synthesis is performed using this value as a mixing ratio of the foreground and the background.

  An image Ir of only the rigid body excluding the non-rigid region is created. This is a background image in terms of video composition, and non-rigid video blending is performed at a constant mixing ratio at the boundary with the non-rigid region.

Here, the image non-rigid-rigid taken are not separated as I _c, the image of the non-rigid only to I _n. The transmittance of the image non-rigid region is represented by α (0 to 1), where 0 is transparent, 1 is non-transparent, and pixel mixing is performed in the meantime, and it can be assumed that video composition represented by the following equation is performed.
I _c = I _n α + I _r (1−α) (6)
I _r is determined by the above-mentioned. On the other hand, I _c is a photographed image itself. In order to obtain a non-rigid image with transparency, the unknowns are strictly I _n and α, and these need to be estimated.

Here, if the α plane is two-dimensional coordinates (x, y) on the image, 0 ≦ α (x, y) ≦ 1. Here, as a conventional method for estimating α from a natural image, there is Poisson Matching (see the following paper).
J. et al. Sun, J. et al. Jia, C .; K. Tang, H .; Y. Shum, Poisson matting, SIGGRAPH, 2004

  These are techniques for extracting a specific subject area including α from a photographed image. In this method, the foreground region and the background region are manually specified as the extraction target, the boundary portion is set as an unknown region, and α of the unknown region is estimated by optimization.

In this example, the non-rigid region, the boundary of the rigid region, the two-pixel width is the unknown region, the region obtained by removing the unknown region from the non-rigid region is the foreground region, and the region obtained by removing the unknown region from the rigid region is the background region. Α is estimated by Poisson Matching. In this embodiment, the method for estimating α is not limited, but a method for estimating α by giving a Trimap of the foreground region, the background region, and the unknown region, for example, the method shown in the following paper May be applied.
Xiaou Chen, Dongqing Zou, PingTan, Image Matching with Local and NonlocalSmooth Priors, CVPR2013
Through the above method, the highly accurate transparency and region information of the non-rigid region are acquired.

  Conventionally, in order to extract a specific subject area, it is necessary to prepare a certain shooting environment or manually perform a clipping operation in post-processing after shooting, which is complicated. In general, the need for cutting out a person on a video is high.

  The image processing apparatus 1 according to the present embodiment can automatically extract a non-rigid body such as a moving object, reduces the cumbersome work of professional creators, and does not have video production skills. Users can also perform advanced video composition.

  Further, in the present embodiment, the configuration and operation of the image processing apparatus that mainly separates a target portion (non-rigid body portion) without manual operation has been described. However, the present invention is not limited to this, and each component is provided. The method may be configured as a method and a program for separating a target portion (non-rigid body portion) without relying on manual work.

  Further, it may be realized by recording a program for realizing the functions of the image processing apparatus on a computer-readable recording medium, causing the computer system to read and execute the program recorded on the recording medium. .

  The “computer system” here includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a hard disk built in the computer system.

  Furthermore, “computer-readable recording medium” means that a program is dynamically held for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It is also possible to include one that holds a program for a certain time, such as a volatile memory inside a computer system that becomes a server or client in that case. Further, the program may be for realizing a part of the above-described functions, and may be capable of realizing the above-described functions in combination with a program already recorded in the computer system. .

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 11 Feature point detection part 12 Corresponding point search part 13 Polygon shape division part 14 Area division part 15 Extraction part 16 Identification part 17 Determination part 18 Separation part 19 Calculation part 20 Transparency calculation part 21 1st similarity determination part 22 second similarity determination unit 23 processing unit 101 camera posture estimation unit 102 non-rigid region extraction unit 103 rigid body image generation unit

Claims (6)

A feature point detection unit for detecting a feature point from an image of each frame in content composed of a plurality of frames;
A corresponding point search unit that searches for corresponding points between images of each frame based on the feature amount of each feature point detected by the feature point detection unit;
In an image of a target frame, adjacent n feature points are connected and divided into a plurality of polygonal shapes, and the plurality of the feature points are compared with respect to an image of a frame before and / or after the image of the target frame. A polygonal shape dividing unit that projects the polygonal shape of
A region dividing unit that divides each image into a plurality of regions based on the similarity between pixels in the image of the target frame and the image of the frame before or / and after the image of the target frame;
The image of the target frame divided into a plurality of polygon shapes by the polygon shape dividing unit and the image of the target frame divided into a plurality of regions by the region dividing unit are compared, and the position of each polygon shape An extraction unit that extracts each of the divided areas corresponding to the overlapping area;
Based on the corresponding points searched by the corresponding point search unit, the corresponding region corresponding to each overlapping region extracted by the extraction unit is specified from the image of the frame before and / or after the target frame image. Specific part to do,
Each overlap region extracted by the extraction unit and each corresponding region specified by the specifying unit are compared to calculate a change amount. If the calculated change amount exceeds a predetermined threshold, the overlap A determination unit that determines that the region includes a non-rigid body;
An image processing apparatus comprising: a separation unit that separates a non-rigid body portion from an overlapping region that is determined by the determination unit to include a non-rigid body based on an area divided by the region dividing unit. Using the corresponding points between the images of each frame searched by the corresponding point search unit, the three-dimensional position of each feature point and the camera posture of the camera that captured the content are estimated by optimization by bundle adjustment. Based on the three-dimensional position of each feature point and the camera posture, the feature point is back-projected into a three-dimensional space and re-projected on the image plane, and the distance between the feature point and the re-projection point is set as a reprojection error. A calculation unit that calculates and identifies feature points that are separated by a predetermined size or more;
The corresponding point search unit excludes the feature point specified by the calculation unit from each feature point detected by the feature point detection unit, and based on the feature amount of the remaining feature point, between the images of each frame The image processing apparatus according to claim 1, wherein a corresponding point is searched.   After separating the non-rigid part from each overlapping area by the separation unit, the non-rigid part is set as the foreground area, the part other than the foreground area is set as the background area, and a trimap is created from the boundary area between the foreground area and the background area. The image processing apparatus according to claim 1, further comprising: a transparency calculating unit that calculates a transparency of a boundary portion of the foreground area based on continuity between the foreground area and the background area. By comparing the separation region in which the non-rigid body part is separated from each overlapping region by the separation unit and the corresponding region corresponding to the separation region in the image of the frame before or / and after the target frame image, A first similarity determination unit for determining similarity;
A second similarity determination unit that compares the corresponding regions corresponding to the separation region in the image of the frame before and after the image of the target frame and determines the similarity;
If the similarity is determined to be lower than a predetermined value from the determination result by the first similarity determination unit, and the second similarity determination unit determines that the similarity is higher than the predetermined value, the target frame 3. The processing unit according to claim 1, further comprising: a processing unit that cuts out a corresponding region having a higher degree of similarity among corresponding regions corresponding to the separation region in an image of a frame before or after the first image and pastes the corresponding region to the separation region. Image processing device. A feature point detecting step of detecting a feature point from an image of each frame in content composed of a plurality of frames;
Based on the feature amount of each feature point detected by the feature point detection step, a corresponding point search step for searching for a corresponding point between images of each frame;
In an image of a target frame, adjacent n feature points are connected and divided into a plurality of polygonal shapes, and the plurality of the feature points are compared with respect to an image of a frame before or after the image of the target frame. A polygonal shape dividing step of projecting the polygonal shape of
A region dividing step of dividing each image into a plurality of regions based on the similarity between pixels in the image of the target frame and the image of the frame before or / and after the image of the target frame;
The image of the target frame divided into a plurality of polygon shapes by the polygon shape dividing step and the image of the target frame divided into a plurality of regions by the region dividing step are compared, and the position of each polygon shape An extraction step of extracting each divided region corresponding to as an overlapping region;
Based on the corresponding points searched in the corresponding point search step, the corresponding region corresponding to each overlapping region extracted in the extraction step is specified from the image of the frame before or / and after the target frame image. Specific process to
Each overlap region extracted in the extraction step is compared with each corresponding region specified in the specific step to calculate a change amount, and when the calculated change amount exceeds a predetermined threshold, the overlap A determination step of determining that the region includes a non-rigid body,
An image processing method comprising: a separation step of separating a non-rigid portion from an overlap region determined to contain a non-rigid body by the determination step based on the region divided by the region division step. A feature point detecting step of detecting a feature point from an image of each frame in content composed of a plurality of frames;
Based on the feature amount of each feature point detected by the feature point detection step, a corresponding point search step for searching for a corresponding point between images of each frame;
In an image of a target frame, adjacent n feature points are connected and divided into a plurality of polygonal shapes, and the plurality of the feature points are compared with respect to an image of a frame before or after the image of the target frame. A polygonal shape dividing step of projecting the polygonal shape of
A region dividing step of dividing each image into a plurality of regions based on the similarity between pixels in the image of the target frame and the image of the frame before or / and after the image of the target frame;
The image of the target frame divided into a plurality of polygon shapes by the polygon shape dividing step and the image of the target frame divided into a plurality of regions by the region dividing step are compared, and the position of each polygon shape An extraction step of extracting each divided region corresponding to as an overlapping region;
Based on the corresponding points searched in the corresponding point search step, the corresponding region corresponding to each overlapping region extracted in the extraction step is specified from the image of the frame before or / and after the target frame image. Specific process to
Each overlap region extracted in the extraction step is compared with each corresponding region specified in the specific step to calculate a change amount, and when the calculated change amount exceeds a predetermined threshold, the overlap A determination step of determining that the region includes a non-rigid body,
A program for causing a computer to execute a separation step of separating a non-rigid body part from an overlapping region determined to contain a non-rigid body by the determination step based on the region divided by the region division step.

Images