JP2008084076A - Image processor, method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform image processing highly precisely by performing assignment of optimal labels all over an image. <P>SOLUTION: An image processor is provided with: an area division part 112 for dividing a target image into a plurality of areas; and a label assignment part 111 which selects a parametric model where conformity with the parametric model and value of an evaluation function that indicates the degree of smoothness representing whether or not a model identification label changes between adjacent pixels changes become minimum from among a plurality of parametric models including a plurality of model parameters and the model identification labels for identifying models to be models for coordinating points on the target image and points on a reference image, and executes label assignment processing for assigning a model identification label of the selected parametric model to a pixel in the area in order of the number of pixels among the plurality of areas. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, method, and program for introducing a plurality of parameter models related to an input image and performing image processing by assigning the parameter model to each region of the input image.

  As a technique for performing image processing by applying a predetermined parametric model for each region of an image, for example, the technique of Non-Patent Document 1 is known. In the technique of Non-Patent Document 1, with respect to the problem of estimating the motion of an image, a local motion between two images is estimated by a technique such as an optical flow, and a plurality of local motions are estimated from a set of estimated local motions. The model parameters are extracted by the affine transformation and least square method, and the image is divided into multiple regions using the k-means method, watershed method, etc., and labeling is performed to select the optimal parameters for each divided region. Techniques to do are disclosed. According to such a technique, an appropriate motion model is selected for each region of the image.

P. E. Eren, Y. Altunbasak and M. Tekalp, "Region-based affine motion segmentation using color information", IEEE ICASSP1997, Vol. 4, pp. 3005-3008, 1997

  In such a conventional technique, labeling is performed by maximum likelihood estimation using only the fitness of model parameters for each region, and the labeling greatly affects the accuracy of the motion estimation result. For this reason, while high accuracy is obtained in a large area with a large number of samplings, the number of samplings is reduced in an area with a small size, so that optimal labeling cannot be performed and the accuracy of motion estimation is reduced. There is.

  The present invention has been made in view of the above, and is an image processing apparatus, method, and program capable of performing image processing using a parameter model with high accuracy by performing optimal label allocation over the entire image. The purpose is to provide.

  In order to solve the above-described problem and achieve the object, an image processing apparatus according to the present invention includes a region dividing unit that divides a first image into a plurality of regions each including a plurality of pixels, A model for associating a point on the second image with a point on the second image, the parametric model including a plurality of model parameters and a model identification label for identifying the model. Select the parametric model that minimizes the value of the evaluation function indicating the degree of fitness with the model and the smoothness indicating whether the model identification label changes between adjacent pixels, and the parametric model of the selected parametric model A label that executes a process of assigning a label for assigning a model identification label to a pixel in the region in order from the largest number of pixels in the plurality of regions. And an image processing unit for obtaining a correspondence relationship between the first image and the second image based on the parametric model corresponding to the model identification label assigned to each pixel. It is characterized by.

The present invention also provides an area dividing unit that divides a target image including a moving area that is a moving area with respect to a background image that is an image without movement into a plurality of areas each including a plurality of pixels, and a background A label allocating unit that executes a process of allocating a label for allocating an area type label for identifying an area or a moving area to a pixel in the area in order from the largest number of pixels in the plurality of areas; And a moving area detecting unit that detects the moving area from the target image based on the area type label.
The present invention is an image processing method and program corresponding to the image processing apparatus.

  According to the present invention, there is an effect that image processing can be performed with high accuracy by performing optimal label allocation over the entire image.

  Exemplary embodiments of an image processing apparatus, an image processing method, and a program according to the present invention are explained in detail below with reference to the accompanying drawings.

(Embodiment 1)
FIG. 1 is a block diagram of a functional configuration of the motion estimation apparatus according to the first embodiment. The motion estimation apparatus according to the first embodiment estimates the motion of each point of the target image and the reference image. As illustrated in FIG. 1, the local motion estimation unit 101, the model generation unit 107, and the region division Mainly includes a unit 112, a label allocation unit 111, an image processing unit 113, a frame memory 106, and a memory 103.

  The local motion estimation unit 101 is a processing unit that performs motion estimation between two images from a target image and a reference image using a block matching technique, and obtains a motion vector at each point.

  The model generation unit 107 generates a parametric model for associating a point on the target image with a point on the reference image from the result of local motion estimation. The parametric model includes a plurality of model parameters defining the parametric model and a model identification label for identifying the model, and is labeled with the model identification label. In the present embodiment, the parametric model is a model for obtaining a motion vector of a point on the second image of each point in the first image in the region of the model identification label based on the model parameter. The model generation unit 107 includes a model definition unit 108, a clustering processing unit 109, and a parameter estimation unit 110.

  The model definition unit 108 is a processing unit that defines and generates the parametric model and stores it in the memory 103.

  The clustering processing unit 109 is a processing unit that classifies (clusters) the motion vectors of each point on the target image obtained by the local motion estimation unit 101 and assigns an initial value for label allocation.

  For each model identification label, the parameter estimation unit 110 calculates error energy obtained from the error between the motion vector determined by the parametric model and the motion vector obtained by the local motion estimation unit 101 at each point within the same model identification label. It is a processing unit for estimating the model parameter to be minimized by the least square method.

  The region dividing unit 112 is a processing unit that divides (clusters) the frame of the target image into regions based on pixel values. The area dividing unit 112 assigns an area label for identifying an area (cluster) to each further divided area.

The label allocating unit 111 is a parametric that minimizes the value of an evaluation function indicating the degree of conformity between the parametric model and the smoothness indicating whether or not the model identification label changes between adjacent pixels from among a plurality of parametric models. This is a processing unit that selects a model and assigns a label to assign a model identification label of the selected parametric model to pixels in a region (cluster) divided by the region dividing unit 112.
Specifically, the label allocating unit 111 performs regularity on the likelihood energy U ₁ based on the pixel of the target image, the pixel of the reference image, and the motion vector of each pixel for the pixels in the divided region (cluster). A labeling process is performed by selecting a parametric model that the evaluation function based on the conversion energy U ₂ minimizes. Further, the label allocation unit 111 calculates the value of the evaluation function by selecting a region in order from the largest number of pixels (cluster size) from among a plurality of regions (clusters). Running.
The image processing unit 113 performs motion estimation by obtaining the correspondence between the target image and the reference based on the parametric model corresponding to the model identification label assigned to each pixel, and outputs the result as a motion estimation result Part.
The details of each of the above parts will be described later.

  The frame memory 106 is a storage medium that stores the input target image and reference image. The memory is a storage medium that stores calculation results and parameters obtained by each unit, a parametric model generated by the model definition unit 107, and the like.

  Next, the detail of each part is demonstrated. In this embodiment, two images to be processed are set as a target image and a reference image, and a motion from the target image to the reference image is obtained.

Let g ₁ (n) and g ₂ (n) be the pixel values of the point n∈Λ ² of the target image and the reference image, respectively.

  The local motion estimation unit 101 estimates a motion between two images from the target image and the reference image using a block matching technique as follows. Here, in this embodiment, a block matching method is adopted, but the present invention is not limited to this, and for example, motion estimation is performed using each method such as an optical flow estimation method, a gradient method, and a Bayesian method. You may go.

  In block matching, the target image is divided into blocks of a rectangular area set in advance by equation (1).

Assuming that the motion search region is WεR ² , a block matching algorithm in accordance with a sum of squared difference (SSD) is expressed by equation (2).

d (i) is the motion vector, that is, the motion of block i. In block matching, it is assumed that the blocks have the same motion vector, and the motion vector of each pixel can be expressed by equation (3).

The local motion estimation unit 101 performs a process for obtaining a motion vector according to equations (2) and (3).

  Next, details of the model definition unit 108 will be described. The model definition unit 108 defines a parametric model and stores it in the memory 103. As many parametric models as the number of motions to be estimated are prepared. Each parametric model is uniquely identified by an arbitrary number of model identification labels α∈L⊂Z corresponding to the number of motions to be estimated. The label of the point n is set to z (n) εL, and a set of regions to which the model identification label α is assigned, that is, each motion region Nα is represented by equation (4).

Then, a parametric model for each motion region to which the model identification label α is assigned is defined by equation (5).

As the model parameter aα, various forms such as a parameter defining a plane-to-plane geometric transformation from the target image to the reference image can be adopted. For example, when the affine model is used as the model parameter aα, the equations (6) and (7) are expressed by the equations (8) and (9) when the quadratic form model is used.

In addition, a rotation transformation model, Euclidean transformation model, similarity transformation model, projective transformation model, Lie transformation model, and the like may be used as model parameters.

  The model definition unit 108 generates a parametric model defined by the equations (5), (6), and (7) or a parametric model defined by the equations (5), (8), and (9) and stores the memory. 103.

  Next, details of the parameter estimation unit 110 will be described. Since the motion vector determined by the parametric model is expressed by equation (10), using this and equation (5), the motion vector determined by the parametric model is expressed by equation (11).

On the other hand, when the error between the motion vector of each point obtained by the parametric model and the motion vector of each point obtained by the local motion estimation unit 101 is defined by equation (12), the error energy is expressed by equation (13). Accordingly, the error energy becomes as shown in the equation (14) by the equations (10) and (11) for each motion vector, and is eventually expressed as the equation (15).

On the other hand, the error in the model identification label α region is expressed by equation (16), and the model parameter is estimated so that the error in the model identification label α region is minimized as shown in equation (17).

In the present embodiment, assuming that the error between the model parameter and the obtained motion vector follows a Gaussian distribution, the optimum model parameter is obtained by numerical analysis using the least square method. That is, since the normal equation in the least square method is represented by equations (18), (19), and (20), the model parameter is represented by equation (21).

Since the equation (21) is a simultaneous linear equation, the parameter estimation unit 110 obtains the model parameter by the singular value decomposition method using the equation (21). Note that the model parameters may be obtained by other methods such as LU decomposition.

  Next, details of the clustering processing unit 109 will be described. When the local motion estimation process is completed, the model parameter cannot be estimated because the label allocation process is not performed. When the model parameter estimation process is performed first, before the model parameter estimation process is executed, the clustering processing unit 109 clusters the motion vectors d (n) obtained by the local motion estimation unit 101 to assign labels. Is given as the initial value. Here, as motion clustering processing, an image is clustered into a plurality of images by the k-Means method based on pixel values such as luminance, and the image is divided, and a plurality of model parameters are obtained based on the clustering result. The model parameter label is obtained by numerical analysis processing by the k-Means method using the model parameter as an initial value.

  Next, details of the region dividing unit 112 will be described. The region dividing unit 112 divides the frame of the target image into regions based on pixel value information using a technique such as the k-means method or the watershed method.

That is, the area dividing unit 112 divides (clusters) the target image g ₁ into k pieces according to the pixel values. Then, a region label w (n) εK⊂Z for uniquely identifying each region is assigned to each region (cluster). At this time, each region (cluster) is expressed by equation (22).

Next, details of the label allocation unit 111 will be described. The label allocating unit 111 determines a parametric model to be allocated to each region clustered by the region dividing unit 112.

Here, the probability density function of the model identification label z (n) when the target image g ₁ (n), the reference image g ₂ (n), and the motion vector d (n) are given is given by Equation (23). .

Using Bayes' theorem, equation (23) is equivalent to equation (24).

Here, assuming that z, g ₁ , and d are independent, Expression (24) is equivalent to Expression (25).

In the equation (25), p (g ₂ | z, g ₁ , d) is called a likelihood function, and represents the fitness of the model identification label z with respect to the image. p (z) is referred to as prior density, and represents prior knowledge as to whether or not the shape of the model identification label z is appropriate. Assuming that the difference in pixel values according to the parametric model follows a Gaussian distribution, the likelihood function can be defined as shown in equations (26) and (27).

Further, the prior density of the label can be defined as in the equations (28) and (29).

N (n) indicates the vicinity of the point n, and is a density function that is low energy when the model identification label is the same. The model identification label z that is plausible under this formulation is a solution of the MAP estimation problem (Maximum a posteriori probability) expressed by Equation (30).

Taking the log of both sides of equation (30) and multiplying by minus (-) results in the energy minimization problem of equation (31).

In the equation (31), the temperature parameter T and the standard deviation σ of noise are collectively described as a hyper parameter λ. U ₁ is a likelihood energy indicating the degree of conformity with the parametric model, U ₂ is a regularization energy indicating a smoothness indicating whether or not the model identification label changes between adjacent pixels, and Equation (31) is: An evaluation function composed of likelihood energy U ₁ and regularization energy U ₂ is shown.

Here, in the technique of Non-Patent Document 1, the equation (31) is solved only by the likelihood energy U ₁ , and since there is no regularization energy term U ₂ , the accuracy of label allocation is not stable.

In the present embodiment, label allocation can be performed with stable accuracy by solving the equation (31) in consideration of the regularization energy term U ₂ by the label allocation unit 111.

Next, the solution of equation (31) by the label allocating unit 111 will be described. That is, the label assigning unit 111 solves the equation (32) to obtain a model identification label α that minimizes λU ₁ + U ₂ for each region (cluster) K _K.

However, since the regularization energy U ₂ includes the energy of the adjacent label, it cannot be calculated using the equation (32). Here, the regularization energy U ₂ can be transformed as shown in the equation (33) when attention is focused on a clique at the boundary of the region (cluster) (a pair of the own pixel and the adjacent pixel) and an internal clique.

Thereby, it can be divided into clique C ₁ constituted by points included in the same region and clique C ₂ constituted by points of different regions. In this case Creek potential of C ₁ even on the choice of model identification label to each area is not changed. That is, in this case, the C ₂ clique potential affects the energy.

FIG. 2 is a schematic diagram showing a square cluster of one side r, and four peripheral cliques. The number of C ₁ in the cluster of FIG. 2 is counted as follows.

2 creeks at the top. There are 4 vertices.
There are 3 creeks on the side. There are 4 (r-2) points on the side.
4 creeks inside. Inside the point ^(r-2) 2 pieces.
If the number of points is taken into consideration, the equation (34) can be derived.

Further, the number of C ₂ in the region (cluster) of FIG. 2 is counted as follows.

2 creeks at the top. There are 4 vertices.
One creek at the point on the side. There are 4 (r-2) points on the side. The number of internal points is (r-2) ^2. Accordingly, the equation (35) is derived.

FIG. 3 is a graph showing the number of cliques C ₁ composed of points included in the same region and the number of cliques C ₂ composed of different points as a percentage of the total number of cliques in the region. As can be seen from FIG. 3, as the size of the side r increases, the influence of C ₂ decreases and the proportion of C ₁ increases.

From this, it can be seen that as the side r increases, the influence of C _{2 on} the regularization energy U ₂ decreases and C ₁ becomes dominant, that is, the regularization energy U ₂ is no matter how the region label is selected. It turns out that it does not change greatly. Therefore, when the edge r is large, the influence of the regularization energy U _{2 is} relatively small relative to the total energy, the influence of the regularization energy U ₂ is large, and the energy minimization problem is the maximum likelihood estimation for U ₁ . Close to the result.

On the other hand, when the side r is large, the accuracy for the label selection of the maximum likelihood energy U ₁ is high due to the law of large numbers. Therefore, it can be said that the maximum likelihood estimation for the maximum likelihood energy U ₁ is sufficient when the side r is large.

  Although FIG. 3 shows an example with a square area, the present invention is not limited to this, and the same applies to areas other than the square. This is because the proportion of the outer circumference decreases rapidly as the area increases.

  However, if the region is, for example, sesame salt and has a very large number, there is a high possibility that it will not be as described above, but this can be avoided by making the region.

FIG. 4 is a schematic diagram showing that the regularization energy U ₂ is not considered in the region where the side r is large, and the regularization energy U ₂ is considered in the region where the side r is small. As shown in FIG. 4, it can be seen that the regularization energy U ₂ is considered in the boundary region which is a region having a small side r.

  Specifically, the label allocating unit 111 initializes the model identification label z (n) to −1 as shown in Equation (36), and sets the region (in order from the largest region, that is, from the largest number of pixels in the region). Cluster) K is selected. Then, the likelihood energy is calculated by the equation (38), the regularization energy is calculated by the equation (39), the label value is calculated by the equation (37), and the model identification label z (n) is calculated by the calculated label value. Is updated as shown in equation (40). The model identification label is assigned by repeating the calculation process and the update of the model identification label for all regions.

Here, according to the equation (38), when the region is large, the equation (37) works as maximum likelihood estimation itself, and the prior density gradually affects as the region becomes small.

  Here, when there are a plurality of regions of the same size (that is, regions having the same number of pixels), the regions are selected in ascending order of the number of pixels at the boundary of the region.

  Next, a motion estimation process performed by the motion estimation apparatus according to the first embodiment configured as described above will be described. FIG. 5 is a flowchart of a motion estimation process procedure according to the first embodiment.

  First, the count number L is initialized to 1 (step S11), and local motion estimation processing is performed using equations (2) and (3) (step S12).

  When the local motion estimation process is completed, a parametric model is defined by the model definition unit 108 (step S13). Then, motion clustering processing is performed by the clustering processing unit 109 (step S14), and a motion vector of each point on the target image is clustered to obtain an initial value of label assignment.

  Next, model parameter estimation processing is performed by the parameter estimation unit 110 (step S15), and model parameters for each label area are estimated. Then, the region dividing unit 112 performs region dividing processing (step S16), clusters the frames of the target image into regions based on the pixel values, and assigns region labels to each region.

  Next, label allocation processing is performed by the label allocation unit 111 (step S17), the parametric model to be allocated to the area (cluster) divided by the area division unit 112 is selected, and the model identification label of the selected parametric model is stored in the area. Assigned to the pixels. Then, the count number L is increased by 1 (step S18), and it is checked whether or not the count number L exceeds a predetermined number of times (step S19).

  When the count number L does not exceed the predetermined number of times (step S19: No), the processing from step S15 to S18 is repeated.

  On the other hand, when the count number L exceeds a predetermined number in step S19 (step S19: Yes), the optimum parametric model is selected for each region, and the image processing unit 113 Motion estimation is performed using an optimal parametric model, and a motion estimation result is output (step S20).

  Next, the motion clustering process in step S14 will be described. FIG. 6 is a flowchart showing the procedure of the motion clustering process.

  First, the clustering processing unit 109 performs clustering on the target image into a plurality of images by the k-Means method based on pixel values such as luminance and divides the image to obtain a model identification label (step S21).

  And based on the model identification label by the clustering result, an initial model parameter is estimated and several initial model parameters are calculated | required (step S22). Next, the model identification label α of the model parameter is obtained by numerical analysis processing by the k-Means method using the obtained initial model parameter as an initial value (step S23). Here, a plurality of model identification labels α are obtained corresponding to a plurality of initial model parameters. Such motion clustering processing is executed only once immediately after the local dynamic matching processing is completed, as shown in FIG.

  Next, the model parameter estimation process in step S15 will be described. FIG. 7 is a flowchart showing a procedure of model parameter estimation processing.

First, the parameter estimation unit 110 calculates A by equation (19) and b by equation (20) for all lattice points n for the model identification label α obtained by the clustering processing unit 109 ( Step S31). Then, the equation (21) is solved by numerical analysis using the singular value decomposition method (or LU method) (step S32). And the process of said step S31 and S32 is repeatedly performed with respect to all the model identification labels (alpha) (
Step S33).

  Next, the label allocation process in step S17 will be described. FIG. 8 is a flowchart showing the procedure of label allocation processing.

  First, the label assigning unit 111 initializes the model identification label z (n) to −1 as shown in the equation (36) (step S41). Then, the region K is selected in the descending order of the region, that is, in the descending order of the number of pixels in the region (step S42). At this time, if there are a plurality of regions of the same size (regions having the same number of pixels), the regions are selected in ascending order of the number of pixels at the boundary of the region.

  Then, likelihood energy is calculated from the equation (38) (step S43), and regularization energy is calculated from the equation (39) (step S44). Then, the maximum likelihood estimation is performed by the equation (37) to calculate the label value (step S45). Then, the model identification label z (n) is updated with the calculated label value according to the equation (40) (step S46).

  Next, it is determined whether or not the label allocation process has been completed for all areas (step S47). If the label allocation process has not been completed for all the areas (step S47: No), the next largest area (that is, the next largest pixel number) is selected (step S48). The processes from step S43 to S46 are repeated. As a result, label assignment is performed.

  FIG. 9 shows an example of the target image, FIG. 10 shows an example of the reference image, and FIG. 11 shows the result of area division. FIG. 12 shows the result of label allocation processing by a conventional method as a comparative example when the target image of FIG. 9 and the reference image of FIG. 10 are used. FIG. 13 shows the result of label assignment processing in the first embodiment when the target image of FIG. 9 and the reference image of FIG. 10 are used.

  As can be seen from a comparison between FIG. 12 and FIG. 13, it can be seen that the motion estimation result according to the first embodiment performs label assignment with higher accuracy than the comparative example.

  As described above, in the motion estimation apparatus 100 according to the first embodiment, by performing maximum likelihood estimation using the equations (36) to (39), when the area is large, the maximum likelihood estimation itself is obtained, and the area becomes small. Since the prior density is gradually influenced, the optimal model identification label is assigned to the entire image, and the motion estimation process using the optimal parametric model is performed with high accuracy. Can do.

(Embodiment 2)
In the first embodiment, in the course of the motion estimation process, a parametric model is defined, a model parameter is estimated to generate a parametric model, and the motion estimation process is performed. The image processing apparatus is applied to a moving region detection apparatus that performs background difference.

  In the background subtraction method, a moving area is detected by taking a difference between a background image captured in advance and a target image including the moving area.

  FIG. 14 is a block diagram of a functional configuration of the moving region detection apparatus 1400 according to the second embodiment. The moving region detection apparatus according to the present embodiment detects a moving region that is a moving region from a target image that is input using a background image that is a pre-captured image. FIG. As shown in FIG. 3, the image processing apparatus mainly includes an area dividing unit 1412, a label assigning unit 1411, a moving area detecting unit 1401, a frame memory 106, and a memory 103.

  Similar to the first embodiment, the region dividing unit 1412 according to the present embodiment divides the target image into a plurality of regions.

  The label assigning unit 1411 is a processing unit that assigns region type labels to the regions (clusters) divided by the region dividing unit 112. Here, the area type label is a label for identifying whether each area is a background area or a moving area. In this embodiment, an area type label α indicating a background area is set to α = 0. The area type label α indicating the moving area is set to α = 1. However, the value of the area type label is not limited to this.

  More specifically, the label assigning unit 1411 is divided when the difference value d (n) between the pixel value of the point n of the target image and the pixel of the point n of the background image is equal to or less than a predetermined threshold T. The area type label of the background area is assigned to the area. At this time, if the area type label indicating the background area is assigned to all the areas around the area, the threshold T is increased, When the area type label indicating the moving area is assigned to all the areas, the threshold value T is decreased. Then, the label allocation unit 1411 executes label allocation processing for allocating such area type labels to the pixels in the area in order from the largest number of pixels in the plurality of areas.

Hereinafter, details of the label allocation processing by the label allocation unit 1411 will be described. The pixel value of the point n of the target image is g ₁ (n), and the pixel value of the point n of the background image is g ₂ (n). At this time, the difference value d (n) of the background region is expressed by equation (41). If the noise follows a Gaussian distribution N (0, σ ² ), the background area difference value d (n) also follows the Gaussian distribution.

On the other hand, the difference value of the moving area does not follow the Gaussian distribution. If the point n satisfies the equation (42), it is considered to be a background region with a probability of 95% when the threshold value T = 1.96σ (σ is a constant). On the other hand, when the point n does not satisfy the equation (42), it is considered to be a moving region.

The average difference value in the region K _k is defined by equation (43).

Here, N (k) is the number of pixels in the region. When the region K _k belongs to the background region, the average difference value follows the Gaussian distribution of Equation (44) from the central limit theorem. That is, the accuracy is improved by the magnification indicated by the equation (45). Therefore, it can be seen that the region with a large area has higher labeling accuracy.

However, the accuracy of a region having a small area is not improved so much. Therefore, in the present embodiment, a label allocation process in which a spatial constraint condition is introduced is performed as in the first embodiment. In the present embodiment, label allocation processing is performed by a method different from that in the first embodiment.

  If the surroundings of an arbitrary area are all the background area and the area type label α = 0 is assigned, the label assignment is considered to be more stable if it is considered that this area also becomes the background area. On the other hand, if the area around the arbitrary area is all a moving area and the area type label α = 1 is assigned, the label allocation is considered to be more stable if it is considered that this area also becomes a moving area. In the following, this concept is formulated.

  When the area around the determination target area is determined as the background area, the target area is also determined as the background area by satisfying the expression (46) for an arbitrary constant η> 0. It becomes easy to judge. That is, since the value of the threshold value T is increased by the equation (46), there is a high possibility that the difference value d (n) of the pixels in the determination target region is determined as the background region by the equation (42).

Similarly, when the region around the target region is determined to be a moving region, the target region can be determined by satisfying the expression (47) for an arbitrary constant β> 0. It becomes easy to determine the moving area. That is, since the value of the threshold value T is reduced according to the equation (46), there is a high possibility that the difference value d (n) of the pixel in the determination target region is determined as the moving region according to the equation (42).

Here, η and β should be determined according to the situation of the area around the area to be determined. Therefore, when the expressions (46) and (47) are combined and the threshold T for the region K _k is defined, the expression (48) is obtained.

Here, γ (k) is determined so that γ (k)> 0 if there are many background regions around the region K _k , and γ (k) <0 if there are many moving regions. For example, the value range of γ (k) is −σ ≦ γ
If (k) ≦ σ, it can be designed as shown in equation (49).

When all the region type labels of the regions adjacent to the determination target region are background regions (α = 0), N (α = 0) = B (k) and N (α = 1) = 0 are satisfied. , Γ (k) = σ.

  On the other hand, when the region type labels of the regions adjacent to the determination target region are all moving regions (α = 1), N (α = 1) = B (k) and N (α = 0) = 0. Therefore, γ (k) = − σ. Further, when the number of background regions and moving regions is the same in the adjacent regions, N (α = 0) = N (α = 1) is established, and therefore γ (k) = 0.

  Therefore, when the area type label assigned to each area is indicated by z (n), the label assigning unit 1411 according to the second embodiment initializes the area type label z (n) assigned to each area according to the equation (50), The following processing is performed in descending order of the number of pixels in the area. The number of boundary areas B (k), the area type label of the area adjacent to the determination target area is the number N (α = 0), and the area type label of the area adjacent to the determination target area is the moving area. A certain number N (α = 1) is calculated, γ (k) is calculated from equation (51), equation (52) is calculated, and a label value is obtained from equation (53). Then, the region type label z (n) assigned to the region is updated with the obtained label value according to the equation (54).

Based on the result of the label assigning unit 1411, that is, the region type label z (n) assigned to each region, the moving region detection unit 1401 uses the region type label z (n) = 0 among the regions of the target image. A processing unit for detecting an area.

  Next, the moving area detection process according to the second embodiment configured as described above will be described. FIG. 15 is a flowchart illustrating the procedure of the moving region detection process according to the second embodiment.

  First, the count number L is initialized to 1 (step S51). Then, the region dividing unit 1412 performs region dividing processing (step S52), and divides the frame of the target image into regions based on the pixel values (clustering).

  Next, the label allocation unit 1411 performs label allocation processing for allocating area type labels to the divided areas (step S53). Then, the count number L is increased by 1 (step S54), and it is checked whether or not the count number L exceeds a predetermined number of times (step S55).

  When the count number L does not exceed the predetermined number of times (step S55: No), the processing from step S52 to S54 is repeated.

  On the other hand, if the count number L exceeds a predetermined number in step S55 (step S55: Yes), an area type label has been assigned to the area, and the moving area detecting unit 1401 An area where the type label is 0 is detected as a moving area (step S56), and the detection result is output.

  Next, the label allocation process in step S53 will be described. FIG. 16 is a flowchart of a label assignment process according to the second embodiment.

  First, the label assigning unit 1411 initializes an area type label z (n) to be assigned to an area to −1 as shown in Expression (50) (step S61). Then, the region K is selected in the order from the largest region, that is, from the largest number of pixels in the region (step S62). At this time, when there are a plurality of regions having the same number of pixels, clusters are selected in ascending order of the number of pixels at the boundary of the region.

  Then, the number B (k) of boundary regions, the number N (α = 0) where the adjacent label of the cluster is the background region, and the number N (α = number of region type labels of the adjacent regions of the determination target region) 1) is calculated (step S63). Next, γ (k) is calculated from equation (51) (step S64), and an average difference value is calculated from equation (52) (step S65). Then, a label value is obtained from the equation (53) (step S66), and the region type label z (n) is updated with the obtained label value from the equation (54) (step S67).

  Next, it is determined whether or not the label allocation process has been completed for all areas (step S68). If the label allocation processing has not been completed for all the regions (step S68: No), the region with the next largest number of pixels is selected (step S69), and the processing from steps S63 to S47 is repeated. Do. As a result, label assignment is performed.

  As described above, in the moving region detection apparatus according to the second embodiment, a plurality of label allocation processes for assigning region type labels that identify the background region or the moving region to the pixels in the region using the equations (51) to (53) are performed. Since the processing is performed in the order from the largest number of pixels, the background region and the moving region can be stably assigned to each region of the image regardless of noise and the like over the entire image. Thus, it is possible to perform image processing with high accuracy by assigning the optimum label.

(Embodiment 3)
In the third embodiment, the image processing apparatus of the present invention is applied to a stereo matching apparatus.

  FIG. 17 is a functional configuration diagram of the stereo matching device 1700 according to the third embodiment. The stereo matching device 1700 according to the present embodiment inputs the first parallax image for the left eye and the second parallax image for the right eye, captures the two parallax images, and estimates the depth of the space from the corresponding points between the images. To do.

  As shown in FIG. 17, stereo matching apparatus 1700 according to the present embodiment mainly includes region dividing unit 112, label allocation unit 111, corresponding point estimation unit 1701, frame memory 106, and memory 103. ing.

  In the present embodiment, a plurality of parametric models are stored in the memory 103 in advance. In the present embodiment, the parametric model shown in Equation (55) is stored in advance, and the label assignment unit 111 differs from Embodiment 1 in that label assignment is performed using the parametric model of Equation (55). Other processes are the same as those in the embodiment.

Here, n ₁ is the coordinates of a point on the first parallax image, and n ₂ is the coordinates of a point on the second parallax image. Α is a model parameter and a model identification label for identifying a parametric model. That is, in the present embodiment, the model parameter and the model identification label are the same α.
In order to estimate the depth of the space in stereo matching, corresponding points (motion) between parallax images are estimated. In this case, the relationship between corresponding points can often be approximated only by left and right translation. For example, when it is known in advance that the left-right translation is performed in the range of 0 to 16, the translation amount can be set in the range of {0,..., 16}. In the present embodiment, α, which is a model parameter and a model identification label, is set as a translation amount in a range of {0,..., 16}.

  The corresponding point estimation unit 1701 is a processing unit that estimates a corresponding point (motion) between parallax images based on the result of the label assignment process.

  Next, stereo matching processing according to the present embodiment configured as described above will be described. FIG. 18 is a flowchart of a stereo matching process according to the third embodiment.

First, the count number L is initialized to 1 (step S71). Then, the area dividing unit 11
2 is performed (step S72), the frame of the first parallax image is divided into regions based on pixel values (clustering), and region labels are assigned to the regions.

  Next, label allocation processing is performed by the label allocation unit 111 (step S73), a parametric model to be allocated to the area (cluster) divided by the area division unit 112 is selected, and the model identification label of the selected parametric model is stored in the area. Assigned to the pixels. Then, the count number L is increased by 1 (step S74), and it is checked whether or not the count number L exceeds a predetermined number of times (step S75).

  When the count number L does not exceed the predetermined number of times (step S75: No), the processing from step S72 to S74 is repeated.

  On the other hand, when the count number L exceeds a predetermined number in step S75 (step S75: Yes), the optimum parametric model is selected for each region, and is handled by the optimum parametric model. A point is detected (step S76), and the corresponding point estimation unit 1701 estimates the depth of the space.

  Here, the label allocation processing in step S73 is performed in the same manner as in the first embodiment using the parametric model of equation (55).

  As described above, in the stereo matching device 1700 according to the third embodiment, the maximum likelihood estimation itself is performed when the cluster is large by using the parametric model of the equation (55) and performing the maximum likelihood estimation by the equations (36) to (39). Since the prior density gradually affects as the cluster gets smaller, optimal label allocation is performed over the entire image, and corresponding points are estimated with high accuracy. Therefore, the depth can be estimated with high accuracy, and stereo matching processing using the parametric model can be performed with high accuracy.

  Each of the first to third embodiments includes a control device such as a CPU, a storage device such as a ROM (Read Only Memory) and a RAM, an external storage device such as an HDD and a CD drive device, and a display device such as a display device. And an input device such as a keyboard and a mouse, and has a hardware configuration using a normal computer.

  Each program (motion estimation program, motion region detection program, stereo matching program) executed by each device of the present embodiment is an installable or executable file, a CD-ROM, a flexible disk (FD), The program is recorded on a computer-readable recording medium such as a CD-R or a DVD (Digital Versatile Disk).

  Also, each program (motion estimation program, motion region detection program, stereo matching program) executed by each device of the present embodiment is stored on a computer connected to a network such as the Internet and downloaded via the network. You may comprise so that it may provide. Further, each program (motion estimation program, motion region detection program, stereo matching program) executed by each device of the present embodiment may be provided or distributed via a network such as the Internet.

  In addition, each program (motion estimation program, motion area detection program, stereo matching program) executed by each device of the present embodiment may be provided by being incorporated in advance in a ROM or the like.

  Each program (motion estimation program, motion region detection program, stereo matching program) executed by each device of the present embodiment includes the above-described units (local motion estimation unit, model definition unit, clustering processing unit, parameter estimation unit, The module configuration includes an area dividing unit, a label assigning unit, a moving region detecting unit, and a corresponding point estimating unit). As actual hardware, a CPU (processor) reads and executes a program from the storage medium. Are loaded onto the main memory, and the local motion estimation unit, model definition unit, clustering processing unit, parameter estimation unit, region division unit, label allocation unit, motion region detection unit, and corresponding point estimation unit are stored in the main memory. It is generated on the device.

  It should be noted that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

1 is a block diagram showing a functional configuration of a motion estimation apparatus according to a first embodiment. It is a schematic diagram which shows the square cluster of 1 side r, and 4 periphery cliques. It is the graph shown by the ratio with respect to the number of each clique of the inside of a cluster, and a boundary, and the total number of cliques in a cluster. It is a schematic diagram which shows the relationship between a cluster and regularization energy. 3 is a flowchart showing a procedure of motion estimation processing according to the first exemplary embodiment; It is a flowchart which shows the procedure of a motion clustering process. It is a flowchart which shows the procedure of a model parameter estimation process. It is a flowchart which shows the procedure of a label allocation process. It is explanatory drawing which shows the example of a target image. It is explanatory drawing which shows the example of a reference image. It is explanatory drawing which shows the result of an area | region division. It is explanatory drawing which shows the label allocation process result by the conventional method as a comparative example. 6 is an explanatory diagram illustrating a label allocation process result according to Embodiment 1. FIG. It is a functional block diagram of the moving region detection apparatus 1400 concerning Embodiment 2. FIG. 10 is a flowchart illustrating a procedure of moving region detection processing according to the second embodiment. 10 is a flowchart illustrating a procedure of label allocation processing according to the second exemplary embodiment. FIG. 6 is a functional configuration diagram of a stereo matching device according to a third embodiment. 10 is a flowchart illustrating a procedure of stereo matching processing according to the third embodiment;

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Motion estimation apparatus 101 Local motion estimation part 103 Memory 106 Frame memory 107 Model generation part 108 Model definition part 109 Clustering process part 110 Parameter estimation part 111,1411 Label assignment part 112,1412 Area division part 113 Image processing part 1400 Motion area detection Device 1401 Moving Region Detection Unit 1700 Stereo Matching Device 1701 Corresponding Point Estimation Unit

Claims (14)

A region dividing unit that divides the first image into a plurality of regions each including a plurality of pixels;
A model for associating a point on the first image with a point on the second image, and a plurality of parametric models including a plurality of model parameters and a model identification label for identifying the model Selecting the parametric model that minimizes the value of the evaluation function indicating the degree of fitness with the parametric model and the smoothness indicating whether the model identification label changes between adjacent pixels. A label allocating unit that executes a process of label allocation for allocating the model identification label of a model to a pixel in the region, in order from the largest number of pixels in the plurality of regions;
An image processing unit for obtaining a correspondence relationship between the first image and the second image based on the parametric model corresponding to the model identification label assigned to each pixel;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, further comprising a model generation unit that generates the parametric model based on characteristics of the first image. The model generation unit
A model definition section for defining the parametric model;
A clustering processing unit that classifies corresponding points on the second image of each point in the first image, and assigns an initial value of the label assignment to each of the classified corresponding points;
For each model identification label, the model parameter that can represent each point on the first image and the corresponding point on the second image corresponding to each point with a minimum error is estimated by the least square method. A parameter estimator;
The image processing apparatus according to claim 2, further comprising:   The image processing apparatus according to claim 3, wherein the model definition unit defines the parametric model by defining a plane-to-plane geometric transformation from the first image to the second image. . Local estimation of motion between the first image and the second image to obtain a motion vector of the point on the second image of each point in the first image from the corresponding point of each point A motion estimation unit;
The parametric model is a model for obtaining a motion vector of a point on the second image of each point in the first image in the area of the model identification label,
The clustering processing unit classifies the motion vector obtained by the local motion estimation unit, and assigns an initial value of the label assignment for each classified motion vector,
The parameter estimation unit estimates the model parameter for each model identification label based on the motion vector determined by the parametric model and the motion vector obtained by the local motion estimation unit,
The said image processing part estimates the motion with respect to the said 2nd image of the said 1st image based on the said parametric model corresponding to the said model identification label allocated to each pixel. An image processing apparatus according to 1.   2. The label allocation unit according to claim 1, wherein when there are a plurality of the regions having the same number of pixels, the label allocation unit performs the label allocation processing in ascending order of the number of pixels at the boundary of the region. Image processing apparatus. A storage unit for storing the parametric model;
The image processing apparatus according to claim 1, wherein the model parameter is set in advance according to a problem.   The parametric model is a model for obtaining a point on the second parallax image as the second image with respect to a point on the first parallax image as the first image. The image processing apparatus according to claim 1, comprising a point and a horizontal movement amount of the point of the second parallax image corresponding to the point as the model parameter and the model identification label. An area dividing unit that divides a target image including a moving area that is a moving area with respect to a background image that is an image without movement into a plurality of areas each including a plurality of pixels;
A label allocating unit that executes a process of label allocation for allocating a region type label for identifying a background region or a moving region to pixels in the region, in order from the largest number of pixels in the plurality of regions;
Based on the region type label assigned to each pixel, a moving region detector that detects the moving region from the target image;
An image processing apparatus comprising:   The label allocation unit allocates the area type label of the background area to the area when a difference value between the pixel of the target image and the pixel of the background image is equal to or less than a predetermined threshold, When the area type label indicating the background area is assigned to all areas, the threshold value is increased, and the area type label indicating the moving area is assigned to all areas around the area. The image processing apparatus according to claim 9, wherein in the case, the threshold value is increased. Dividing the first image into a plurality of regions each including a plurality of pixels;
A model for associating a point on the first image with a point on the second image, and a plurality of parametric models including a plurality of model parameters and a model identification label for identifying the model Selecting the parametric model that minimizes the value of the evaluation function indicating the degree of fitness with the parametric model and the smoothness indicating whether or not the model identification label changes between adjacent pixels. Performing a process of assigning a label for assigning the model identification label of a model to a pixel in the region in order from the largest number of pixels in the plurality of regions;
Obtaining a correspondence relationship between the first image and the second image based on the parametric model corresponding to the model identification label assigned to each pixel;
An image processing method comprising: Dividing a target image including a moving area that is a moving area with respect to a background image that is an image without movement into a plurality of areas each including a plurality of pixels;
A step of performing label assignment processing for assigning a region type label for identifying a background region or a moving region to pixels in the region in order from the largest number of pixels in the plurality of regions;
Detecting the moving region from the target image based on the region type label assigned to each pixel;
An image processing method comprising: Dividing the first image into a plurality of regions each including a plurality of pixels;
A model for associating a point on the first image with a point on the second image, and a plurality of parametric models including a plurality of model parameters and a model identification label for identifying the model Selecting the parametric model that minimizes the value of the evaluation function indicating the degree of fitness with the parametric model and the smoothness indicating whether the model identification label changes between adjacent pixels. A procedure for performing label assignment processing for assigning the model identification label of a model to pixels in the region in order from the largest number of pixels in the plurality of regions;
Obtaining a correspondence relationship between the first image and the second image based on the parametric model corresponding to the model identification label assigned to each pixel;
An image processing program for causing a computer to execute. Dividing a target image including a moving area that is a moving area with respect to a background image that is an image without movement into a plurality of areas each including a plurality of pixels;
A procedure of executing label assignment processing for assigning an area type label for identifying a background area or a moving area to pixels in the area in order from the largest number of pixels in the plurality of areas;
A procedure for detecting the moving area from the target image based on the area type label assigned to each pixel;
An image processing program for causing a computer to execute.