JPH0561977A - Area extracting device - Google Patents

Abstract

PURPOSE:To realize an image area extracting device capable of stably performing the area extraction of an image by using a method in which image characteristic and the adequate contour of shape can be obtained in process to minimize energy, that is a method of so-called dynamic contour model. CONSTITUTION:The Mahalanobis distance for the mean of picture element data in a supplied initial contour is calculated. Thereby, the dynamic contour model is controlled, and the area extraction is performed. Such control and extraction can be performed by providing an edge strength arithmetic means 207 which finds edge strength by a differential value based on the Mahalanobis distance between the picture elements in the direction of outside the area and that of inside the area for the candidate position of a dynamic contour, and a shape complexity arithmetic means 208 which finds the complexity of contour shape. The contour position for the candidate position of the contour is changed so as to minimize total energy obtained by the linear sum of energy due to the shape complexity and the energy due to the edge strength by dynamic programming.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a region extraction device for editing and processing images, for example.

[0002]

2. Description of the Related Art Area extraction methods are complex and diverse, but methods for image editing and processing include, for example, "Image Processing Application for Broadcasting", The Institute of Electronics, Information and Communication Engineers Vol.74, No.4. , p
p.386-391, (1991) and "Image Processing in Presentations", ibid, pp, 392-397. Listed below are the following.

1) The contour is manually specified for each pixel. 2) Roughly search and connect edges near a given contour by hand.

3) Consider a three-dimensional color space for pixels near the contour and create a histogram in the three-dimensional color space. In the three-dimensional histogram, the one having good separability is sequentially divided.

4) Using the color information, a person explicitly designates a representative point inside the object and a representative point outside the object, and a discrimination plane is calculated at these representative points and separated.

5) An attempt is made to obtain a smooth boundary by minimizing the energy shown in (Equation 1), which is called an active contour model.

[0007]

[Equation 1]

In (Equation 1), s is a line element on the contour, the first term on the right-hand side is energy related to the size and smoothness of the contour (larger and larger as it is not smooth), and the second term is energy due to matching with the image. The third term is the energy for forcibly changing the contour position (the larger the non-conformance, the larger). The contour position is obtained in the process of energy minimization.

[0009]

A first problem to be solved by the present invention is to linearize each energy that determines the motion of an active contour model when an area is extracted using the active contour model. It is difficult to set the coupling coefficient.

The second problem is that when considering the introduction of the Mahalanobis distance to the average vector of the region to be extracted so that the value of the image energy is not affected by the change of the target image, the calculation of the Mahalanobis distance is the covariance. When the rank of the matrix is falling, it cannot be calculated as it is.

The third problem is that the image area to be extracted is generally a projection of an object in which a plurality of objects overlap, but if this is obtained by one dynamic contour, the contour becomes discontinuous. It is impossible to deal with it.

An object of each invention of the present application is to provide an image area extracting device which solves the above problems and stably extracts an area in an image.

[0013]

According to a first aspect of the invention, in order to separate an image based on a region Mahalanobis distance, an eigenvector and an average vector are used to obtain n pixels of each pixel of an input image.
Orthogonal transformation means for obtaining the KL expansion of the dimensional feature amount and the eigenvector, if the eigenvalue is less than the threshold value for a predetermined eigenvalue, the component corresponding to the eigenvector in the KL expansion of each pixel. A first area determining unit that excludes a pixel having a large value from the extraction target, and a component whose eigenvalue is greater than or equal to a threshold value with respect to the predetermined eigenvalue for pixels that are not excluded by the first area determining unit. The KL partial expansion is provided with a Mahalanobis distance calculating means for calculating the Mahalanobis distance and a second area determining means for outputting a pixel with a small Mahalanobis distance as an area.

A second invention is a device for extracting a region for controlling an active contour model based on Mahalanobis distance,
Therefore, regarding the candidate position of the active contour, the edge strength calculation means for obtaining the edge strength from the Mahalanobis distance between the pixels in the direction outside the area and the direction in the area and the complexity of the contour shape between the candidate positions adjacent on the contour For the candidate position, a shape complexity calculating means and an optimizing means for obtaining an optimum sequence on the contour using a linear combination of the edge strength and the complexity of the contour shape as an evaluation measure are provided.

In a third aspect of the invention, in order to solve the discontinuity of the contour due to the overlapping of objects, the overlapping is determined for a plurality of candidate positions of the contour, and a part of the contour is shielded by another region. And a means for determining an occluded portion for detecting the optimum sequence on the contour using the linear combination of the edge shape complexity and the edge strength at the candidate position excluding the occluded portion as an evaluation scale. is there.

[0016]

In the first invention, the covariance matrix calculation means allows
Obtain the eigenvalues and eigenvectors required to obtain the KL expansion by eigenvalue analysis of the covariance matrix. From the sequence of eigenvectors
After obtaining the KL transformation matrix, the KL expansion is obtained for the pixels of the input image. Among the eigenvalues, those that are less than a threshold value for a predetermined eigenvalue are considered to have a lower rank in the KL transformation matrix. If the component corresponding to the small eigenvalue is large in the KL expansion of each pixel, it is excluded by the first region determining unit. This is because the obtained Mahalanobis distance becomes large. For the remaining pixels, calculate the Mahalanobis distance using the KL transformation matrix with a reduced rank,
A region having a small value is extracted as a region by the region determining unit.

In the second invention, the mean vector and the covariance matrix are calculated for the area to be extracted, and the Mahalanobis distance calculating means obtains the distance near the candidate point of the contour position. The edge strength calculating means is obtained from the Mahalanobis distance inside and outside the candidate point, and a new contour position is obtained by the optimizing means that optimizes the evaluation scale determined by the edge strength and the complexity of the contour shape by the sum on the contour. Then, the area surrounded by the contour is output by the contour determining means.

In the third invention, the scale of the edge strength is not specified, but the region to be extracted is expressed by a plurality of contours, which is different from the second invention. The shielded portion determination means determines overlap for a plurality of contour candidate positions,
It is determined whether a part of the contour is covered by another area.
In the shielded portion, the edge strength is ignored by the optimizing means for calculation. Then, the area determining means extracts the area as the sum of the areas represented by the plurality of coordinate sequences.

[0019]

Embodiments Embodiments of the first, second and third inventions will be described below with reference to the drawings.

FIG. 1 shows a block diagram of an embodiment of the first invention. In FIG. 1, 101 is an image memory, 102 is a sample point extraction unit, 103 is a covariance matrix calculation unit, 104 is a principal component calculation unit, 105 is a difference calculation unit, 106 is an orthogonal transformation unit, 107 is an eigenvalue determination unit, and 108 is A Mahalanobis distance calculation unit, and 109 is a region extraction unit. The operation of the first invention configured as above will be described.

The input image is a color image, which is composed of a luminance component Y and two color difference components I and Q.
If the RGB values are r, g, and b, respectively, the YIQ value is expressed by (Equation 2).

[0022]

[Equation 2]

The color image is stored in the memory 101, and the sample point extracting unit 102 extracts a part of the area to be extracted. The set of extracted pixels is represented by S. Here, the covariance matrix R is (Equation 3), and the covariance matrix calculation unit 1
Obtained with 03.

[0024]

[Equation 3]

E () means an expected value or an average. Next, by the principal component analysis unit 104, the eigenvalue λ ₁ > λ ₂ shown in (Equation 4)
> λ ₃ and the eigenvector c _i (1 ≦ i ≦ 3) are calculated.

[0026]

[Equation 4]

Using the average vector E (x) obtained by the covariance matrix calculation unit 103, the difference calculation unit 105 calculates the difference between the feature vector and all the pixels of the image. Further, the orthogonal transformation unit 106 finds the KL expansion. If this is put together, it will become (Equation 5).

[0028]

[Equation 5]

Based on the eigenvalues obtained by the principal component analysis section, the eigenvalue determination section performs the operations shown in (Table 1) and (Table 2).

[0030]

[Table 1]

[0031]

[Table 2]

In Table 2, ε is the expected value of the quantization error. The constants of 7.88, 10.60, and 12.84 in the table are calculated as boundary values within 99.5% from the probability density function of the χ ² distribution with 1, ^{2, and} 3 degrees of freedom. In this embodiment, it is assumed that each element of the KL expansion has a Gaussian distribution. The Mahalanobis distance calculation unit 108 obtains the Mahalanobis calculation distance Md shown in (Equation 6).

[0033]

[Equation 6]

Di is the i-th component of (Equation 6) d, and N is the order determined in Table 1. When the order is zero, the Mahalanobis distance is all zero. The area extraction unit determines the pixels that have passed the eigenvalue evaluation unit by the threshold value of the Mahalanobis distance to the pixels, and outputs the one with the smaller value as the area.

The Mahalanobis distance has a χ ₂ distribution with three degrees of freedom based on its Gaussian assumption. Therefore, the Mahalanobis distance can be generally calculated by using the covariance matrix R (Equation 7). However, if the number of samples is small, or if a large number of regions are desired to be extracted, the rank of R decreases.

[0036]

[Equation 7]

In this case, the Mahalanobis distance cannot be obtained. In order to avoid this, in this embodiment, the eigenvalue of the covariance matrix is examined. 12.84 is the χ ² distribution with 3 degrees of freedom, which is 99.
It represents the distance that includes 5% of the points. Therefore, since the Mahalanobis distance is calculated by (Equation 6), if the third eigenvalue is 12.84 or less and the third component of the KL expansion is non-zero, the attribution to that pixel is at a risk rate of at least 0.5%. Will be rejected. Considering the quantization error, the pixels to be rejected are calculated as shown in Table 2. Thereafter, for the second component and the first component, if the eigenvalue is small, the same rejection is performed, and for the remaining pixels, the Mahalanobis distance is obtained by the partial KL expansion, and the region having a small distance is obtained as the in-region pixel.

As described above, according to the present embodiment, since the Mahalanobis distance is not directly calculated from the covariance matrix and threshold processing is performed by the eigenvalue of the principal component analysis, even if the rank of the covariance matrix falls, the χ ² distribution Area extraction can be performed by a statistical measure assuming.

Next, the second invention will be described with reference to the second embodiment. FIG. 2 is a block diagram of this embodiment. In Figure 2
201 is a memory for storing the YIQ value (image data) of the image, 202 is a sample extracting unit for reading the image data in the contour coordinates, 203 is a covariance calculating unit for calculating the covariance matrix and the mean vector of the sample set 204 Is the Mahalanobis distance calculator that calculates the Mahalanobis distance using the mean vector and the covariance matrix, 205 is the coordinate string memory that holds the contour as a coordinate string, and 206 is the new coordinate position for the currently held coordinate string. A coordinate sequence candidate position calculation unit that calculates different coordinate positions, 207 is an edge strength calculation unit that calculates edge strength based on the Mahalanobis distance, and 208 is a shape complexity calculation unit that calculates the shape complexity of the contour that is currently a candidate, Reference numeral 209 denotes a new coordinate position calculation unit that selects an optimum position for edge strength and shape complexity for the new coordinate position of the contour, and writes it in the memory 205 as a new contour coordinate sequence. It

An embodiment configured as above will be described below. The coordinate memory 205 has a length N as shown in (Equation 8).
The coordinate series of is stored.

[0041]

[Equation 8]

This initial coordinate series is used for devices such as a mouse.
It should be roughly given by hand.
It The sampling unit 202 selects the area enclosed by this initial contour.
Read the YIQ image data. Co-division of the sample
The covariance calculation unit 203 obtains a dispersion matrix and an average vector. D
Image position input from the edge strength calculator
The covariance matrix and mean vector obtained by the Nobis distance calculator 204
Outputs the Mahalanobis distance according to Tor. In addition, this performance
The calculation is performed by the same procedure shown in the first embodiment. However
However, in the first embodiment, the rank of the covariance matrix has dropped.
When the corresponding pixels are excluded under the conditions shown in (Table 1) and (Table 2)
But here it is big enough, for example 8bit code
None Mahalanobis distance such as maximum 255 in fixed point
give. As shown in FIG. 4, the coordinate sequence candidate position calculation unit
± 2 pixels in the direction perpendicular to the coordinate positions before and after the contour
Calculate the position. Candidate position v_iIs located 1 pixel outside
Pixel to v_i ⁺Then, the edge strength calculation unit is Mahalanovi
Value E shown in (Equation 9) based on the distance obtained from the distance calculation unit 204
_imageIs calculated.

[0043]

[Equation 9]

In (Equation 9), f () is the probability density function of the χ ² distribution and χ () is the Mahalanobis distance at that position. Since the Mahalanobis distance is χ ² distributed,
It can be obtained as (Equation 11) using the function of the probability density shown in (Equation 10) (where n is the degree of freedom).

[0045]

[Equation 10]

[0046]

[Equation 11]

This is because the ratio of the probability that a pixel belongs to the area is v _i
^We want to maximize ⁺ and v _i . In the shape complexity calculation unit, the candidate positions v _i-1 and v _{i + 1} that move back and forth
Then, the E _shape shown in ( _Equation 12) is calculated.

[0048]

[Equation 12]

The term α represents the distance between coordinate positions, and the term β represents the evaluation regarding the bend between coordinate positions. The term γ is a term for making the evaluation good if the candidate position is a position outside (+ 2, + 1) of the current position. This gives an outward force, but the contour does not expand infinitely due to the squared term of α. E _image +
The new coordinate position calculation unit determines the position of the pixel of the entire contour so that the value of E _shape becomes the minimum for the entire contour.
The dynamic programming can be applied to this from (Equation 11) and (Equation 12), and its function recurrence formula can be expressed as (Equation 13).

[0050]

[Equation 13]

Using the above calculation, the coordinate position of the contour is updated and written in the coordinate string memory 205. By repeating this processing, the area surrounded by the contour position when the position of the pixel to be changed disappears or the position change amount falls below a predetermined small value is set as the extraction result. As described above, in the present embodiment, the conventionally proposed active contour model is used to obtain the statistical distribution of the area surrounded by the contour and apply the Mahalanobis distance as the edge strength. This makes the absolute value of the edge strength meaningful. Therefore, the values of the coefficients α, β, and γ can be adjusted according to the range of the difference value of the Mahalanobis distance. Further, in the present embodiment, by the optimization by the dynamic programming, it is possible to introduce a constraint such that the contour is not moved to a region having a Mahalanobis distance of 12.84 or more.

Next, the third invention will be described with reference to the third embodiment. FIG. 3 is a block diagram for explaining this embodiment.
In FIG. 3, 301 is a memory for storing the YIQ value (image data) of the image, 302 is a sample extracting unit that reads the image data in the contour coordinates excluding the shielded portion, and 303 is the covariance matrix and average of the sample set. A covariance calculation unit that calculates a vector, 304 is a variance information storage memory that holds a covariance matrix and a mean vector for each different sample set, and 305 is a Mahalanobis distance that is calculated using the mean vector and the covariance matrix with different contour IDs. Mahalanobis distance calculation unit, 306, 307, and 308 are coordinate sequence memories that hold contours as coordinate sequences, 309 is an overlap determination unit that determines the overlap of multiple contours and identifies the shielded portion, and 310 is the Mahalanobis distance. And an edge strength calculation unit that calculates the edge strength using the overlap information, 311 is a shape complexity calculation unit that calculates the shape complexity of the current candidate contour,
Reference numeral 312 denotes a coordinate sequence candidate position calculation unit that calculates a coordinate position that can be used as a new coordinate position for a plurality of coordinate sequences currently held, and 313 selects an edge strength for the new coordinate position of the contour and an optimal position for shape complexity. Then, it is a new coordinate position calculation unit that newly writes it in the memories 306, 307, 308 as a contour coordinate sequence. In the block diagram, the coordinate sequence memory
Although three of 306, 307, and 308 are shown, they are actually composed of many.

The present embodiment configured as described above is different from the second embodiment in that the coordinate positions of a plurality of contours that allow overlapping are updated. It is assumed that the given contour coordinates are overlapped as shown in FIG. The broken line portion of L0 shielded here can be obtained by the correlation of the regions R0, R1, and R2.
That is, whether R1 has a correlation with R0 or R2 is determined by the difference between the average values. This is the overlap determination unit 30
Done at 9. Using the result of the overlap determination unit 309, the sample extraction unit 302 extracts the sample in the contour area excluding the shielded portion. In FIG. 5, only the region of R0 is obtained for L0, and the regions of R1 and R2 are obtained for L1.

The edge strength calculation unit 310 further uses this overlap information and returns the E _image shown in (Equation 13) using the coordinate position and the contour identifier as input. The expansion in log () is the same as (Equation 12). The difference from (Equation 10) is that the shielded portion extends smoothly because the edge strength is set to 0. In the occluded part, the contour position is determined only by E _shape . By setting the edge strength of the shielded portion to 0 by using a plurality of contours in this way, it is possible to accurately represent a discontinuous contour caused by overlapping projections of objects such as the X junction portion in FIG. In the third embodiment, similarly to the second embodiment, the contour position is updated by the new coordinate position calculation unit by the dynamic programming, and the coordinate sequence memory 30 is updated.
Written on 6, 307, 308. The stop condition is that the position of the pixel changed for all of the plurality of contours disappears or falls below a predetermined small value.

The third embodiment will be further described using a processing example. FIG. 6 is an example of an input image. The original image is shown in (a) and the line drawing is shown in (b). Consider extracting the contour l in this image. From background R _B , hair R _H , face R _F , collar R _S1 , ribbon R
_The purpose is to extract the areas of _R , name tag R _P , vest R _V , sleeves R _S0 , R _S2 . Here, the operation of extracting the region of the best R _V will be described with reference to FIGS. 7 and 8. 7A and 7B show the original image and its line drawing. In FIG. 7, l _g is the contour that is the target of region extraction, and l _s is the initial contour roughly given by the user with the mouse. In the present embodiment, 30 iterations of dynamic programming are calculated by the new coordinate position calculation unit 313, and the contour shown by l _e in FIG. 8 is obtained as a coordinate sequence. FIG. 9 shows the above processing performed on the other parts of hair R _H , face R _F , collar R _S1 , ribbon R _R , name tag R _P , sleeves R _S0 and R _S2 shown in FIG. 9 as shown in FIG. 9 (b). Contour l _H , l _F , l
_S1 , l _R , l _P , l _V , l _S0 , l _S2 are obtained. The target contour l shown in FIG. 6B is obtained as the outer circumference of the contour thus obtained. FIG. 9A shows contours l _H , l _F , l _S1 , l _R ,
It is an image corresponding to a region obtained as a common region of l _P , l _V , l _S0 , and l _S2 .

In the second and third embodiments, the optimizing means is realized by using the dynamic programming method, but it may be loosely realized by the framework of the variational method.

In this example, E _image was calculated as the logarithm of the ratio of (Equation 10) as shown in (Equation 11). However, since we assume that the pixel values are multidimensional normally distributed,
E _image ignoring the "log" term, which is the third term of (Equation 11)
May be used.

[0058]

As described above, according to the present invention, when a rough initial contour is given, 1) a dynamic contour contour model for obtaining an edge strength from a standardized distance called a Mahalanobis distance for an image in the initial contour. Parameter setting becomes easy. In addition, it is possible to introduce a constraint based on the absolute value of the Mahalanobis distance to the inside / outside determination of the area.

2) When the target area is uniform or has a decimal number, it is impossible to calculate the Mahalanobis distance because the rank of the covariance matrix drops, but the eigenvalue of the covariance matrix considering the χ ² distribution. The Mahalanobis distance can be calculated in the subspace by the threshold processing.

3) With a plurality of contours, it is possible to deal with the discontinuity of the contours that occurs in the projection of an object in which a plurality of objects overlap.

[Brief description of drawings]

FIG. 1 is a block diagram of a region extraction device according to a first embodiment of the present invention.

FIG. 2 is a block diagram of a region extraction device according to a second embodiment of the present invention.

FIG. 3 is a block diagram of a region extraction device according to a third embodiment of the present invention.

FIG. 4 is an explanatory diagram of changing a contour position.

FIG. 5 is an explanatory diagram of overlapping areas.

FIG. 6A is a diagram showing an original image of an input image example, and FIG. 6B is a diagram showing the line drawing.

7A is a diagram showing an original image of an example of an initial contour, and FIG. 7B is a diagram showing the line drawing.

8A is a diagram showing an original image of an example of contour convergence, and FIG. 8B is a diagram showing the line drawing.

9A is a diagram showing an original image of an example of region extraction by a plurality of contours, and FIG. 9B is a diagram showing the line drawing.

[Explanation of symbols]

 101 image memory 102 sample extraction unit 103 covariance matrix calculation unit 104 principal component calculation unit 105 difference calculation unit 106 orthogonal transformation unit 107 eigenvalue determination unit 108 Mahalanobis distance calculation unit 109 region extraction unit 201 image memory 202 sample extraction unit 203 covariance calculation Part 204 Mahalanobis distance calculator 205 Coordinate sequence memory 206 Coordinate sequence candidate position calculator 207 Edge strength calculator 208 Shape complexity calculator 209 New coordinate position calculator 301 Image memory 302 Sample extractor 303 Covariance calculator 304 Covariance information storage Memory 305 Mahalanobis distance calculation unit 306 307 308 Coordinate sequence memory 309 Overlap determination unit 310 Edge strength calculation unit 311 Shape complexity calculation unit 312 Coordinate sequence candidate calculation unit 313 New coordinate position calculation unit

Claims (3)

[Claims] 1. A sample extracting means for obtaining an n-dimensional (n ≧ 1) feature quantity as a sample vector for pixels of an image area to be extracted or an image area similar to the image area, and (b) being obtained by the sample extracting means. Covariance matrix computing means for obtaining a mean vector and a covariance matrix from the sampled vector, (c) principal component analysis means for obtaining n eigenvectors and their eigenvalues by eigenvalue analysis of the covariance matrix, ) Orthogonal transformation means for obtaining the KL expansion of the n-dimensional feature amount of each pixel of the input image using the eigenvector and the average vector, and (f) in the eigenvector whose eigenvalue is less than the threshold value for the eigenvalue. If there is, in the KL expansion of each pixel, a first area determining unit that excludes a pixel having a large component value corresponding to the eigenvector from the extraction target; and (g) the first area determining unit. Pixels not excluded And a partial KL expansion consisting of components whose eigenvalues are greater than or equal to the threshold value for the eigenvalues, Mahalanobis distance calculation means for calculating Mahalanobis distances, and (h) a second pixel for outputting pixels with small Mahalanobis distances as an area A region extraction device having a region determination means. 2. (a) A sample extracting means for obtaining an n-dimensional (n ≧ 1) feature amount as a sample vector for pixels of an image region to be extracted or an image region similar thereto, and (b) for each pixel of an input image. Mahalanobis distance calculating means for calculating Mahalanobis distance with respect to the average vector of the sample vector, (c) a memory for holding the image region to be extracted as a coordinate sequence of the contour, (d) a new candidate position for the coordinate sequence for each coordinate Candidate position calculating means for calculating, (e) for the candidate position, edge strength calculating means for obtaining an edge strength by Mahalanobis distance between pixels in the area outside direction and the area inside direction, (f) adjacent on the contour Between the candidate positions, a shape complexity calculating means for obtaining the complexity of the contour shape from the difference between the positions, and (h) for the candidate position, a line of the edge strength and the complexity of the contour shape. An area extraction device comprising: optimization means for obtaining an optimum sequence on a contour using shape combination as an evaluation measure; and (g) area determination means for obtaining an area surrounded by the coordinate sequence obtained by the optimization means. 3. (a) Edge detection means for detecting an edge based on a change in brightness of an image, (b) a memory for holding a plurality of coordinate sequences representing an outline, and (c) a new candidate for the coordinate sequence. Candidate position calculating means for calculating a position for each coordinate; (d) edge strength calculating means for obtaining an edge strength using the result of the edge detecting means for the candidate position; and (e) the adjacent candidate on the contour. A shape complexity calculating unit that obtains the complexity of the contour shape between the positions, and (f) the overlap is determined for the candidate positions of the plurality of contours, and a portion where a part of the contour is occluded in another area is determined. Shielded portion determination means to discover, (g) an optimization means for obtaining an optimum sequence on the contour using a linear combination of the edge strength at the candidate position excluding the shielded portion and the contour shape complexity, (h) Coordinate sequence obtained by the optimization means A region extraction device having a region determining means for obtaining a plurality of regions surrounded by.