JPH0793561A - Edge and contour extractor - Google Patents

Abstract

PURPOSE:To stably extract gradation with a few luminance differences, colors, texture edges and contours from supplied images. CONSTITUTION:This extractor is provided with a mask area storage part B extracting the image information of the mask area for a designated point in supplied images, degree of separation calculation parts 22x and 22y calculating a degree of separation from the image information extracted in the mask area storage part B, an edge intensity calculation part 23 calculating the edge intensity of the designated point from the obtained degree of separation and an edge extraction part D extracting the contour from this obtained degree of separation.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an edge and contour extracting device for stably extracting shades, colors and texture edges in a supplied image.

[0002]

2. Description of the Related Art In image processing, the edge extraction technique is
This is the most basic and important elemental technology. Edges reflect the contour of the object, the structure inside the object, the structure of the background, etc. Therefore, in order to realize subsequent high-level processing such as image understanding, the edge is extracted stably and with high accuracy. Is essential.

Various edge extraction methods have been proposed so far. Among them, a typical method is a method based on spatial differential calculation. This method is based on the idea that the position where the brightness changes abruptly corresponds to the edge, as shown in FIG. Typical spatial differential operation-based methods include differential operators such as first-order differential type Roberts, Prewitt, Sobel, second-order differential type, and Lapacaian.
[Latest Trends in Image Processing Algorithms Separate Volume OplusE New Technology Communications (1986)]. In addition, 8 masks are prepared to obtain edge strength and direction. Prewitt, kirsc
The method of H. Robinson et al. is also mentioned [Latest trend of image processing algorithms (1986)]. In addition to these, various edge extraction methods such as Hueckel's model fitting method have been proposed as methods for edge extraction by edge model fitting in a local region.

At present, the edge extraction method which is most used in the practical application field is a method based on a spatial differential operation, but it has a problem that it is easily affected by noise. In particular, when extracting a small brightness gradient (hereinafter referred to as a weak edge), it was difficult to accurately extract it due to the influence of noise.

Further, as shown in FIG. 22, a region where the luminance is linearly changed gradually and a weak edge (stepped)
Could not be distinguished. If an attempt is made to extract weak edges, even regions that change linearly will be mistakenly extracted as edges. To be robust against the effects of noise,
A method has been proposed in which the luminance average is obtained in a local region and the luminance gradient is obtained from this, but the problem of erroneous detection of a linearly changing region has not been solved.

Further, in order to extract the edge region from the edge intensity image obtained by the calculation, it is necessary to perform binarization processing with an appropriate edge intensity as a threshold value. The edge intensity image obtained by the conventional edge extraction method includes a wide range of edge intensities from 0 to 255 when the luminance gradation is 255. Therefore, in the case of a fixed threshold value over the entire screen, if the threshold value is increased, weak edges will not be extracted. On the contrary, when the weak edge is extracted by lowering the threshold value, the strong edge (area having a large luminance gradient) is thickly extracted and becomes unclear. Therefore,
In the conventional edge extraction method, it is necessary to dynamically change the threshold value for each pixel, which requires an extra amount of calculation.

Furthermore, it has not been possible to easily extend to the extraction of colors and texture edges other than luminance edges.
This is a major obstacle in handling color and texture images.

In order to stably extract the object contour from the edge information, ActiveContourModel using a contour model is used.
A method called s (Snakes) has been proposed. FIG. 23
Figure 2 shows a conceptual diagram of this method. This contour extraction method uses a force (Image Force) to attract a dynamic contour model (snake model) previously defined in an image to an edge.
Then, it is a method of gradually fitting to the edge of the object. Generally, the sum of the internal energy Eint representing the smoothness of the contour model and the potential energy Eimage attracted to the edge is defined as follows.

E = Σ (Eint + Eimage) Then, the contour extraction can be realized by minimizing this energy E. In the figure, (1) shows the contour model initially arranged, (2) to (3) show the contour extraction process, and (4) shows the state where the contour of the object is extracted with the minimum energy state.

This method is effective for extracting the contour of an object composed of a background image and an edge whose brightness difference is clear [Fukui, Kuno, "Contour tracking of moving object by multi-snake", PRU92-68 (1992) / Fujimura, Yokoya, Yamamoto:
"Dynamic contour tracking of non-rigid objects using multi-scale images", Jikosho Kenho, CV-78-4, pp.25-32 (1992)]. In particular, even when the edges are discontinuously extracted, the contour is robustly extracted because the model is used.

However, basically, in order to utilize the edge information extracted by the conventional spatial brightness gradient,
Similar to the edges described above, the contour extraction of an object composed of weak edges having a small brightness difference between the object and the background has been sometimes unsuccessful. Furthermore, when extracting an object with a specific color or texture from the image,
It was difficult to expand in terms of color and texture.

[0012]

Although the method based on the spatial differential operation which is widely used at present as described above, there are the following first to third problems including the case of applying to the contour model. there were.

First, the first problem is that the spatial differential calculation is easily affected by noise. Usually, the captured image contains noise from the camera CCD element, lighting change, transmission path, etc.
Weak edges) are indistinguishable from luminance changes caused by noise. Therefore, an edge having a luminance gradient larger than noise (strong edge) can be easily extracted, but it is very difficult to accurately extract a weak edge. A method using a local luminance average has been proposed in order to make it robust against the influence of noise. However, it is basically based on the spatial luminance gradient calculation, and therefore the following second and third problems are involved. Can not be solved.

Secondly, it is impossible to distinguish a weak edge (step-like) from a region where the luminance is linearly changed slowly. Since the conventional edge extraction method outputs a value proportional to the spatial gradient of the brightness as the edge strength, when an attempt is made to extract even weak edges, even a linearly changing region is erroneously extracted.

For the same reason, in order to extract an edge by binarization from the edge image output by the conventional edge extraction method, it is necessary to dynamically change the threshold value for each pixel. It was Normally, the obtained image contains a wide range of edge intensities (luminance gradients) due to the characteristics of reflection on the object surface and the influence of illumination.

For example, FIG. 24A is the current image, FIG. 24B is the image obtained by binarizing the edge intensity image of the conventional spatial differential operation (Sobel operator) with 77, and FIG. 24C is the same edge as that of FIG. The intensity image is a binary image at 160.
In (B), the outer contour of the can is extracted to some extent, but the character contour is unclear. On the contrary, in the case of increasing the threshold value (C), the character contour is clearly extracted, but the boundary between the table and the background image and the outer contour of the can are not extracted. In this way, the extracted edge intensity (luminance gradient) is distributed over a wide range, so it is necessary to locally binarize it with an optimal threshold value.

The third problem is that the hue other than the luminance is
The point is that it cannot be easily extended to other image information such as saturation and texture information. When a plurality of objects are mixed in the image, it is difficult to accurately associate the edge with the object only by the grayscale edge information. Correspondence becomes easy by adding color and texture edge information. Generally, when dealing with color information, instead of the RGB space, human-understandable luminance (Hue), hue (Sat), and saturation (Inten
It is easier to think in the HSI space consisting of (sity). However, in the conventional edge extraction method based on the spatial differential operation,
It was difficult to apply the hue and saturation spatial gradients as they are because the spatial gradients are smaller than the luminance gradients. The same was true for texture edges.

The above-mentioned problem is that ActiveContou extracts the object contour in the image by using the above-mentioned contour model.
r Models (Snakes) method (M.Kass, A.Witkin,
D. Terzopouls, “Snakes: Active Contour Models”,
In Proc. 1st Int.Conf.on ComputerVision, pp.259-268
(1987)).

In this method, the contour is extracted based on the edge information obtained from the spatial differential calculation. Therefore, it has a common problem with the conventional edge extraction. That is, there is a problem that it is difficult to extract a contour composed of edges having a small luminance difference, which is easily affected by noise. In addition, there is a problem that expansion is not easy when extracting the contour of a specific color or texture area.

The present invention has been made in view of the above problems, and an object thereof is to provide an edge and contour extracting apparatus capable of stably extracting the shade, color, and texture edge in a given image. Especially.

[0021]

An edge extraction device according to a first aspect of the present invention includes a means for extracting image information of a mask area for a specified point in a supplied image, and a degree of separation from the extracted image information. , An edge strength calculation means for calculating the edge strength of a point specified from the obtained separation strength, and an edge extraction means for extracting an edge from the edge strength obtained by the edge strength calculation means. Composed of and.

In the edge extracting apparatus according to the second aspect, the mask area is divided into two areas 1 and 2, and the separation degree calculating means divides the image information set belonging to the area 1 and the image information set belonging to the area 2. As for the degree of separation, the image feature amount of the pixel i in the mask region is Pi, the average of the image feature amounts belonging to the region 1 is 5, the average of the image feature amounts belonging to the region 2 is 6, and the image feature amount of the entire mask region is Is defined as N, the total number of pixels in the mask region is N, and the numbers of pixels in regions 1 and 2 are n1 and n2, respectively, the separation degree η is obtained from the following equation.

[0023]

[Equation 5]

[0024]

[Equation 6]

[0025]

[Equation 7]

[0026]

[Equation 8] An edge extracting apparatus according to a third aspect is a contour extracting method using an active contour model, wherein an energy function to be minimized has a parameter of the degree of separation of the image feature amount of the second aspect inside and outside the contour model. It is characterized by

[0027]

The edge extracting apparatus of the present invention is capable of stably extracting shades, colors and texture edges by obtaining the degree of separation for the specified mask small area in the supplied image and using it as the edge strength. It is a thing.

The contour extraction apparatus of the present invention minimizes the degree of separation of the image feature amounts inside and outside the contour model under the constraint of the smoothness of the contour model to stably extract the object contour. It is the one.

[0029]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first embodiment of the present invention will be described below with reference to the drawings. First, the basic concept of the present invention will be described before specifically describing the embodiments.

In order to solve the above-mentioned problems, an edge is regarded as a boundary between "regions" and "regions" in contrast to the conventional method in which the edge is regarded as a position where brightness changes abruptly. here,
The “area” refers to an area in which the statistics such as the brightness, the hue, the saturation of each of the constituent pixels, or the brightness distribution defined in the area are constant. To extract the region boundaries, FIG.
As shown in, a rectangular small area is set in the image. The shape of this small region may be a circle or an ellipse. This rectangular small area
It is divided into two areas, and how much the sets of image feature amounts (luminance, hue, etc.) belonging to each area are separated from each other. In this case, the edge can be defined as a position with a high degree of separation.

As shown in FIG. 4, when the division position is exactly on the edge, the two attribute sets are in the most separated state. As the edges become dull, the degree of separation between the two sets decreases. In the flat area where the edge is not present, the separation is the lowest.

Here, in order to express the degree of separation between two sets, an amount called "separation degree" between regions is defined. This degree of separation is a quantity that indicates how much each class is separated when a given data set is divided into several classes in the discriminant analysis of multivariate analysis. This is the ratio of fluctuations between the two (Ishimura, Arima, “Multivariate analysis of multivariate analysis”). Therefore, the maximum is normalized to 1.0 and the minimum is 0.
It approaches 0 without limit. An edge can be defined as a position where this degree of separation is high.

Specifically, as shown in FIG. 3, consider a rectangular mask small area set near a certain point (ij) in the image. When this mask area is divided into two areas 1 and 2, the degree of separation η between the areas is defined as follows.

[0034]

[Equation 9] Here, N = the total number of pixels in the rectangular area, n1 = the number of pixels in the search area 1, n2 = the number of pixels in the search area 2, Pi is the image feature amount of the position i, and the number 10 is the image of the area 1 The average value of the feature amount, the number 11 indicates the average value of the image feature amount of the area 2, and the number 12 indicates the average value of the image feature amount in the entire rectangular area. The range of values is 0 <η ≦ 1.

[0035]

[Equation 10]

[0036]

[Equation 11]

[0037]

[Equation 12] FIG. 4 shows the luminance distribution and the value of the degree of separation for the case where the mask size is size = 8 pixels and width = 1 pixels. (A) is an ideal step edge,
(B) to (C) are dull step edges, and (D) is
The linear gradient region, (E) shows a substantially flat region.
As the ideal step edge collapses, the separation value gradually decreases from 1.0. For a completely flat region, the variance of the denominator is 0, and therefore it cannot be calculated from the above equation. Therefore, it is defined as 0 in advance.

FIG. 5 shows changes in the degree of separation when the search area is divided at the division position Psep. At Psep = 7, the degree of separation is maximum. This position coincides with the area boundary (edge). As the distance from the edge increases, the degree of separation decreases.

FIG. 6 shows a method of calculating the edge strength and the direction of a specified point (i, j) by using this degree of separation. As shown in the figure, two masks for the X and Y directions are prepared, and the degree of separation is obtained for each. The edge strength and the direction of the specified point are calculated from the obtained two degrees of separation.

As an advantage of the edge extraction based on this degree of separation, the same edge strength (1.0) is output after normalization for all ideal step edges regardless of the step edge height (luminance difference). There are points. On the other hand, the edge strength of the region that changes linearly is 0.75. Therefore, by setting the threshold value larger than 0.75, the second problem described above can be solved. Further, since the edge is globally extracted as the area boundary based on the two areas, the edge is more robust against the influence of noise as compared with the method based on the spatial gradient, and the above-mentioned second problem can be addressed. .

The third problem described above is based on the idea of separating the sets belonging to the area. Therefore, in addition to the brightness information, the image feature amount ( It can be easily extended to color, texture information, etc.).

Next, a method of extracting an edge from a plurality of image feature amounts will be described. Here, as the image feature amount, three pieces of RGB luminance information will be described as an example. Separation degree ηrg
b is calculated from the following equation.

[0043]

[Equation 13] Similarly, σg and σb can be obtained. Here, N = the total number of pixels in the search area, n1 = the number of pixels in the search area 1, n2 = the number of pixels in the search area 2, Pir is the R brightness level at position i, and Equation 14 is the area 1
, The average of the R brightness in the region 2, and the number 15 is the average of the R brightness in the region 2.
Equation 16 is the average of the R brightness of the whole, and ωr, ωg, ωb are R
, G, and B (0 to 1.0). The value range is 0 <η ≦ 1.0.

[0044]

[Equation 14]

[0045]

[Equation 15]

[0046]

[Equation 16] In addition to the above, each separation degree of RGB may be calculated independently, and the weighted average value or maximum value of the three separation degrees may be used as the edge strength. Also, from the RGB information, the brightness (H), hue (S)
, Saturation (I) and calculate the separability. The weighted average value of the three degrees of separation or the maximum value may be used as the edge strength.

When extracting texture edges, the first-order statistics (variance, third-order moment, fourth-order moment), second-order statistics (co-occurrence matrix, difference statistic, local coordinate co-occurrence matrix) are calculated from luminance information. ). Depending on the characteristics of the texture edge to be obtained, these texture amounts are combined, and the degree of separation is calculated from the following formula to obtain the edge strength.

Even when the image feature amount is increased to four or more, the degree of separation can be defined by the following equation.

[0049]

[Equation 17] Here, W is the number of image feature amounts, and Eq. 18 is the image feature amount k
, The average value, the numerical value 20 is the average of the image characteristic amounts k 1 in the region 1, and the numerical value 21 is the average of the image characteristic amounts K 2 in the region 2. The value range is 0 <η ≦ 1.0.

[0050]

[Equation 18]

[0051]

[Formula 19]

[0052]

[Equation 20]

[0053]

[Equation 21] The first embodiment will be described below with reference to FIGS. 1 and 2. FIG. 1 is a block diagram showing the configuration of an edge extracting apparatus according to the first embodiment of the present invention, and FIG. 2 is a flow chart for explaining the operation of the first embodiment.

In FIG. 1, A is an image input unit for inputting an image of an object whose edges are to be extracted. The image input unit A is an ITV camera 1 for capturing an image of an object whose edges are to be extracted.
And an analog image data imaged by the ITV camera 1.
A / D converter 2 for converting the digital data into digital data, a color look-up table 3 for color-converting the digital image data output from the A / D converter 2, and its color. An image memory 4 for storing the converted image data. The image memory 4 includes three image memories 4-
It is composed of R, 4-G, and 4-B.

The output of the image input section A is connected to the mask area storage section B which stores the image data in the specified mask area in two buffers. The mask area storage unit B has a mask position control unit 11 for designating a mask position, and two widths shown in FIG. 6 for a designated point from the image memory 4.
, An image capturing unit 12 for reading image data of a rectangular mask area of length size in the x and y directions and storing the image data in buffers 13x and 13y, respectively, and a texture calculation unit 14 for calculating a texture amount. This length size is
It is proportional to the resolution, and if it is increased, it is less susceptible to noise, but the resolution becomes rough. In practice, size is
It is preset in the range of 2 to 6 pixels. Alternatively, the size that maximizes may be selected by calculating the variance value for masks of various sizes. The width is set in the range of 1 to 5 pixels.

The image capturing section B determines the degree of separation η of the mask area.
Is connected to the edge calculation unit C that calculates The edge calculation unit C has x, y stored in the buffers 13x, 13y.
The dispersion degree σ of the image data in the direction is calculated, and when the dispersion degree σ is equal to or more than a predetermined value, the subsequent x-direction separation degree calculation unit 22x,
Dispersion threshold processing units 21x and 2 for causing the y-direction separation calculation unit 22y to calculate the x-direction separations ηx and ηy respectively.
1y, x-direction separation degree calculation unit 22x, y-direction separation degree calculation unit 22y respectively, x, y-direction separation degree ηx, η
A separability calculator 23 for calculating the separability η at a point given by y, and a scale converter for multiplying the separability η calculated by the separability calculator 23 by an appropriate scale factor, for example, 255. 24, and an image memory 25 for storing the separation degree η after the scale conversion.

The output of the edge calculation section C is connected to an edge extraction section D for extracting the edge of the object. The edge extraction unit D includes a binarization processing unit 31 that binarizes the separation degree η of a given point stored in the image memory 25 with a predetermined threshold value.

The output of the edge extraction unit D is the image output unit E.
Connected to. The image output unit E is an image memory 41 for storing the image data binarized by the binarization processing unit 31.
And a monitor 42 for displaying the image data stored in the image memory 41. The image memory 25, the binarization processing unit 31, the image memory 41, and the monitor 4
2 are connected to the image bus G, respectively.

Next, the first embodiment of the present invention constructed as above will be described with reference to the flow chart of FIG. The image input from the ITV camera 1 is A /
It is digitized by the D converter, color-converted via a color lookup table, and stored in the image memory 4.

First, the mask position is designated by the mask position controller 11 (step S1). Next, the image capturing unit 12 stores the image data of the rectangular mask area having the two widths width and length size shown in FIG. 6 for the specified point from the image memory 4 in the buffers 13x and 13y, respectively ( Step S2).

Next, the distributed threshold processing units 12x and 12y.
Calculates the variance value σ in each mask area, and calculates the degree of separation only when the variance value is larger than the specified variance threshold (steps S31, S32, S41, S42, S52,
S53).

On the other hand, the variance value σ in the mask area is calculated, and when it is smaller than the designated variance threshold value, the degree of separation “0” is given (steps S51 and S53).

The x-direction separability calculator 22x calculates the separability ηx in the X-direction from the luminance data of the regions X1 and X2 stored in the buffer 13x by using the equation 22 (step S52). Similarly, the Y-direction separation degree calculation unit 22y
Is calculated from the luminance data of the regions Y1 and Y2 stored in the buffer 13y by using the following equation 23.
y is calculated (step S54). Separation degree calculation unit 23
Calculates the degree of separation Out at a point given by the degree of separation ηx in the X direction and the degree of separation ηy in the Y direction and the edge direction φ by the following formulas (steps S7 and S8).

[0064]

[Equation 22]

[0065]

[Equation 23]

[0066]

[Equation 24] When extracting a texture edge, the texture amount calculation unit 14 obtains a texture amount, and this value is used to calculate the degree of separation.

Next, it is determined whether the edges of all pixels have been calculated (step S8). If the edges of all the pixels have not been calculated, the mask position control unit 1
1 moves the mask area position to the position of the next pixel (step S2). When the edges of all the pixels are calculated by repeating such processing, they are multiplied by an appropriate scale factor, for example, 255, and stored in the image memory 25.

Next, after the image data stored in the image memory 25 is binarized by the binarization processing unit 31, it is stored in the image memory 41 (step S9).

Then, the processing result stored in the image memory 41 is displayed on the monitor 42 via the image bus B (step S10).

As described above, according to the first embodiment, the separability η is calculated from the x-direction separability ηx and the y-direction separability ηy for all positions of the image data of the object whose edge is to be detected,
Since the edges are extracted based on the degree of separation η, the edges can be extracted stably.

In the first embodiment, the mask area is x
Although there are only two in the direction and the y direction, the number of masks may be three or more. For example, the number of masks may be increased to 6 as shown in FIG. 7 to have a total of 6 masks.
In FIG. 7, the mask corresponds to an edge with a unique slope. In the figure, (A) is a horizontal edge, and (C) is 3 from horizontal.
Corresponds to a 0 degree tilted edge.

As described above, in the case of having six masks, the mask area storage unit B stores the brightness data of the mask area from the image memory 4 in six buffers,
The edge calculation unit C calculates the degree of separation in six directions using the luminance data of the six buffers, and the degree-of-separation calculation unit 23 obtains the maximum degree of separation among these six degrees of separation and sets it as the edge strength. The edge direction may be the direction of the mask that maximizes the edge strength.

FIG. 24D shows the result of edge extraction according to this embodiment with respect to FIG. Mask size is
The threshold is 6 pixels, the variance threshold is 16, and the edge strength threshold is 0.6. It can be seen that, in FIG. 24D, both the outer contour and the character region are uniformly extracted despite the fixed threshold value as compared with FIGS. 24B and 24C.

Next, an object contour extracting apparatus according to the second embodiment of the present invention will be described with reference to FIGS. 8 to 14. The second embodiment relates to an object contour extraction apparatus that stably extracts an object contour from an image using an active contour model.

First, the contour extraction using the conventional active contour model will be described. This method is a method for converting the contour extraction problem from the object in the image into the energy minimization problem and solving it, as described above. A contour model is defined in the image, and an energy function E that minimizes the contour model in the image is defined.

E = Σ (Eint + Eimage + Eext) where Eint is the energy indicating the smoothness of the contour model, and Eimage is the minimum potential energy on the edge, that is, the force attracted to the edge. , Eext represents the energy given from the outside,
Not considered here. The state in which this energy is the minimum is the state in which the contour model fits the contour.

The contour extraction by the conventional active contour model is
It was based on the edge obtained from the spatial differentiation operation. Therefore, as described above, there is a problem in that contours, colors, and texture contours having a small difference in brightness cannot be extracted. In order to solve this problem, the energy Eimage is replaced with the energy term Esep determined from the degree of separation η of the image features inside and outside the contour model.

E = Σ (Eint + Esep) Esep = A / η where A is a positive constant. Esep is the smallest because the image feature amount is most separated on the object contour.

Various methods for solving the above minimization problem have been proposed [AAA mini TE Weymouth, RC Jain :,
“Using dynamic programming for solving variationa
l problems in vision “IEEE Trans.onPattern Analysi
s and Machine Intelligence, PAMI, vol12, no9, pp.855-8
67 (1990) / Fujimura, Yokoya, Yamamoto: “Dynamic contour tracking of non-rigid objects using multi-scale images”, Kenjo Kenkyuho, CV-78-4,
pp.25-32 (1992) / Ueda, Mase, Suenaga "Contour Tracking Based on Energy Minimization", Information Processing Research Report, CV-73, pp33-39 (1991)].
In the present embodiment, a B-spline contour model [R. Ci that approximates smoothness with a B-spline curve without using iterative calculation]
polla, A. Blake: “The dynamic analysisof apparentc
ontours ”In Proc. 3rd Int. Conf.on Computer Visio
n, pp616-623 (1990)] is used. Details of the B-spline contour model are described in the already filed patent [13A92X092-1].

FIG. 12 shows an example in which the degree of separation is applied to the contour model. In the figure, (A) is a contour model that has been initially arranged, (B) is a state in which one-step processing has progressed, (C)
Shows a state in which some steps have been further advanced, and (D) shows a state in which the energy function E is minimized and the contour is extracted. The rectangular area provided for each sample point is a search area for searching a division position where the degree of separation is maximum. Small circles indicate sample points. The waveform inside the rectangle shows the distribution of the degree of separation. The vector is the amount of movement of each sample point in each step, and the large "●" indicates the control point of the B-spline curve.

In the human I / F, extracting the face contour is an important elemental technique. Here, in order to make the present embodiment more specific, the object will be described in detail with reference to the drawings as a human face.

FIG. 8 is a block diagram showing the structure of the object contour extracting apparatus. In FIG. 8, H is an image input unit that captures an image of an object. The image input section H includes an ITV camera 51 for picking up an image of an object and an A / D converter 52 for converting analog image data picked up by the ITV camera 51 into digital data and outputting the digital data to an image bus G1. And an image memory 53 for storing digital image data output from the A / D converter 52 and connected to the image bus G1.

The image input unit H is connected to an initial placement unit I that extracts face candidate regions from the image and initially places the contour model on an ellipse near the candidate regions. In the initial arrangement unit I, the difference circuit 54 and the color lookup table (CL
UT) 55 is connected.

The initial arranging section B has a face candidate area extracting section I1.
And the contour model placement unit I2. The face candidate area extraction unit I1 is a difference circuit 54 that performs a difference operation between the image data (also referred to as a background image) p1 that does not include a face stored in the image memory 53 and the image data p2 that is currently input. And the binarization circuit 56 for thresholding the output of the difference circuit 54 and outputting the image data p3 to the logical sum circuit 57, and the image data p2 input at the present time to color conversion. Image data p4
As a result, the color look-up table 55, which is output to the OR circuit 57, is ORed with the image data p3 and the image data p4, and is output to the circumscribing circuit 57.
And a circumscribed rectangle calculation circuit 58 that calculates the circumscribed rectangle coordinates of the face candidate area from the image data output from the OR circuit 57 and outputs the coordinates to the inscribed ellipse calculation circuit 59, and inscribes the circumscribed rectangle. An inscribed ellipse calculation circuit 59 for calculating an inscribed ellipse.

The contour model placement section I2 places the contour model on the inscribed ellipse obtained as shown in FIG. A mouse 60, which a person interactively inputs while viewing an image, is connected to the contour model placement unit I2.

The initial placement section I is connected to a contour extracting section J for extracting contours. The contour extraction unit J calculates a contour model curve from the given control points, and the contour model calculation unit J1
And the separation degree around the contour model itself, and the contour model deforming section J2 that deforms the contour model in the direction in which the separation degree between the inside and outside of the contour model increases, and all sample points before the temporary point. Convergence determination unit J3 that determines whether or not the contour extraction processing has converged based on the temporal change in the average resolution and the current average resolution.

The contour model calculation unit J1 calculates the contour model curve from the control points given from the contour model arrangement unit I2. As shown in FIG. 23, the contour model curve is n
It is represented by a B-spline curve determined by the individual control points Pi (i = 0, n−1). There are m sample points on the B-spline curve.
Si (i = 0, m-1) is set.

The contour model calculation unit J1 calculates the contour model (B-spline curve) determined from the given control points from the following equation (3).

[0089]

[Equation 25] Here, m is the total number of control points, Qix is the Y coordinate of the control point i, Qiy is the Y coordinate of the control point i, and Ni (s) is the B spline basic function.

As shown in FIG. 12, the contour model transforming section J2 provides a rectangular search area around each sample point S, and determines the search area for storing the image data of the search area in a buffer (not shown). The division 61, the division calculation unit 62 that calculates the degree of separation when the image data stored in the buffer is divided at the boundary B, and the division boundary Bmax that maximizes the separation while changing the division position B. And a maximum separability calculation unit 63 that efficiently searches for
Each sample point S 1 is moved toward the obtained division position Bmax to deform the contour model, and a model deforming unit 64.

By the way, the contour model (B-spline curve) determined by the contour model calculation unit J1 is output to the average separation degree calculation unit 65, and the average separation degree of all sample points is calculated in this average separation degree calculation unit 65. To be done.

The convergence determination unit 66 determines whether or not the contour extraction processing has converged, based on the time change of the average separation degree of all the sample points before the temporary point and the current average separation degree.

The contour extraction unit J is connected to a face area cutout unit K which extracts a face area from the converged contour model shape.
The face area cutout unit K is connected to the image bus G1 described above, and the face area cutout unit 71 to which the convergence signal from the convergence determination unit 66 is input, and the face contour extracted when the contour extraction processing converges A buffer 72 for storing the coordinate information of the above and the color gamut information inside the contour curve, and an image memory 73 for storing the image of the face area output from the face area cutout unit 71.

The face area cutout section K is connected to the image output section L. The image output unit L includes a display changeover switch unit 74 capable of changing over the display to one of three modes and a monitor 75 for displaying an image via the display changeover switch unit 74. That is, the input side of the display changeover switch unit 74 is connected to the face contour coordinate information, the buffer 72 for storing the color gamut information inside the contour curve, the image memory 73 for storing the face region image, and the image bus G1. In the first display mode, only the contour curve is displayed, in the second display mode, only the image of the face area is displayed, and in the third display mode, the extracted contour is displayed overlaid on the original image.

Next, the operation of the second embodiment of the present invention constructed as described above will be described with reference to the flowchart of FIG. The face candidate area extraction unit I1 extracts an area in the image where the brightness is changed. First, the difference circuit 54 performs a difference operation between the background image data p1 stored in advance in the image memory 53 and the image data p2 input at the present time. The output of this difference circuit is thresholded by the binarization circuit 56 and sent to the OR circuit as image data p3 (step S1).

The face candidate area extraction unit I1 uses the same image data.
Extract the skin color area from p2. The image data p2 is color-converted by a color lookup table (CLUT). CL
The contents of UT can be rewritten arbitrarily. When extracting a human skin color region from an image, it is sufficient to fill the CLUT in advance with a region of the table corresponding to the skin color region with appropriate data. Other than that, it is set to 0. (Step S2).

The logical sum circuit 57 logically sums the image data p3 and the image data p4 and sends the logical sum to the circumscribing rectangle calculation circuit 58. (Step S4). The circumscribed rectangle calculation circuit 58 calculates the circumscribed rectangle coordinates of the face candidate area from the sent image, and sends the coordinates to the inscribed ellipse calculation circuit 59. Inscribed ellipse calculation circuit 5
9 calculates an inscribed ellipse inscribed in the circumscribed rectangle (step S6).

The contour model placement section I1 places the contour model on the inscribed ellipse obtained as shown in FIG. 13 (step S7). Control points are placed on the inscribed ellipse as shown in the figure. The control points are arranged such that the angles are evenly spaced.

For the initial placement of the contour model, the method of the already filed patent [13A92X092-1] may be used. In this case, the contour model is composed of a plurality of short segment curve models instead of one curve model. Each segment curve model is arranged on each side of the extracted circumscribed rectangle.

Further, this initial placement processing may be performed interactively by a human while watching an image by using input means such as a mouse and a keyboard.

The contour model calculation section J1 calculates a contour model curve from the given control points. That is, the contour model calculation unit J1 calculates the contour model (B-spline curve) determined from the given control points, using the above-described formula (3). (Step S8).

The contour model transformation section J2 examines the degree of separation around the contour model itself, and transforms the contour model in a direction in which the degree of separation between the inside and outside of the contour model increases.

As shown in FIG. 12, the search area determining section 61 provides a rectangular search area around each sample point S 1 (step S9). As shown in the figure, the search area is evenly arranged inside and outside in the normal direction of the contour model curve. The search area determination unit 61 stores the image data of this area in a buffer (not shown).

The separability calculation section 62 calculates the separability when the image data stored in the buffer is divided at the boundary B 1. The maximum separability calculation unit 63 changes the division position B while determining the division boundary Bmax at which the separability is maximum.
Efficiently search by Newton's method (step S1)
0).

When the division position Bmax is found, the search area determining unit 61 similarly obtains the division position for the adjacent sample point s + 1 (returns to step S9). When the division positions for all the sample points are obtained, the model transformation is performed. The unit 64 assigns each sample point S to the obtained division position Bma.
The contour model is deformed by moving it toward x (step S12). The amount of movement is an amount obtained by weighting the distance from the current sample point to the division position.

The processing of the contour extracting section J can be realized by software or dedicated hardware.

The convergence determination unit 66 determines whether or not the contour extraction processing has converged from the time change of the average separation degree of all the sample points before the temporary point and the current average separation degree (step S1).
3, 14). When the amount of change in the average degree of separation becomes smaller than the reference value, it is determined that the processing has converged, and the contour extraction processing ends. If the processing has not converged, the process returns to step S9 again.

When the contour extraction processing converges, the coordinate information of the extracted face contour and the color gamut information inside the contour curve are
It is stored in the buffer 72.

The face area cutout unit 71 extracts a face area from the converged contour model shape (step S15).

The image output section L includes a display changeover switch section 7
According to the operation of No. 4, only the contour curve is displayed as the first display mode, only the image of the face area is displayed as the second display mode, and the extracted contour is superimposed on the original image as the third display mode. To display.

Next, the contour extraction unit J will be described in detail with reference to FIGS. Basically, as shown in the figure, it is composed of two-stage processing.

When extracting the human face contour, the color information is important information. Here, it is assumed that the rough color information of the face (skin color) is known in advance. While looking at the image,
Using a mouse or the like, 3 to 5 points in the face area are sampled to create a rough color map of skin color.

In the first stage, a rough outline is extracted using the color information of the object given in advance. Using CLUT, color binarization is performed, rough face (skin color) areas are extracted, and contours are extracted. However, at this stage, only rough outlines can be extracted because the CLUT is incomplete.

In the second stage, the boundary having a high degree of separation is searched again only in the periphery of the contour obtained in the preceding process, and an accurate contour is extracted. In this case, without using CLUT,
The degree of separation is calculated from the three brightness data of R, G, and B from the above-mentioned formula (2).

In the first stage, 1. Some control points are automatically generated on the circumscribed ellipse extracted by the initial placement unit. For example, for M control points,
Place at 360 / M degree intervals. Alternatively, a human may specify a control point with a mouse while looking at the image (step S
5). 2. From the given control points Qix, Qiy (i = 0, M-1), the coordinates X (s) of the sample point on the B-spline curve,
Y (s) is calculated from the above equation (3) (step S
6). Here, s is a parameter defined on the curve. 3. A rectangular search area extending in the normal direction of the B-spline curve is set around each sample point s (step S
7). 4. Within the set search area, the division position Bmax that maximizes the degree of separation is searched for using the Newton method (step S8). 5. The distance from each sample point s to the division position Bmax is multiplied by a weight (0.1 to 1.0) to obtain a movement amount (dsx, dsy). This amount of movement is calculated for all sample points. 6. Each sample point moves by (dsx, dsy), and the deformed contour model is defined as P (s). Find a new B-spline curve control point that best approximates P (s). A new control point is obtained by minimizing the following function Eint (step S9).

[0116]

[Equation 26] When the least squares method is applied to the above equation, a new control point is obtained by solving the following simultaneous linear equations (step S
11).

[0117]

[Equation 27] 7. The contour model is calculated and updated for the new control point (step S12). 8. A convergence determination is made based on the amount of change in the degree of separation in the time direction (step S13). 9. If not converged, the process returns to step S7. 10. In the case of convergence, the first stage processing is ended.

The second stage is also similar to the process of the first stage, but the degree of separation is calculated using equation (2) instead of equation (1). After convergence, the face area is cut out and displayed.

FIG. 14 shows the experimental result of this embodiment. It is a result of face area extraction for three images extracted from continuous images of a moving face. The area white line indicates the contour model.

As described above, the contour of an object can be stably extracted by utilizing the degree of separation.

Next, a contour tracking device for tracking a part of the contour of a moving object according to the third embodiment of the present invention will be described with reference to FIGS. FIG. 15 is a block diagram showing the overall configuration. In FIG. 15, M is an image input unit and N is a contour extraction unit, which have the same configurations as the image input unit H and the contour extraction unit J of the second embodiment described above, respectively. The output of the contour extraction unit N is output to the correlation calculation unit O that searches for a region on the extracted contour that has a high correlation with the contour one time before. The output of the correlation calculation unit O is output to the contour slide unit P. The contour slide unit P slides on the extracted contour and searches for an area having a high correlation with the contour before the temporary point.

The correlation calculation section O slides the mask within the search area set by the search area setting section 81 for setting the search area, the mask area setting section 82 for setting the mask area, and the search area setting section 81. , Template before the temporary point
A correlation calculation unit 83 for calculating the correlation value R (s) between the current mask region and the current mask region, and a maximum correlation search unit 84 for searching the region having the maximum correlation value. Correlation value R
(S) is calculated by SSDA (sequential residual test method) from the following formula.

[0123]

[Equation 28] Here, M is the number of mask data, T (R) is a template set on the side of the object region before the temporary point, s is a parameter along the contour, and I (i) is luminance information at point i. is there.

The slide section P slides the contour model from A to B.

An image output section Q is connected to the slide section P.

Next, the operation of the third embodiment of the present invention constructed as described above will be described with reference to the flow chart of FIG. Usually, when tracing a part of the contour, there is a problem (window problem) that the movement in the vertical direction of the contour can be detected, but the movement in the horizontal direction of the contour cannot be detected.

In the third embodiment, in order to solve this problem, after the contour extraction, a region having a high correlation with the luminance distribution of the contour region which is slid on the contour and traced before the temporary point is searched for, Slide the contour model into that position. Figure 1
7 shows how the slide process is performed. In the figure, (A)
Is the position of the contour model at time t, (B) is the state of tracking the contour by following the moving object, and (C) is the maximum correlation position searched while the mask is moving on the contour. It shows the situation.

The contour extraction unit N transforms from the position of the contour model to the current contour position before the temporary point (step S
2).

As shown in FIG. 17, the slide portion P slides the contour model from the point A to the point B on the contour. First, a new straight line portion Lnew is added to the contour model in the tangential direction from both ends E1 and E2 of the contour curve (step S).
3). The added straight line portion Lnew is again drawn to the contour (step S4). A search area is set on the object area side.

The correlation calculator 83 calculates the correlation between the mask area before the temporary point and the current mask area while sliding the mask in the search area (steps S5 and S6). It is assumed that the position B having the highest correlation value is the same contour area as before the temporary point.

The correlation value R (s) is calculated by SSDA (residual sequential test method).

The slide unit slides the contour model from A to B and removes the added portion Lnew. Moreover, it returns to step S2.

Then, the result is output to the monitor Q.

As described above, according to the third embodiment, when tracing a part of the contour, the movement in the horizontal direction can be detected in the contour.

Next, a change detecting device according to the fourth embodiment of the present invention will be described with reference to FIGS.
The fourth embodiment relates to a change detecting device that stably extracts even a slight change in an image.

In order to detect a change in an image, an image difference calculation is performed between consecutive images or with a background image captured in advance, and an area having a change amount larger than a threshold value is detected as a change area. . Generally, in order to be robust against noise, the amount of difference in luminance averaged within a small area is used as the amount of change. However, when the difference in brightness between the background image and the object to be extracted is small, the threshold value is lowered to detect the change, and noise is also picked up. In order to solve this, the degree of separation is defined as the degree of separation between the luminance set in the small area at a certain time t and the luminance set at the next time t + dt.

FIG. 18 is a block diagram showing the structure of the change detecting device. In FIG. 18, R is an image input unit, S is a separation degree calculation unit, T is a threshold processing unit, and U is an image output unit. The image input unit R is an ITV camera 1 that images an object.
And an analog image data imaged by the ITV camera 1.
A / D converter 2 for converting the digital data into digital data, a color look-up table 3 for color-converting the digital image data output from the A / D converter 2, and its color. An image memory 4 for storing the converted image data.

The separability calculation section S reads the image data of the specified small area (i, j) from the image memory 4 and stores the image data in the buffer 91a. The image capturing section 92 stores the image data stored in the buffer 91a. And a separation degree calculation unit 93 that calculates the separation degree from the data.

The threshold processing unit T binarizes the degree of separation with a predetermined threshold value and outputs it to the scale conversion unit 94 of the image output unit U. The scale conversion section 94 multiplies the output of the threshold value processing section T by an appropriate scale, for example, 255 times, and outputs the image memory 9
Output to 5. The contents of the image memory 95 are displayed on the monitor 96.
Is output to.

Next, the operation of the fourth embodiment of the present invention constructed as above will be described. The image input from the ITV camera 1 at time t 1 before the temporary point is referred to as image 1 and the image 2 currently input. The input image 1 is digitized by the A / D converter 2 and stored in the image memory 4. Here, the image input from the ITV camera 1 at time t 1 before the temporary point is referred to as image 1 and the image 2 currently input.

The input image is divided into small areas.
FIG. 19 shows how the division is performed. In the mask area data capturing unit 92, the small area (i,
The image feature amount of j) is stored in the buffer 91a. FIG. 20 shows the structure of the buffer 91a. The brightness data of image 1 is T
The luminance data of 1 and the image 2 are stored in T2, respectively.

A dispersion threshold value processing unit (not shown) interposed between the buffer 91a and the separation degree calculation unit 93 compares the data distribution value of the buffer 91a with a reference value, and if it is smaller than the reference value, the separation degree is determined. Set to 0. The standard value is 1
Set to ~ 20. The separation degree calculation unit 93 uses the buffer 91.
The degree of separation in the time direction when the luminance data of a is separated into T1 and T2 is calculated and sent to the threshold circuit T. The threshold circuit T binarizes with a specified threshold value, multiplies it by 255 in the scale conversion unit 94, and stores it in the image memory 95 as a change amount.

The mask region is moved over the entire screen until the change amount for all the small regions is obtained, and the change amount is obtained.

The separability calculation section 93 can be replaced by a general-purpose processor. Alternatively, a plurality of processors may be substituted for parallel processing in several areas. Further, software processing may be performed using a general-purpose computer such as a workstation.

The image output unit U displays the data in the image memory 95 on the monitor 96 via the image bus B2.

As described above, according to the fourth embodiment, it is possible to stably extract even a slight change in an image by utilizing the degree of separation.

The processing of the mask area storage unit B, the edge calculation unit C, and the edge extraction unit of the first embodiment shown in FIG. 1 may be realized only by dedicated hardware or software.

[0148]

As described above in detail, according to the present invention, a great effect is expected in practical use such as realizing a device for stably extracting edges and contours from an image input by using a degree of separation. It is possible to provide an edge and contour extraction device that can do this.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of an edge extraction device according to a first embodiment of the present invention.

FIG. 2 is a flowchart for explaining the operation of the first embodiment.

FIG. 3 is a diagram relating to the first embodiment and defining an edge as a region boundary.

FIG. 4 is a diagram showing a luminance distribution and a separation degree value for the luminance distribution, according to the first embodiment.

FIG. 5 is a diagram related to a first embodiment and showing a division position and a separation degree value for the division position.

FIG. 6 is a diagram showing a method of calculating edge strength from a degree of separation according to the first embodiment.

FIG. 7 is a diagram showing another mask shape for edge extraction of the apparatus according to the first embodiment of the present invention.

FIG. 8 is a block diagram showing the configuration of an object contour extraction apparatus according to a second embodiment of the present invention.

FIG. 9 is a flowchart for explaining the operation of the second embodiment.

FIG. 10 is a part of a flowchart showing in detail the function of a contour extraction unit J according to the second embodiment.

FIG. 11 is a remaining part of the flowchart showing in detail the function of the contour extracting unit J according to the second embodiment.

FIG. 12 is a diagram relating to the second embodiment and showing the concept of an active contour model using the degree of separation.

FIG. 13 is a diagram for explaining an initial placement of a contour model according to the second embodiment.

FIG. 14 is a diagram illustrating a result of face area extraction according to the second embodiment.

FIG. 15 is a block diagram showing the configuration of a contour tracking device for tracking a part of the contour of a moving object according to the third embodiment of the present invention.

FIG. 16 is a flowchart for explaining the operation of the third embodiment.

FIG. 17 is a diagram for explaining a slide search according to the third embodiment.

FIG. 18 is a block diagram showing the configuration of a change detection device according to a fourth embodiment of the present invention.

FIG. 19 is a diagram illustrating division of an image into small regions according to the fourth embodiment.

FIG. 20 is a diagram showing a configuration of a buffer according to the fourth embodiment.

FIG. 21 is a diagram for explaining the definition of a conventional edge.

FIG. 22 is a diagram for explaining a case where a conventional area in which the brightness changes linearly is mistakenly extracted as an edge.

FIG. 23 is a diagram showing the concept of object contour extraction using an active contour model.

FIG. 24 is a diagram illustrating binarization processing for an edge strength image including a wide range of edge strengths.

[Explanation of symbols]

A ... Image input section, B ... Mask area storage section, C ... Edge calculation section, D ... Edge extraction section, E ... Image output section, 1 ... ITV
Camera, 3 ... Color lookup table (CLUT), 4
Image memory, 2 ... A / D converter, 11 ... Mask position control unit, 12 ... Image capturing unit, 13x, 13y ... Buffer, 14 ... Texture amount calculation unit, 21x, 21y ... Dispersion threshold processing unit, 22x , 22y ... Directional separation degree calculation unit,
23 ... Separation degree calculation unit, 24 ... Scale conversion unit, 31 ... 2
Quantization circuit, 42 ... Monitor.

Claims (3)

[Claims] 1. A means for extracting image information of a mask area for a specified point in a supplied image, and a separation degree calculating means for calculating a degree of separation from the extracted image information,
It is characterized by further comprising edge strength calculation means for calculating the edge strength of a specified point from the obtained degree of separation, and edge extraction means for extracting an edge from the edge strength obtained by the edge strength calculation means. Edge extraction device. 2. The mask area is divided into at least two areas 1 and 2, and the separation degree calculation means determines the separation degrees of the image information set belonging to the area 1 and the image information set belonging to the area 2 in the mask area. The image feature amount of pixel i in
, The average of the image feature amounts belonging to the region 2 is the number 2, the average of the image feature amounts of the entire mask region is the number 3, the total number of pixels in the mask region is N, the region 1, The edge extraction device according to claim 1, wherein the separation degree η is obtained from the following equation, where the number of pixels of 2 is n1 and n2, respectively. [Equation 1] [Equation 2] [Equation 3] [Equation 4] 3. In the contour extraction using the active contour model, the energy function to be minimized uses the degree of separation of the image feature amount inside and outside the contour model as a parameter. Item 2. The contour extraction device according to item 2.