JP3638845B2 - Image processing apparatus and method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus and method, and a recording medium, and more particularly, to a process for extracting an outline of an object from an original image using a spatial differential value.
[0002]
[Prior art]
Conventionally, various methods for extracting an object, specifically, an outline (edge) of an object from an image have been proposed. For example, a gradient is obtained from the spatial differentiation of the luminance of the original image, and a method for extracting edges by suppressing non-maximum values (removing points other than those that are locally maximal), and a template that assumes an object shade pattern are prepared in advance. A method of extracting an edge by performing a correlation operation with the original image (template matching), a method of extracting a point at which the second-order differential value of the spatial differentiation of the luminance of the original image is zero, and the like.
[0003]
[Problems to be solved by the invention]
However, in the template matching method, it is premised that the shade pattern of the object to be extracted in advance is known, and the edge of the object other than the template cannot be extracted, or the edge extraction accuracy is greatly influenced by the template accuracy. There's a problem.
[0004]
In addition, in the method of extracting edges from the spatial differentiation of the original image, edges with sharply changing brightness levels, that is, so-called step edges, can be extracted with high precision, but the boundaries of shadow areas where the brightness level changes gently are extracted as edges. There was a problem that could not be done. In the case of an object, particularly an object composed of a free-form surface, there is almost always a shadow area as well as a step edge. If the boundary of this shadow area cannot be extracted, the overall shape of the object will not be clear, and the object It is also difficult to specify the type.
[0005]
The present invention has been made in view of the above-described problems of the prior art, and an object of the present invention is to provide an apparatus, a method, and a recording medium that can reliably extract an edge of an object, particularly a boundary of a shadow area. is there.
[0007]
[Means for Solving the Problems]
In order to achieve the above object, the present invention is an image processing apparatus for extracting an object in an image, wherein the second-order spatial differential value is substantially equal to a differentiation means for second-order spatial differentiation of the luminance of the original image. Integrating the boundary between the step edge and the shadow area, the shadow area extraction means for extracting the shadow area of the object by grouping zero areas, the edge extraction means for extracting the step edge of the object in the original image have a an integrated unit that, the differentiating means is assumed by the second order spatial derivative by further spatial differentiation only area spatial differential value of the luminance of the original image is not 0 as the processing target, and, When the step edge of the object extracted by the edge extraction means does not constitute a closed curve and is missing, a processing window is set in the missing portion, and the processing window is set in the processing window. And characterized in that the floor space differential.
[0008]
The present invention is also an image processing apparatus for extracting an object in an image, and groups differentiation means for differentiating the luminance of an original image by second-order space, and regions where the second-order spatial differential value is substantially zero. A shadow area extracting means for extracting the shadow area of the object, an edge extracting means for extracting the step edge of the object in the original image, and an integrating means for integrating the boundary between the step edge and the shadow area. The differential means performs the second-order spatial differentiation by further spatially differentiating only a region where the spatial differential value of the luminance of the original image is not 0, and the edge extraction means A processing window is set on an extension line of the shadow region boundary of the object extracted by the region extraction means, and the step edge is extracted in the processing window .
[0013]
The present invention is also an image processing method for extracting an object in an image, wherein a differentiation step for second-order spatial differentiation of the luminance of the original image and a region where the second-order spatial differential value is substantially zero are grouped. A shadow region extraction step for extracting a shadow region of the object, an edge extraction step for extracting a step edge of the object in the original image, and an integration step of integrating the boundary between the step edge and the shadow region. In the differentiation step, the second-order spatial differentiation is performed by further spatially differentiating only a region where the spatial differential value of the luminance of the original image is not 0, and extracted in the edge extraction step. When the step edge of the selected object does not constitute a closed curve and is missing, a processing window is set in the missing portion, and the object falls within the processing window. Characterized by differentiating the second floor space.
[0014]
The present invention is also an image processing method for extracting an object in an image, wherein a differentiation step for second-order spatial differentiation of the luminance of the original image and a region where the second-order spatial differential value is substantially zero are grouped. A shadow region extraction step for extracting a shadow region of the object, an edge extraction step for extracting a step edge of the object in the original image, and an integration step of integrating the boundary between the step edge and the shadow region. In the differentiating step, the second-order spatial differentiation is performed by further spatially differentiating only a region where the spatial differential value of the luminance of the original image is not 0, and in the edge extracting step, the shadowing is performed. A processing window is set on an extension line of the shadow region boundary of the object extracted in the region extraction step, and the step edge is set in the processing window. Characterized in that it out.
[0019]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0020]
FIG. 1 is a block diagram showing the overall configuration of the image processing apparatus according to this embodiment. The image sensor 10 acquires an image and can be constituted by, for example, a CCD camera. Image data acquired by the image sensor 10 is digitized by the A / D converter 12 and stored in the frame memory 14. The digital image data stored in the frame memory 14, that is, the original image data, is stored in the main memory 16, and is appropriately read by the CPU 18 to perform the processing described later, thereby extracting step edges and shadow areas. The extracted step edge data and shadow area data are stored in the main memory 16, supplied to the display control unit 20, and displayed on a display such as a CRT or a liquid crystal display. The A / D 12, the frame memory 14, the main memory 16, the CPU 18, and the display control unit 20 can be configured by a computer.
[0021]
FIG. 2 shows an overall process flowchart of the CPU 18 in FIG. The CPU 18 reads the original image data stored in the main memory 16 and first calculates the gradient (S101). Next, threshold processing is performed to obtain edge point candidates, and only the points that are locally maximum are left by non-maximum value suppression processing. Further, an edge having a width of one pixel is extracted by thinning processing (S102). In addition, it is also suitable to perform the process which calculates | requires a boundary by extending the calculated | required edge point as needed.
[0022]
FIG. 3 schematically shows gradients calculated by spatial differentiation in the step edge portion (a) where the luminance changes sharply and the shadow portion (b) where the luminance changes gently. Since the peak value appears clearly at the step edge part, the peak value can be extracted by threshold processing. However, since the peak value does not appear in the shadow area, the boundary is extracted using the threshold value. Is difficult.
[0023]
Therefore, after extracting the step edge in S101 and S102, the CPU 18 determines whether or not the boundary of the shadow area is extracted from the original image (S103). This determination is based on the fact that, depending on the object, the entire edge can be extracted with only the step edge, and the necessity of extracting the boundary of the shadow region may be low. Specifically, when it is considered that the extracted step edge is closed and no missing portion exists, it is considered that the shadow area extraction processing is unnecessary.
[0024]
When it is determined that shadow area extraction is necessary, the CPU 18 uses the gradient data calculated in S101 to extract pixels having substantially the same gradient direction, and groups these pixels (S104). Since the shaded area is a change in luminance that can be formed in a shadow portion when light is irradiated on the object from one direction, the gradient is in a certain direction. In other words, a shadow region can be extracted by extracting a pixel group having a constant gradient. Specifically, pixel groups in which the difference in gradient direction between adjacent pixels is within a predetermined value (for example, ± 5 degrees) can be grouped with pixel groups having substantially the same gradient direction. For grouping, for example, the same flag may be set for pixel groups having substantially the same gradient direction.
[0025]
After extracting the shadow area, the step edge extracted in S101 and S102 and the shadow area extracted in S103 are integrated to extract a complete whole edge of the object (S105). As an integration method, there are simple addition of the step edge and the shadow area edge, weight addition of the step edge and the shadow area edge, and the like. In the case of performing weighted addition, it is preferable to give a higher weight to the higher accuracy in consideration of the calculation accuracy of the step edge and the calculation accuracy of the shadow region in a portion where the step edge and the shadow region edge overlap each other. Note that the edge of the shadow area can be extracted by selecting, for example, a boundary pixel of a pixel group with the same flag. The entire edge of the object obtained by integrating the step edge and the shadow area edge is supplied to the display control unit 20 and displayed on the display. It is also preferable to specify the type of an object from the entire edges obtained by integration and display the type of the object on a display. For this purpose, a database for storing the relationship between the entire edge and the type of object may be provided in advance.
[0026]
FIG. 4 schematically shows the processing described above. (A) is an original image, and it is assumed that a solid object 100 and a planar object 102 exist. (B) is the figure which calculated the gradient by carrying out the space differentiation of the original image, extracted the step edge by threshold processing. For the planar object 102, the step edge 102a is closed, and the entire shape of the object can be obtained by extracting the step edge 102a. However, in the case of the three-dimensional object 100, only a part is extracted as the step edge 100a. The lower right portion in the figure is a shaded area in which the luminance changes gently, so that the edge is lost. Therefore, in order to extract the entire edge of the three-dimensional object 100, the shadow area of the three-dimensional object 100 is extracted. (C) schematically shows the gradient direction of the original image with arrows. A gradient having substantially the same direction, that is, a pixel group in which the difference in gradient direction is equal to or less than a predetermined value is indicated by reference numeral 103 in the figure. It is shown.
[0027]
Note that when determining whether or not the gradient directions are substantially the same, it is also preferable to make a determination only for an area that is considered to be a shadow area, instead of making a determination for all the pixels of the original image. Specifically, a part where the extracted step edge of the object is missing is regarded as a shadow area, a processing window is set near the missing part, and pixels existing in the window have substantially the same gradient direction. It is determined whether or not. In (b), the window 104 shows the processing window set in the part where the edge is missing. By setting the window, it is possible to improve the processing efficiency and reduce the time required for grouping.
[0028]
(D) is a group of pixels having substantially the same gradient direction (no window is set), and other portions are deleted, and corresponds to a shaded area of the three-dimensional object 100. (E) is obtained by extracting the boundary of the shadow area extracted in (d). In the figure, almost the entire contour of the three-dimensional object 100 is extracted, but depending on the object, only a part of the contour may be extracted. Since the shaded area is formed according to the degree of unevenness or curved surface of the object surface, in the case of an object with a large degree of unevenness, the boundary of the shaded area occupies a large specific gravity in the entire contour.
[0029]
(F) is obtained by integrating the boundary of the shadow area extracted in (e) and the step edge extracted in (b), and a complete contour is obtained not only for the planar object but also for the three-dimensional object 100. It has been. The integration is performed by weighted addition of the step edge of (b) and the shadow area boundary of (e). Specifically, the step edge is determined at the portion where the step edge of (b) and the boundary of (e) overlap. (Step edge weight is set to 1 and shadow area boundary weight is set to 0), and a shadow area boundary is selected in a portion where a step edge is missing. The reason why the step edge is prioritized at the portion where the step edge and the shadow area boundary overlap is because of an error in grouping when extracting the shadow area.
[0030]
As described above, in this embodiment, the step edge is extracted, and when the step edge is missing, the shadow area is extracted under the condition that the gradient directions are substantially the same, and the step edge and the shadow area boundary are extracted. By integrating, it is also possible to extract objects with smooth curved surfaces such as human hands and fingers in scenes consisting of complicated backgrounds that have been difficult to extract, from backgrounds of similar colors. Enabled,
The object recognition rate can be improved.
[0031]
In this embodiment, in order to extract a shadow area, an area having substantially the same gradient direction (spatial differentiation) is extracted, but extraction is performed using the second-order derivative (secondary differentiation) of the luminance of the original image. You can also. It is equivalent that the gradient directions are substantially the same and the second-order differential value is substantially zero.
[0032]
FIG. 5 shows a processing flowchart in this case. First, first-order differentiation (first-order differentiation) or spatial differentiation of the original image is calculated (S201). After calculating the spatial differentiation, a step edge can be extracted by performing threshold processing, non-maximum value suppression processing, and further thinning processing.
[0033]
Next, instead of extracting a pixel group having substantially the same gradient direction, the spatial differentiation is further differentiated to calculate a second-order differentiation (S202). However, the second-order differentiation is limited to the first-order differentiation, that is, the region where the value of the spatial differentiation is not zero. This is because even in a region that is not originally a shaded region, a region having a spatial differential of 0 has a second-order differential value of 0, and the second-order differential value alone cannot identify both. After calculating the second derivative, pixels whose second derivative value is substantially 0, specifically, the absolute value of the second derivative is smaller than a predetermined value ε near 0, and the same flag is set to these pixels. And grouping (S203). The grouped pixel group represents a shaded area, and the entire contour of the object can be extracted by extracting the boundary and integrating it with the step edge (S204).
[0034]
In the present embodiment, step edges are first extracted, and when the step edges are missing and not closed, a shadow area is extracted and integrated, but for example, as shown in FIG. In addition, when the shadow region occupies most of the contour of the object, it is possible to first extract the shadow region and extract the step edge region based on the result.
[0035]
FIG. 6 shows a schematic processing flowchart in this case. First, a shadow area is extracted from the original image (S301). This processing can be extracted by differentiating the original image once (spatial differentiation) and grouping pixels having substantially the same gradient direction as in the above-described processing, or for pixels whose first-order differentiation value is not 0. Extraction can be performed by grouping pixels whose second-order differential values are substantially zero. After extracting the shadow region, a processing window is set on the extension line of the shadow region boundary, and the step edge is extracted by thresholding the first-order differential value in the processing window (S302) . When extracting scan Teppuejji, it may of course be subjected to non-maximum value suppression processing and thinning processing.
[0036]
In this embodiment, since the first-order differential value is used again after the second-order differential value is used, when calculating the second-order differential value after calculating the first-order differential value, the calculated 2 It is preferable to store the first order differential value in a different area of the memory from the first order differential value. It is also preferable to set the processing window sufficiently large in consideration of errors due to grouping when setting the processing window for the missing part of the shadow area boundary.
[0037]
In addition, when extracting the shadow area after extracting the step edge and integrating both, it is also possible to feed back the extraction result of the shadow area to the step edge extraction and correct the step edge extraction for higher accuracy. It is.
[0038]
FIG. 7 shows a processing flowchart in this case. First, processing similar to S101 and S102 is performed to extract step edges from the original image (S401). Next, it is determined whether or not the step edge has been sufficiently extracted, that is, whether or not the boundary of the shadow area needs to be extracted as in S103 (S402), and if the shadow area needs to be extracted, Similar to S104, pixels having substantially the same gradient direction are grouped to extract a shadow region, and the boundary is extracted (S403). Of course, the shadow area may be extracted using the second-order differential value.
[0039]
After extracting the shadow area and its boundary, the identity of the step edge extracted in S401 and the shadow area boundary extracted in S403 is determined. The identity is to determine whether the step edge and the reference area boundary belong to the same object. Specifically, whether the shadow area boundary exists on the extension line of the step edge or vice versa. Judgment is made based on whether or not a step edge exists on the extension line of the shadow area. And, for example, if the ratio of the shadow region boundary extracted is larger than the step edge, or the degree of gradient direction in grouping is high, and if the step edge does not exist on the extension line of the shadow region boundary, It is determined that the extraction accuracy of the step edge is not sufficient, and the process returns to S401 again to repeat the step edge extraction process. When performing step edge extraction processing again, it is possible to perform processing with higher accuracy, such as setting a processing window on the extended line of the shadow area boundary and correcting the threshold value to a higher value in threshold processing. Is preferred. Thereby, the extraction precision of a step edge can be raised and the outline of an object can be extracted more accurately.
[0040]
Although the embodiment of the present invention has been described above, the various processes shown in the present embodiment, specifically, the processes of FIGS. 2 and 5 to 7 are installed in a computer from a medium on which a processing program is recorded. Can be realized by sequentially executing. For example, in the case of FIG. 2, the processing program is stored in advance as a routine for calculating a difference between adjacent pixel data (luminance data) of the original image, a routine for storing the calculated difference value, that is, a spatial differential value at a predetermined address in the memory. A routine for reading out a predetermined threshold value from the memory and comparing it with the difference value to extract a step edge, a routine for storing the extracted step edge data in the memory, and calculating whether or not the step edge data constitutes a closed curve Routine, the difference value stored in the memory, that is, the spatial differential value is read, and it is determined whether or not the angle formed by the adjacent pixel data is below a certain value. Routine for setting a flag, demarcating the boundary by setting another flag for the pixel located at the boundary of the pixel group for which the flag is set Routine, and the boundary data of the stored step edge data and shaded area in the memory may include a routine for outputting to the display control unit integrated. The medium for storing the processing program includes any medium that can hold information electromagnetically, optically, or chemically, such as a CD-ROM, DVD, hard disk, or semiconductor memory.
[0041]
【The invention's effect】
According to the present invention, it is possible to reliably extract an edge in an image, particularly a shadow region having a gentle luminance change. This makes it possible to easily extract an object that has been difficult to detect in the past, such as a human hand or finger in a scene having a complicated background, and improve the accuracy of object recognition.
[Brief description of the drawings]
FIG. 1 is a configuration block diagram of an embodiment.
FIG. 2 is a processing flowchart of the embodiment.
FIG. 3 is an explanatory diagram showing a relationship between a luminance change and a luminance gradient.
FIG. 4 is a schematic explanatory diagram of the processing flowchart of FIG. 2;
FIG. 5 is a processing flowchart of another embodiment.
FIG. 6 is a processing flowchart of still another embodiment.
FIG. 7 is a processing flowchart of still another embodiment.
[Explanation of symbols]
10 image sensor, 12 A / D, 14 frame memory, 16 main memory, 18 CPU, 20 display control unit.

Claims (4)

An image processing apparatus for extracting an object in an image,
Differentiating means for differentiating the brightness of the original image by the second order space,
A shadow area extracting means for grouping areas where the second-order spatial differential value is substantially 0 to extract a shadow area of the object;
Edge extraction means for extracting a step edge of the object in the original image;
Integration means for integrating the boundary between the step edge and the shaded area;
And the differential means performs the second-order spatial differentiation by further spatially differentiating only a region where the spatial differential value of the luminance of the original image is not 0 , and the edge extraction means When the step edge of the object extracted in step 1 does not constitute a closed curve and is missing, a processing window is set in the missing portion, and the second-order spatial differentiation is performed in the processing window. Image processing device. An image processing apparatus for extracting an object in an image,
Differentiating means for differentiating the brightness of the original image by the second order space,
A shadow area extracting means for grouping areas where the second-order spatial differential value is substantially 0 to extract a shadow area of the object;
Edge extraction means for extracting a step edge of the object in the original image;
Integration means for integrating the boundary between the step edge and the shaded area;
And the differentiating means performs the second-order spatial differentiation by further differentiating only the region where the spatial differential value of the luminance of the original image is not 0,
The edge extracting means sets a processing window on an extension line of the shadow area boundary of the object extracted by the shadow area extracting means, and extracts the step edge in the processing window. . An image processing method for extracting an object in an image,
A differentiation step for second-order spatial differentiation of the luminance of the original image;
A shadow region extraction step of grouping regions where the second-order spatial differential value is substantially zero to extract a shadow region of the object;
An edge extraction step of extracting a step edge of the object in the original image;
An integration step of integrating a boundary between the step edge and the shaded area;
In the differentiation step, the second-order spatial differentiation is performed by further spatially differentiating only a region where the spatial differential value of the luminance of the original image is not 0, and the edge extraction step When the step edge of the object extracted in step 1 does not constitute a closed curve and is missing, a processing window is set in the missing portion, and the second-order spatial differentiation is performed in the processing window. Image processing method . An image processing method for extracting an object in an image,
A differentiation step for second-order spatial differentiation of the luminance of the original image;
A shadow region extraction step of grouping regions where the second-order spatial differential value is substantially zero to extract a shadow region of the object;
An edge extraction step of extracting a step edge of the object in the original image;
An integration step of integrating a boundary between the step edge and the shaded area;
In the differentiating step, the second-order spatial differentiation is performed by further spatially differentiating only the region where the spatial differential value of the luminance of the original image is not 0,
In the edge extraction step, a processing window is set on an extension line of the shadow region boundary of the object extracted in the shadow region extraction step, and the step edge is extracted in the processing window. .

Images