JPH0534117A - Image processing method - Google Patents

Abstract

PURPOSE:To make it possible to perform simple and highly accurate image processing without limiting a light source in the image processing, wherein the three-dimensional shape of an object is detected based on the equal luminance line with shading and stereographic display is performed. CONSTITUTION:The right and left stereographic images of an object are inputted (step S3). The luminance value at each point is adjusted (step S4). The luminance threshold value is changed at every constant interval (step S5), and the value is binary-coded (step 6), Smoothing (step S7) is performed, and an equal luminance line is extracted (step S8). Then, the corresponding of the equal luminance line is performed (step S9). The parallax of each point for which the corresponding point is obtained is extracted (step S10). The part between the lines is interpolated based on the parallax (step S12). The distance of each picture element is operated (step S13) based on the parallax data. The stereographic display of the object is performed (step S14) by using the distance data.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing method for performing three-dimensional display by calculating three-dimensional shape information of an object.

[0002]

2. Description of the Related Art In the case of displaying an image that gives a three-dimensional three-dimensional object a more realistic texture, image processing such as that disclosed in Japanese Laid-Open Patent Publication No. 2-29878 is performed. This is because, for example, the same object is imaged by two cameras whose image pickup surfaces are substantially the same plane and whose optical axes are parallel to each other, and pixels corresponding to each texture region are associated with each other, and the parallax is obtained from the result. Is obtained to obtain the distance to the object, and this is reproduced as a three-dimensional image.

Further, conventionally, when the shape of an object is determined using a light source, the normal direction of the surface is obtained from the characteristics of the reflectance of the surface of interest and the characteristics of the light source. As a result, since only the data in the normal direction of the surface can be obtained, it is necessary to give information such as surface irregularities in advance. This method is called shape from shading.

[0004]

In the conventional image processing method as described above, it is necessary to limit the light source used for shape determination, and the reflectance of the target surface must be obtained. However, the processing becomes complicated, and as a result, only the data in the normal direction of the surface can be obtained, so that the actual distance cannot be measured.

The present invention has been made by paying attention to the above problems, and is not limited to a light source. It is suitable for hardware implementation with simple processing, and an image capable of highly accurate processing. The purpose is to provide a processing method.

[0006]

According to an image processing method of the present invention, isoluminance lines are extracted from left and right stereo images of an object by shading, and stereo correspondence data of the isoluminance lines are obtained to obtain each point of the object. The parallax is extracted, the distances of the respective points of the object are measured from the parallax data, and the three-dimensional shape information of the object is calculated.

Further, the distance of the object is measured by using light sources from a plurality of directions, and the shielding contour and the shielding area are detected when the three-dimensional shape information of the object is calculated. It is the one.

[0008]

In the image processing method of the present invention, isoluminance lines due to shading are extracted from the left and right stereo images of the object, and stereo correspondence data of the isoluminance lines are obtained. Then, the parallax of each point of the object is extracted from the corresponding data, and the distance of each point of the object is measured from the parallax data. Thereby, the three-dimensional shape information of the target object is obtained.

At this time, the distance to the object is measured by using light sources from a plurality of directions, for example. Further, it is possible to prevent a false contour from being generated by detecting the occluding contour and the occluding area when the three-dimensional shape information of the object is calculated.

[0010]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is a block diagram showing the structure of an apparatus for carrying out image processing according to the present invention, and FIG. 2 is a schematic view of the structure of the camera. This apparatus has two ITV camera units 2 and 3 controlled by a microcomputer 1, and further includes an image memory 4 for storing image data, a display unit 5, a printer 6, a floppy disk 7 and a keyboard terminal 8. Is provided. Further, a host computer is also connected via the data bus 9, and the above-mentioned respective parts are controlled by the instruction given from here or the support given from the keyboard terminal 8 and the corresponding processing is performed. .

The above two ITV camera units 2 and 3
Is composed of ITV cameras 21 and 31 for outputting analog image pickup data, and A / D converters 22 and 32 for converting the analog data into digital data. The two ITV cameras 21 and 31 are horizontally arranged as shown in FIG. 2, and their optical axes 21a and 31a are parallel to each other and their coordinate systems are also parallel to each other. The projection center points O _L and O _{R of the} ITV cameras 21 and 31 are
On the other hand, the X-coordinate ground and the Y-coordinate of each of the points P _L and P _R of the projected image of the three-dimensional point P have the following relationship.

X _L > X _R Y _L = Y _R That is, a corresponding point on one image on one image lies on the half line of the same scan (raster scan) line. , The so-called epipolar condition is satisfied.

The image memory 4 is readable and writable, and has original image data of a left image taken by the left (L) side ITV camera 21 and a right image taken by the right (R) side ITV camera 31, and various data. The processing data of is stored. The display unit 5 and the printer 6 are
The processing result and the like of the computer 1 are output and the processing result and the like are registered in the floppy disk 7.

Next, the operation of the image processing of the present invention, that is, the stereoscopic processing by the distance measurement, by the apparatus having the above-mentioned configuration will be described. This stereoscopic processing is performed by the computer 1. First, isoluminance lines are extracted from the left and right stereo images of the object by shading, and then stereo correspondence data of the isoluminance lines are obtained to obtain each point of the object. The parallax is extracted, the distances of the respective points of the object are measured from the parallax data, and the three-dimensional shape information of the object is calculated. Hereinafter, the details of the above process will be described step by step with reference to the flowchart of FIG.

(1) When the stereoscopic processing is started in step S1, the computer 1 first initializes the image memory 4 and a register (not shown) used in this processing in step S2. Then, in step S3, the left and right original image data quantized by the ITV camera units 2 and 3 are written in the image memory 4, and the left and right images are input.

(2) The above two ITV cameras 21, 31
In step S4, the grayscale test pattern is used to obtain the same brightness value for each image so that the same brightness value is obtained for each of the left and right input images (the optical characteristics of the two cameras are slightly different). Adjust the brightness value of the point and adjust the input image. As a result, left and right stereo images of the object by shading as shown in FIG. 4 are obtained.

(3) Next, in step S5, the brightness threshold value is set while changing it at regular intervals, and in step S6, it is binarized. The image obtained by this is an image showing the luminance cut plane as shown in FIG.

(4) Since each area of the luminance cut surface contains noise near the boundary, smoothing processing is performed in step S7 to remove noise. In this smoothing process,
The contraction / expansion process is performed plural times, and as a result, an image as shown in FIG. 6 is obtained.

(5) The area is labeled while tracking the boundary of each area. At that time, a pixel in the vicinity of the pixel of interest is searched for, and the inclination and coordinates of the boundary line are checked from those pixel data. In this way, isoluminance lines (region boundary lines) are extracted in step S8, and an image of isoluminance lines as shown in FIG. 7 is obtained. Here, there is an inclusion relationship as shown in FIG. 8 with the cut surface with another brightness threshold value. FIG. 8 shows the inclusion relation of the brightness values t1, t2, t3 by the threshold value in the form of a tree structure using the label numbers of the areas.

(6) Next, equiluminance lines are associated in step S9. This is to obtain stereo correspondence data of equiluminance lines. Here, first, it is determined whether or not there is a point satisfying the above-mentioned epipolar condition between the areas. For this determination, the vertex coordinates of the circumscribed rectangle of the area are used for simplification. This correspondence can also be obtained from the inclusive relation information obtained in the process (5). Then, when the correspondence between the regions is obtained in this manner, the corresponding points of the boundary line (isoluminance line) are then determined as shown in FIG. 9 using the epipolar condition or the isoluminance line obtained by the process of (5). Calculated from the slope of.

(7) In step S10, the parallax (deviation in the main scanning direction) of each of the above-mentioned corresponding points is calculated from each coordinate value, and the parallax is extracted. This is to extract the parallax of each point of the object from the stereo correspondence data of the isoluminance lines.

(8) The above processes (3) to (7) are repeated over the entire density range while changing the brightness threshold value. That is, the operations of steps S5 to S10 are repeated until the extraction of all parallaxes is completed in step S11. (9) Then, in step S12, the portion between the lines is interpolated from the parallax obtained on the isoluminance line.

(10) Next, in step S13, the distance of each point of the object is measured from the extracted parallax data, that is, the distance of each pixel is calculated. At that time, as shown in FIG. 2, the point P on the same object is moved to the left and right ITV cameras 21,
As a result of performing the above-described processing on the image captured by the image sensor 31 and the pixel P _L of the left image and the pixel P _{R of the} right image, the point P is the focus O of the ITV camera 21 on the left side. _L and the line connecting the points P _L of pixels in the projection surface, will be present at the intersection of the straight line connecting the focal point O _R and the point P _R of the pixel of the projection surface of the right side of ITV camera 31. Therefore, the distance between the optical axes of the ITV cameras 21 and 31 is 2a,
The focal length of each ITV camera 21, 31 is f, and the point P _L
If the parallax between P _R and P _R is D, the distance Z to the point P on the object can be obtained from the following equation. Z = 2af / D (11) When the three-dimensional shape information of the target object is calculated from the distance data, the target object is stereoscopically displayed in step S14. At this time, a display image as shown in FIG. 10 is obtained.

The processing from the imaging of the object to the stereoscopic display has been described above, but it is considered that the actual object contains some diametrically opposite components. Therefore, in order to improve this point, it is effective to measure the distance to the object using light sources from a plurality of directions as shown in FIG. The processing operation will be described below step by step with reference to the flowchart of FIG.

(12) First, in step S15a, as shown in FIG.
As shown in (a) of FIG. 13, the object is illuminated from the first light source 11 on the left side, and a stereo image in the first direction as shown in FIG. 13 is obtained. Then, in step S16a, the same processing as described above is performed to calculate the distance data to the object.

(13) Next, in step S15b, as shown in FIG.
As shown in (b), the object is illuminated from the second light source 12 on the right side, and a stereo image in the second direction as shown in FIG. 14 is obtained. Then, in step S16b, the distance data to the object is similarly calculated.

(14) Each point P obtained as described above
The distance data of (ij) are integrated in step S17 according to the condition of the following equation. That is, if | I1 (i, j) -T _max / 2
If | <| I2 (i, j) -T _max / 2 |, then D (i,
j) = D1 (i, j) or D (i, j) = D2 (i, j). Here, D1: distance of corresponding points (first light source) I1: luminance value of corresponding points (first light source) D2: distance of corresponding points (second light source) I2: luminance value of corresponding points (second light source) T _max : Maximum brightness value.

This is because it is considered that the very bright part of the image is largely influenced by the specular reflection component, and the very dark part of the image cannot obtain isoluminance lines. In the integration of, the final value is the distance obtained from the one closer to the intermediate value of the dynamic range of luminance by comparing the brightness of each point of the image. And step S1
At 8, the stereoscopic display of the object is performed based on the distance data. At this time, a display image as shown in FIG. 15 is obtained.

In this way, the stereo correspondence of the isoluminance lines obtained from the shading is obtained, the parallax is extracted from the result, and the three-dimensional shape information of the object is calculated without contact. Further, by using the light sources from a plurality of directions, the accuracy becomes higher and the actual distance can be easily measured. Further, the processing can be performed independently for each cut surface, which is suitable for parallel processing, and individual processing is simple and suitable for hardware implementation.

Here, when calculating the above-mentioned three-dimensional shape information, the shielding contour or the shielding region may be detected. That is, in a stereo image, occlusion (occlusion) occurs in a portion that can be seen by one camera but cannot be seen by the other camera. And if such a part cannot be detected as occlusion, it will cause the generation of a plane. Therefore, the occlusion is detected by using the above isoluminance line.

FIG. 16 shows an example of occurrence of occlusion. As shown in FIG. 16A, there are two isoluminance lines on the same epipolar line in one image, but one point in the other image. become. That is, when the parallax at the point A is smaller than the parallax at the point D, the region surrounded by at least the points A _L , B _L , C _L , and D _L of the left image in the right image as shown in FIG. It becomes an invisible occlusion area.

Further, the occlusion detected by the shielding contour portion formed by a cylinder or the like may be detected by detecting a portion where there is no corresponding portion as shown in FIG. In FIG. 17,
The area corresponding to A and the area B indicate occlusion.

FIG. 18 shows an example of a stereo image of an object having such occlusion, and FIG. 19 shows an image of its isoluminance lines. Further, FIG. 20 shows a display image obtained as a result of performing occlusion detection on the left image of this object.

As described above, the occlusion can be detected at the same time as the distance measurement by the left and right correspondence can be performed by the information on the luminance information and the inclination of the isoluminance line, and the generation of the false surface can be prevented.

[0035]

As described above, according to the present invention, the isoluminance lines by shading are extracted from the left and right stereo images of the object, and the stereo correspondence data of the isoluminance lines are obtained to determine each point of the object. Since the parallax is extracted and the distance is measured from the parallax data to calculate the three-dimensional shape information of the object, the light source is not limited and the actual distance can be easily measured. The simple processing is suitable for hardware implementation, and has the effect of enabling highly accurate processing.

Further, it is possible to prevent the influence of the specular reflection component by using the light sources from a plurality of directions, and it is possible to prevent the generation of a false surface by detecting the shield ring portion and the shield region. it can.

[Brief description of drawings]

FIG. 1 is a block diagram showing a device configuration for performing image processing according to the present invention.

2 is a schematic diagram showing the configuration of the ITV camera of FIG.

FIG. 3 is a flowchart showing a processing operation by the computer of FIG.

FIG. 4 is an explanatory diagram showing a display example of a stereo image obtained by the processing of FIG.

FIG. 5 is an explanatory diagram showing a display example of a luminance cut surface obtained by the processing of FIG.

FIG. 6 is an explanatory diagram showing an example in which the image of FIG. 5 is smoothed.

FIG. 7 is an explanatory diagram showing an example of an image in which isoluminance lines are extracted by the processing of FIG.

FIG. 8 is an explanatory view showing the inclusion relationship of cut surfaces by a threshold value in the label image of FIG.

FIG. 9 is an explanatory diagram showing how corresponding points of isoluminance lines are obtained by the processing of FIG.

FIG. 10 is an explanatory diagram showing an example of a stereoscopic display image by the processing of FIG.

11 is a block diagram showing an example of using light sources from a plurality of directions in the device of FIG.

FIG. 12 is a flowchart showing the processing operation of the apparatus having the configuration of FIG.

13 is an explanatory diagram showing a display example of a stereo image obtained by the processing of FIG.

14 is an explanatory diagram showing a display example of a stereo image obtained by the processing of FIG.

FIG. 15 is an explanatory diagram showing an example of a stereoscopic display image by the processing of FIG.

FIG. 16 is an explanatory diagram showing how to detect a shielding area.

FIG. 17 is an explanatory diagram showing how to detect a shielding contour.

FIG. 18 is an explanatory diagram showing an example of a stereo image with occlusion.

FIG. 19 is an explanatory diagram showing an image example in which isoluminance lines are extracted from the image of FIG.

20 is an explanatory diagram showing a result of performing occlusion detection on the image of FIG.

[Explanation of symbols]

1 computer 2 ITV camera unit 3 ITV camera unit 4 image memory 5 display unit 6 printer 7 Proppy disk 8 keyboard terminals 21,31 ITV camera 22,32 A / D converter

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Takashi Yoshimi             1-4, Umezono, Tsukuba-shi, Ibaraki Industrial Technology             Inside the Institute for Electronic Technology Research (72) Inventor Yutaka Ishiyama             3-2-1 Takanawa, Minato-ku, Tokyo

Claims (4)

[Claims] 1. An isoluminance line is extracted from left and right stereo images of an object by shading, stereo correspondence data of the isoluminance line is obtained, and parallax of each point of the object is extracted.
An image processing method, characterized in that the three-dimensional shape information of the target portion is calculated by measuring the distance of each point of the target from the parallax data. 2. The image processing method according to claim 1, wherein the distance to the object is measured using light sources from a plurality of directions. 3. The image processing method according to claim 1, wherein a shielding contour is detected when the three-dimensional shape information of the object is calculated. 4. The image processing method according to claim 1, wherein the shielded area is detected when the three-dimensional shape information of the object is calculated.