JPH09231369A - Picture information input device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a picture information input device capable of obtaining depth information flexibly corresponding to processing for converting a subject into an objective picture format. SOLUTION: Depth information extracting processing executes separating processing and an image picked up at a main point 1110 indicating the image pickup reference point of a camera head becomes a background separated picture 1100 in which a background 1102 and a subject 1101 are separated by the separating processing. The existence probability of a main subject in voxel space 1120 is voted by respective segments 1130 to 1134. In the voting action, the inside part of the segments 1130 to 1140 is regarded as the subject and '1' is integrated, and when the inside part of the segments 1130 to 1140 exists in the voxel space, '1' is added. Thus voting, e.g. integrating processing, by plural segments started from the outline of the subject and passing the main point is executed.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image information input device for extracting depth information of a subject based on a subject image picked up from an arbitrary position.

[0002]

2. Description of the Related Art Conventionally, techniques for obtaining a three-dimensional shape of an object are roughly classified into a passive technique and an active technique.

As a typical passive method, there is a stereo image method in which two cameras are used for triangulation.
In this stereo image method, a place where the same one is projected is searched from the left and right images, and the three-dimensional position of the subject is measured from the amount of displacement of the position indicating that place.

On the other hand, as a typical active method, light is projected onto a subject from a predetermined position, and the time until the reflected light from the subject returns to the predetermined position is measured and measured. There is a method for obtaining the distance from the time to the subject, and this method is applied to an optical radar type range finder and the like.

Further, there is a slit light projection method in which a slit-shaped light pattern is projected on a subject and the three-dimensional shape of the subject is measured from the displacement of the pattern shape reflected on the subject.

From the three-dimensional data of the subject obtained by these methods, a subject image is generated when the subject is viewed from an arbitrary viewpoint, and the subject image is displayed on a two-dimensional display or the like. The technique for obtaining a three-dimensional shape is widely used in the field of image processing.

In recent years, with the spread of personal computers and the rapid improvement in performance thereof, application software capable of capturing images taken by an electronic camera and editing the images by personal computers has appeared. For example, if a landscape is divided into a plurality of images by an electronic camera and the plurality of images are captured, the plurality of images are captured by a personal computer, and the personal computer activates the application software so that the captured images can be captured. It is possible to process the captured image in a limited range, such as performing editing or the like.

[0008]

However, in the above-mentioned stereo image method, the main purpose is to calculate the distance information from the specific position where the camera is installed, and the object is to directly measure the three-dimensional shape itself of the subject. And not.

Further, in the active method, since the projection on the subject is performed by irradiating the subject with a laser beam or the like, a complicated work is required in its use.

Furthermore, by applying either the passive method or the active method to a dynamic imaging system in which an object is imaged while moving around it, a method of extracting depth information for the object. However, depending on the method, it is not possible to extract depth information that can flexibly deal with image processing.

Further, the processed image is often output to a two-dimensional display, paper, or the like, and the image form has various forms such as a natural image and a line drawing representing an object by a contour line. It consists of cases. That is,
Since the focus is on faithfully outputting the subject shape to a two-dimensional display or paper based on the distance information obtained by the above-mentioned method, etc., in general, various three-dimensional data of the subject can be obtained. Depth information that can flexibly handle image processing for processing from the side has not been extracted, and the subject has not been converted to an image form that requires depth information that can flexibly respond. .

On the other hand, although application software capable of capturing an image captured by an electronic camera and editing the image has appeared, the application software is used for processing three-dimensional data of a subject from various aspects. Since depth information that can flexibly deal with image processing cannot be obtained, the processing content for the captured image is limited, and the subject cannot be converted into the desired image form.

It is an object of the present invention to provide an image information input device capable of obtaining depth information that can flexibly deal with processing for converting a subject into a desired image form.

[0014]

According to the first aspect of the present invention,
An image pickup means for picking up an image of a subject through at least one optical system and outputting the subject image, and a depth information extraction means for extracting depth information of the subject based on the subject image picked up from an arbitrary position by the image pickup means. In the image information input device, the depth information extraction means performs a process for obtaining depth information in a voxel space from a boundary portion of object images captured from a plurality of viewpoints, and in the voxel space obtained by the process. It is characterized in that the ratios of the subject images existing in the are integrated and the depth information is extracted according to the integration result.

According to a second aspect of the invention, in the image information input apparatus according to the first aspect, the depth information extracting means is
The accumulation of the ratio of the subject image existing in the voxel space is performed by integrating the existence probabilities from the respective viewpoints, and the one that exceeds a set threshold value in the integration result is extracted as the depth information. It is characterized by

[0016]

Embodiments of the present invention will be described below with reference to the drawings.

(First Embodiment) FIG. 1 is a block diagram showing the configuration of the first embodiment of the image information input apparatus according to the present invention.

The image information input device is, as shown in FIG.
A camera head 1 for capturing an image of a subject is provided. In this example,
The camera head 1 images a detected object (subject) 2 with the back surface 3 as the background.

The camera head 1 includes image pickup lenses 100L and 100R provided on the left and right, and an image pickup lens 100L.
And the illumination unit 2 disposed between the image pickup lens 100R and
00.

The image pickup lens 100L is arranged on the left side when viewed from the device side, and the image pickup lens 100R is arranged on the right side when viewed from the device side. The imaging range of the imaging lens 100L is a range represented by 10L in the figure, and the imaging range of the imaging lens 100R is a range represented by 10R in the figure. The camera head 1 is provided with each imaging lens 100.
A subject captured via L and 100R is imaged on an image pickup device (not shown) including a CCD, and the CCD converts the subject image into a corresponding electric signal by photoelectric conversion, and this electric signal is converted into an analog / digital signal. Is converted into digital signal image data.

The attitude and the position of the camera head 1 are detected by the camera attitude position detector 4. The camera posture position detection unit 4 has means for calculating angle information based on a detection signal from a sensor such as a gyro and means for calculating posture position information by performing image processing based on information obtained from the back surface 3. Then, the posture and position of the camera head 1 are detected by these means. In the present embodiment, the camera head 1 is moved with the position between A0 and An as the imaging position, and the position of the camera head 1 is detected by the camera attitude position detection unit 4.

The image data obtained by the camera head 1 and the posture position information obtained by the camera posture position detection unit 4 corresponding to the image data are stored in one of the image memories 5a and 5b in association with each other. The image data stored in the image memories 5a and 5b and the posture position information corresponding to the image data are given to the depth information calculation unit 6.

The depth information calculation unit 6 performs a calculation based on the image data and the posture position information corresponding to the image data, and calculates three-dimensional image data (depth information) that defines the three-dimensional shape of the subject by this calculation.

The calculated three-dimensional image data is given to the two-dimensional image data operation unit 7, and the two-dimensional image data operation unit 7 specifies the image form and the user's specified image form based on the three-dimensional image data. Two-dimensional image data of the subject is calculated under the condition that the subject is viewed from the viewpoint. From the calculated two-dimensional image data of the subject, a two-dimensional image in the image form specified by the user and viewed from the specified viewpoint can be obtained. The designation of the image form and the designation of the viewpoint are performed from the operation unit 11, and the viewpoint can be designated at an arbitrary position.

The two-dimensional image data calculation unit 7 includes a camera head 1, a camera attitude position detection unit 4, and image memories 5a and 5a.
b, the image information input device is configured in cooperation with the depth information calculation unit 6.

2 calculated by the two-dimensional image data calculation unit 7
The three-dimensional image data includes operation means 11 and data synthesizing means 10.
00 or the printer 9.

The data synthesizing means 1000 is the operating means 11
Based on the instruction, the synthesizing process for fitting the image represented by the two-dimensional image data supplied to the sentence created by the sentence data creating unit 1001 is performed, and the result of the synthesizing process is displayed on the monitor 8 via the operating unit 11. It

The printer 9 prints the image represented by the two-dimensional image data supplied to the operating means 11 based on the instruction, and also prints the image composite text supplied via the operating means 11.

The operating means 11 sets the above-mentioned conditions and the like, and gives an instruction to the data synthesizing means 1000 and the printer 9, an image indicated by the supplied two-dimensional image data, and a monitor 8 of the supplied image composite text. Is displayed.

A photographing operation by the image information input device will be described.

First, the camera head 1 is directed toward the object to be detected 2 at the initial position A0, for example.
Next, the user operates the release button (not shown), and the detection object 2 is photographed at the position A0.

When the photographing at the position A0 is completed, the camera head 1 sequentially moves from the position A0 to the position A around the detected object 2.
It is moved on the circular arc toward the n at a predetermined shift angle amount. When the camera posture position detection unit 4 detects the movement of the camera head 1 by a predetermined angle amount, the next shooting is performed, and the position A
Shooting up to n is performed. For example, between A0 and An is 18
When shooting is performed in sequence with a moving pitch of 0 degree and 3 degrees, 60
Individual image data will be obtained.

The setting of the movement angle amount is determined from the detection performance of a sensor such as a gyro of the camera posture position detection unit 4 and the resolution of depth information to be obtained. For example, when the detection capability of the gyro is 90 degrees / sec, the movement angle amount is set to 3 degrees / frame speed.

During the photographing, the image data and the displacement amount with respect to the photographing position and the direction thereof are associated with each other, and the image memory 5
It is stored in a and 5b. Further, when at least one of the position and the direction of the camera head 1 moves more than a predetermined value, a warning is issued from a warning means (not shown). Further, when sufficient image data is obtained for the calculation of the depth information of the detected object 2, the user is notified that sufficient image data has been obtained by the image capturing end notifying means (not shown), and the user is notified. Shooting is terminated based on the notification.

When the photographing is completed, the image memories 5a and 5b
The image data and the corresponding posture position information stored therein are given to the depth information calculation unit 6, and the depth information calculation unit 6 performs calculation based on the image data and the posture position information corresponding thereto, and by this calculation, Three-dimensional image data (depth information) that defines a three-dimensional shape is calculated.

The calculated three-dimensional image data is given to the two-dimensional image data calculation unit 7, and the two-dimensional image data calculation unit 7 specifies the image form and the designation by the user based on the three-dimensional image data. Two-dimensional image data of the subject is calculated under the condition that the subject is viewed from the viewpoint. From the calculated two-dimensional image data of the subject, a two-dimensional image in the image form specified by the user and viewed from the specified viewpoint is obtained.
The obtained two-dimensional image is displayed on the monitor 8. When the image form and the viewpoint are changed by the operating means 11, a two-dimensional image in the changed image form and viewed from the viewpoint is obtained.

Next, the detailed structure of the image information input device will be described with reference to FIGS. 2 and 3 are block diagrams showing a detailed configuration of the image information input device of FIG. 1, and FIG. 4 is a block diagram showing a configuration of the system controller of FIG.

As shown in FIGS. 2 and 3, the camera head 1 is substantially symmetrical with the left and right imaging lenses 100L and 100R.

As shown in FIG. 2, the image pickup lens 100L has a zoom mechanism (not shown) and an autofocus mechanism (not shown), and the zoom mechanism is the zoom controller 10.
6L, the autofocus mechanism is the focus control unit 10
Each of them is driven and controlled by 7L.

A diaphragm 101L for adjusting the amount of light is arranged behind the image pickup lens 100L, and the diaphragm amount of the diaphragm 101L is driven and controlled by a diaphragm controller 108L.

A subject image whose light amount has been adjusted is formed on an image sensor (CCD) 102L, and the image sensor 10
The 2L converts a subject image into a corresponding electric signal by photoelectric conversion. The image sensor 102L is driven and controlled by the image sensor driver 120L.

The electric signal of the image sensor 102L is A /
The D conversion unit 103L converts the digital signal, and the digital signal is stored in the memories 83 and 85.

The image pickup lens 100R is the image pickup lens 100R.
Like L, it has a zoom mechanism (not shown) and an autofocus mechanism (not shown). The zoom mechanism is driven and controlled by the zoom control unit 106R and the autofocus mechanism by the focus control unit 107R.

A diaphragm 101R for adjusting the amount of light is arranged behind the image pickup lens 100R, and the diaphragm amount of the diaphragm 101R is driven and controlled by a diaphragm controller 108R.

A subject image whose light amount has been adjusted is formed on an image sensor (CCD) 102R, and the image sensor 10
The 2R converts a subject image into a corresponding electric signal by photoelectric conversion. The image sensor 102R is driven and controlled by the image sensor driver 120R.

The electric signal of the image sensor 102R is A /
The D conversion unit 103R converts the digital signal, and the digital signal is stored in the memories 73 and 75.

The system controller 210 controls the zoom controller 106L, focus controller 107L, aperture controller 108L, image sensor driver 120L, zoom controller 106R, focus controller 107R, aperture controller 108R, and image sensor driver 120R. Is given.

The system controller 210 controls the lighting 20.
While controlling the drive of 0, the camera posture position detection unit 4 monitors the posture position information, and controls the drive of the sounding body 97 for issuing the above-mentioned warning sound. The sounding body 97 emits a photographing end notification sound indicating that the photographing end notification is issued based on an instruction from the system controller 210, together with the warning sound.
The sounding body 97 constitutes the above-mentioned warning means and shooting end notification means.

The image data stored in each of the memories 83 and 85 is, as shown in FIG. 3, a video signal processing unit 104L.
And to the overlap detector 92.

The video signal processing unit 104L includes each memory 8
A conversion process for converting 3,85 digital signals into luminance signals and chrominance signals of appropriate forms is performed. The luminance signal and the color signal of the video signal processing unit 104L are the subject separation unit 1
05L and the display unit 240.

The subject separation unit 105L separates the main subject, which is the measurement target of depth information, and the background thereof from the luminance signal and the color signal of the video signal processing unit 104L. As this separation method, for example, an image of the rear surface is captured in advance, the image is held in a memory, and then a main subject to be measured is captured together with its background, and the captured image and the rear surface stored in the memory in advance. The method of performing the matching and the difference processing with the image of and separating the back area is used. Instead of this method, a method of separating the main subject and its background based on color or texture information can also be used. In the present embodiment, the detected object 2 is used as the main subject.
Are separated from the back surface 3.

The image of the main subject separated from the background by the subject separation unit 105L is given to the image processing unit 220 and the focus detection unit 270.

Similarly, the image data stored in the memories 73 and 75 are given to the video signal processing unit 104R and the overlap detecting unit 92.

The video signal processing unit 104R is provided in each memory 7
A conversion process for converting the 3,75 digital signals into luminance signals and chrominance signals of appropriate forms is performed. The luminance signal and the color signal of the video signal processing unit 104R are the subject separation unit 1
05R and the display unit 240.

The subject separation unit 105R separates the main subject, which is the measurement target of the depth information, and the background thereof from the luminance signal and the color signal of the video signal processing unit 104R. This separation method is the same as the above-mentioned method.

The image data of the main subject separated from the background by the subject separation unit 105R is given to the image processing unit 220 and the focus detection unit 270.

The overlap detecting section 92 detects the degree of weight of the object captured by each of the image pickup lenses 100L and 100R based on the image data captured by each of the image pickup lenses 100L and 100R, and the detection result information is the system. Provided to controller 210 (shown in FIG. 2).

The image processing section 220 is provided for each image pickup lens 100.
The depth information of the subject is extracted based on the image data of each main subject captured by L and 100R, and the depth information of the subject at each of the extracted image pickup points is based on the posture position information obtained from the camera posture position detection unit 4. Integrate and output. The extraction of the depth information and its integration will be described later. The image processing unit 220 includes the image memories 5a and 5b, the depth information calculation unit 6, and the two-dimensional image data calculation unit 7 shown in FIG. The depth information extracted by the image processing unit 220 is stored in the recording unit 25.
Recorded as 0.

The display section 240 includes the image pickup lenses 100L,
The subject image captured at 100R is displayed.

The focus detection section 270 performs a calculation based on the image data of each main subject, and obtains the focal length information in each of the image pickup lenses 100L and 100R by the calculation.
The obtained focal length information of each of the imaging lenses 100L and 100R is given to the system controller 210.

The system controller 210 includes an overlap detector 92 and video signal processors 104L and 104.
R, subject separation units 105L and 105R, image processing unit 22
0, the release button 230, the focus detection unit 270, and the external input I / F 760 are connected, and transfer control of information between these blocks and operation control are performed.

The external input I / F 760 is connected to an external device such as a computer, and a setting instruction from the external device is transmitted via the external input I / F 760 to the system controller 210.
Given to.

As shown in FIG. 4, the system controller 210 includes a microcomputer 900 and a memory 91.
0, and an image calculation processing unit 920.

Next, the operation of extracting depth information in this image information input device will be described with reference to the drawings.

First, the depth information extraction processing in the image processing unit 220 (shown in FIG. 3) will be described in detail.

The image processing section 220 includes the image pickup lenses 100.
Depth information extraction processing (referred to as depth information extraction processing # 1) for each main subject based on each parameter at the time of imaging each main subject captured by L and 100R, and the brightness of each main subject that is the target of depth information measurement. Extraction processing for obtaining parallax based on signals and color signals and extracting depth information from the obtained parallax (referred to as depth information extraction processing # 2)
And the depth information obtained by the process # 1 and the depth information obtained by the process # 2 are combined to obtain the entire depth information.

The focal length is set based on the distance information. The relationship between the focal length and the distance information is expressed by the following equation (1).

[0068]

[Equation 1] In this equation, r is the distance, f is the focal length, b is the baseline length, and d is the parallax.

Here, when the distance resolution determined by the parallax is used as a parameter, the focal length f is calculated from the following equations (2) and (3).

[0070]

[Equation 2]

[0071]

(Equation 3) Therefore, it is also possible to set the resolution from a computer or the like via the external input I / F 760 and set the focal length f based on the value of this resolution.

Next, the depth information extraction processing # 1 will be described with reference to FIGS. 5 is a diagram conceptually showing depth information extraction processing using polar coordinate voxel space, FIG. 6 is a diagram conceptually showing a method of extracting depth information from a multi-viewpoint contour image, and FIG. 7 is an 8-viewpoint contour image. FIG. 8 is a diagram conceptually showing the state of the voting process for FIG. 8, and FIG. 8 is a diagram showing the voting situation by the voting process of FIG.

For the depth information extraction processing # 1, a method of extracting depth information from the image of the main subject separated from the background in the voxel space coordinate system is used. In this method, first, the subject separation units 105L and 105R separate the main subject, which is the measurement target of depth information, and the background thereof from the luminance signal and the color signal of the video signal processing units 104L and 104R.

By this separation processing, as shown in FIG. 5, the image picked up at the principal point 1110 indicating the image pickup reference point of the camera head 1 becomes a background separated image 1100 in which the background 1102 and the subject 1101 are separated. This separation method is the method described above, and the flag “1” indicating that the main object after separation is the depth measurement target object has the background 1
A flag "0" is set in each of 102, whereby a background separated image 1100 is obtained. This background 11
From the contour of the subject 1101 separated from 02, the principal point 11
A plurality of line segments 1130 to 1134 passing through 10 are extended toward the voxel space 1120 represented by polar coordinates.

The line segments 1130 to 1134 vote for the existence probability of the main subject in the voxel space 1120. The act of voting is this line segment 1130-11
It is assumed that the portion inside 40 is a subject, and for example, it is an act of accumulating "1". If the portion inside line segments 1130 to 1140 exists inside the voxel space, "1" is added.

Next, while changing the principal point, as described above, the voting, that is, the integrating process is performed from the contour of the subject by a plurality of line segments passing through the principal point. In the present embodiment, the camera head 1 sequentially moves from the position A0 to the position A around the subject.
It is assumed that the camera head 1 moves along an arc at a predetermined shift angle amount toward n, and this processing will be described with reference to the moving cross section of the camera head 1.

For example, as shown in FIG. 6A, the camera head 1 positions the main point, that is, the viewpoint in order at positions 1200 to 1.
The sensor surface 12
For each subject obtained in 40 to 1243, boundary lines 1220 to 1227 extending from the outline to the voxel space 1210 via the principal points 1200 to 1203 are drawn, and the inner portion of each of the boundary lines 1220 to 1227 is the subject. The area 1230 is obtained as the highest vote area by the vote.

By using the subject from multiple viewpoints, a region 1230 surrounded by abcdefgh is obtained in the voxel space 1210 as the subject extraction result.

The obtained area 1230 is shown in FIG. 6 (b).
As shown in, the shape becomes a shape including the true object shape 1231. By further increasing the viewpoint, it can be seen that the shape obtained in the voxel space comes close to the true object shape.

In the above description, the processing is performed assuming that the movement of the camera head 1 is performed along the two-dimensional plane. However, when this processing is performed three-dimensionally, for example, as shown in FIG. , Calculate the line segment that passes through the principal point from the position (X, Y, Z) on the voxel space side, calculate the projection position for the image of that line segment, and refer to the image flag for that projection position to refer to the background A region for the subject is obtained in the voxel space by determining whether or not. This operation is repeated from the origin (0,0,0) of the voxel space to (X0, Y0, Z0).

Here, assuming that the projected point is (x, y), x, y can be obtained from the following equations (4), (5) and (6).

[0082]

(Equation 4)

[0083]

(Equation 5)

[0084]

(Equation 6) However, (a, b, c) indicates the coordinates of the principal point 1110.

Whether the projection position (x, y) is the background region or the subject region is determined by referring to the flag of the projection position. When x, y is not an integer value, the flag of the neighboring region is used. It is determined whether the background area or the subject area. That is, it is determined based on the flag data whether or not the voxel data is in the subject, and when it is in the subject, the vote is incremented by "1".

By repeatedly performing the operation of drawing a line segment from the voxel space side toward the image as described above while changing the principal point, the depth information of the subject is expressed in the voxel space as a vote.

In the above example, only the highest-voting part is selected as the extraction result of the depth information from the voting result. However, a predetermined threshold value is set in advance, and the voting result equal to or more than the threshold value is incorporated as the depth information. Can also be used. This method can reduce the error. The reason is that, as a process similar to this voting process, there is a process of cutting out the voxel space using the boundary part (outline part) of the subject, and after cutting out by this process, the same region as the highest-voting part region is obtained. However, too much cutting may occur such that the position cut out by the boundary is too inward. On the other hand, with the method of setting a predetermined threshold value in advance, it is possible to capture the highest number of votes, the next point, and the next point, and it is possible to suppress the occurrence of an error due to overcutting as much as possible.

Next, as shown in FIG.
A voting process similar to the process shown in (b) will be described with reference to FIGS. 7 and 8. In FIG. 7, the results of the voting state from the eight viewpoints are shown two-dimensionally, and the sensor cross section, the boundary line, etc. are omitted.

As shown in FIG. 7, the voting state for this cylindrical object is such that, depending on the gradation, the degree to which the vote is obtained on a part 5210 of the voxel space in the Cartesian coordinate system depends on the gradation.
It is displayed dimensionally. In this figure, black indicates the vote “0”, the higher the gradation toward white, the higher the vote, and the maximum vote portion is shown in white.

When the voting results are three-dimensionally expressed, as shown in FIG. 8, the horizontal axis is the axis indicating the coordinates of the cross section of the voxel space, and the vertical axis is the number of votes. In this figure, the maximum number of votes is 8, the runner-up is 7, and so on. As is clear from this figure, the higher the number of votes, the closer to the correct shape of the subject. In addition, as described above, in order to reduce the error, the preset threshold value is set to 7 votes, and the portion of 7 votes or more is extracted as the depth information.

Next, the depth information extraction processing from the stereo image will be described with reference to FIGS. 9 to 11. FIG. 9 is a diagram showing a process of extracting depth information from a contour image, FIG. 10 is a diagram showing point plate matching, and FIG. 11 is a diagram schematically showing calculation of depth information from a stereo image.

First, a processing procedure for extracting depth information from a stereo image will be described with reference to FIG.

Referring to FIG. 9, an R image 110R (hereinafter referred to as image 110R) stored in the image memory.
The L image 110L (hereinafter, referred to as image 110L) that forms a stereo image in cooperation with the edge extraction unit 111L,
Feature point extraction unit 117L, stereo image corresponding point extraction unit 11
2 given. Similarly, the image 110R is provided to the edge extraction unit 111R, the feature point extraction unit 117R, and the stereo image corresponding point extraction unit 112.

Each of the edge extraction units 111L and 111R generates an image in which edges are extracted from the corresponding image. The feature point extraction units 117L and 117R perform processing for identifying the features of the back surface. The stereo image corresponding point extraction unit 112 performs a process of extracting the correspondence relationship between pixels of each image.

The images generated by the edge extracting units 111L and 111R are given to the edge image corresponding point extracting unit 113, and the edge image corresponding point extracting unit 113 performs a process of extracting the correspondence between pixels of each image. In the processing, the correspondence relationship of each pixel is extracted from the brightness value of each image.

The correspondence relationship extracted by the edge image corresponding point extracting section 113 is given to the removal processing section 114 together with the correspondence relationship extracted by the stereo image corresponding point extracting section 112. The removal processing unit 114 uses the edge image corresponding point extraction unit 11
It is determined whether there is a contradiction between the correspondence extracted in 3 and the correspondence extracted by the stereo image corresponding point extraction unit 112, and the contradiction between the correspondences is determined according to the determination result. Perform the removal process. In this process, when the correspondence relationship extracted by the edge image corresponding point extraction unit 113 and the correspondence relationship extracted by the stereo image corresponding point extraction unit 112 do not match, the correspondence relationship is excluded because of low reliability. . Further, it is also possible to weight each correspondence relationship to make the determination.

The processing result of the removal processing unit 114 is given to the occlusion area determination processing unit 115 and the depth information distribution calculation processing unit 116.

The occlusion area determination processing section 115
The occlusion area is determined from the corresponding points obtained from the processing result of the removal processing unit 114 and an index indicating the degree of correlation used in the process of obtaining the corresponding points, for example, the residual.
Reliability is added to the determination result and the result is output. For example, if the correlation coefficient or residual is used as an index indicating the degree of correlation, and the correlation coefficient is low or the residual is large, it is determined that the reliability of the correspondence relationship is low, and the low portion is occluded. Treat as an area or an area with no correspondence.

The depth information distribution calculation processing unit 116 calculates the depth information distribution from each correspondence relationship by the principle of triangulation,
The calculated depth information distribution is output. The calculation of the depth information distribution uses shooting parameters, position detection information, and the like from the correction data calculation processing unit 118 described later. The triangulation is as described in the equation (1).

The back face feature points identified by the respective feature point extraction units 117L and 117R are given to the correction data calculation processing unit 118, and the correction data calculation processing unit 118 acquires the photographing parameters and is detected by the gyro 880. The attitude and movement relationship of the camera head 1 are acquired from the information such as the angle change. The acquired shooting parameters and the position detection information including the posture and movement relationship of the camera head 1 are given to the depth information distribution calculation processing unit 116.

Next, the corresponding point extraction method in the edge image corresponding point extraction section 113 will be described with reference to FIG.

A typical matching point extraction method in the edge image corresponding point extraction section 113 is a template matching method. Figure 1 shows the template matching method.
As shown in 0, a template image of N × N pixels is cut out from the image obtained from the left imaging lens. The cut out N × N pixels are moved on a search region range RA (a region represented by the following formula (7)) in the input pixels of M × M pixels of the image obtained from the right imaging lens, Residual R
The position of the template image is obtained so that (a, b) (obtained by the following equation (8)) is minimized. The obtained position is the center pixel position of the N × N template image.

[0103]

(Equation 7)

[0104]

(Equation 8) Next, the edge extraction method will be described with reference to FIG.

As the edge extraction method, for example, a method such as a Robert filter is used, and instead of this, a Sobel filter can also be used.

When a Robert filter is used and an input image f (i, j) and an output image g (i, j) are used, the output image g (i, j) is calculated from the following equation (9) or (10). Desired.

[0107]

[Equation 9]

[0108]

(Equation 10) In the case of a Sobel filter, the following (11), (12),
It is obtained from the equation (13).

[0109]

[Equation 11]

[0110]

(Equation 12)

[0111]

(Equation 13) In this way, the edge component is extracted by performing the binarization process on the image in which the edge portion is emphasized. The binarization process is performed using an appropriate threshold value.

According to the above method, as shown in FIG. 11, depth information indicated by a circle can be obtained. In addition, B shows a base line length.

Next, processing of depth information obtained from a multi-view stereo image will be described with reference to FIG. FIG. 12 is a diagram showing a flow of processing for integrating depth information from multi-view stereo images.

From each of the stereo images picked up from multiple viewpoints, the depth information of each viewpoint is extracted by the depth information extraction processing from the stereo image of one viewpoint described above, and each depth information is added in order. Integrated.

As shown in FIG. 12, the depth information 120 obtained from the stereo images of the respective viewpoints, together with the position / direction information 125 of the corresponding camera heads, is coordinate system conversion means 1.
21.

The coordinate system conversion means 121 converts the depth information obtained for each viewpoint into information of an arbitrary unified coordinate system based on the corresponding camera head position / direction information 125. As an example of this unified coordinate system, a voxel space of a polar coordinate system is used. By converting the depth information obtained for each viewpoint into information of an arbitrary unified coordinate system, it becomes easy to integrate depth information of multiple viewpoints, which is performed later. Further, as a method of converting the depth information into information of a unified coordinate system, a method of making the Euler angles the same by using affine transformation or the like is used.

The depth information of the unified coordinate system is given to the depth information integrating means 122 together with the occlusion information sent from the occlusion area information sending means 123, and the depth information integrating means 122 excludes the depth information corresponding to the occlusion information. Meanwhile, the process of integrating the depth information of each unified coordinate system is performed. In this integration processing, local deviation information is obtained with respect to the depth information 120 of the subject from at least two or more arbitrary positions, and based on the deviation information, the depth information viewed from the same coordinate system is used for each depth information. A process of determining the point where 120 is located, interpolating between the coordinates of the point, and further determining the majority of the depth information of the overlapping regions among the depth information obtained from at least three or more viewpoints. To do.

The depth information integrated by the depth information integration means 122 is given to the display section 124 together with the occlusion information from the occlusion area information transmission means 123, and the display section 124 displays the integrated depth information and occlusion information. To do.

Next, the processing of the depth information integration means 122 will be described with reference to FIG. 13 is shown in FIG.
It is a figure which shows the content of the majority decision process in the depth information integration means 122 of FIG.

As depth information at each viewpoint, FIG.
As shown in FIG. 3, depth information from three viewpoints is obtained, one of which is depth information indicated by Δ (20000), and the depth information includes information from 2002 to 200222. The other one is ○ (20001)
Is the depth information, and the other one is □ (200
02) is the depth information. In this example, for simplification, the depth information is represented by two-dimensional coordinates of r−θ, and the subject does not change in the φ-axis direction.

Currently, depth information from three viewpoints, Δ (20
000), ◯ (20001), and □ (20002) are obtained, and a plurality of pieces of the information are obtained in the overlapping area 20010.

However, depth information 20020 to 2002
2 may be different compared to information obtained from other perspectives. This may be because, for example, the specular reflection component on the surface of the subject is strong. Generally, when the specular reflection is strong, the brightness intensity is more prominent than others. In such a case, the corresponding point extraction process is affected by the strong luminance information, and the parallax information cannot be accurately obtained. Therefore, as described above, the depth information 2002-2002
A situation occurs in which 2 is different compared to information obtained from other viewpoints.

The depth information 20020 to 20022 is considered to have low reliability, and the depth information 20020 to 2022 is considered to be low.
022 is removed by using the depth information of another viewpoint.
When the viewpoint is changed, a strong luminance component is brought to another place, so that the depth information 20020 to 20022 has moved to the outside of the figure in the sense of another place in the θ axis direction in another place.

From this, a case is adopted in which two or more of the areas 20010 in which the depth information from at least three viewpoints overlaps have the same depth, and the depth information that differs by only one is removed. In this example, the depth information 2
0020 to 20022 are removed. A threshold is used as a criterion for this removal, and depth information that exceeds this threshold is removed.

Unreliable depth information 20020 to 2020
After 022 is removed, the depth information, Δ (2000
0), ◯ (20001), □ (20002) are averaged and the depth information indicated by ☆ (20003) is obtained as the averaged result.

Since the depth information is obtained in the overlapping area 20010, in order to obtain the whole, the operation of obtaining the depth information of the area using the overlapping areas of the other three viewpoints is sequentially performed, and the adverse effect of the specular reflection component is reduced. It becomes possible to obtain the depth information in the removed form.

Next, in addition to the averaging process and one selection process, there is a process for interpolating a local shift, and this process will be described with reference to FIGS. 14 and 15. FIG. 14 is a diagram schematically showing how depth information is integrated from a multi-view stereo image, and FIG. 15 is a diagram showing an intermediate point interpolation method.
In this description, for simplicity of description, it is assumed that information from two viewpoints is handled instead of information from three viewpoints.

In the process of integrating depth information from multi-view stereo images, the depth information obtained at one viewpoint (t) is rt (θ, φ), as shown in FIG. Depth information rt + obtained from the viewpoint (t + δt)
δt (θ, φ) is r′t + δt (θ, φ) when viewed from a unified direction, as shown in FIG.

Next, as shown in FIG. 14C, if the depth information rt (θ, φ) and r′t + δt (θ, φ) are locally shifted (i0, j0), respectively. , FIG. 14
As shown in (d), the depth information rt (θ, φ) and r′t + δt (θ, θ,
φ) is almost superposed.

When the camera posture position detection unit (shown in FIG. 1) is extremely accurate and the depth extraction accuracy from the stereo image is also highly accurate, the local deviation (i0, j0) has a small value. Further, the adverse effect of the specular reflection component has already been removed.

The above shift amount is calculated based on the following equation (14).

[0132]

[Equation 14] Of the Q's in this equation, (i0, j0) that gives the smallest Q is derived, and the depth information is moved with the (i0, j0) as the shift amount. For example, as shown in FIG. rt (θ, φ) and r′t + δt (θ, φ) are almost superposed.

After the superposition, if the same point exists, the same point is excluded and the intermediate point is interpolated.

In the exclusion of the same point, one of the points determined to be the same point is taken out and the other points are eliminated. Therefore, the amount of information can be reduced. The determination as to whether or not they are the same point is made based on whether or not the relationship represented by the following expression (15) or (16) is satisfied.

[0135]

(Equation 15)

[0136]

(Equation 16) However, ε1 and ε2 are reference values, and a, b, c, and d are appropriately selected coefficients. For example, by setting b = c = 1 and a = 2, it becomes possible to more sensitively determine the difference in distance.

Interpolation is performed from the coordinates of the obtained points. This interpolation method will be described with reference to FIG.

As shown in FIG. 15, the depth information rt (θ, φ) is r′t which is (i0, j0) -shifted with a circle.
+ Δt (θ, φ) is represented by a ● symbol, and the depth information rt (θ, φ) and r′t + δt (θ, φ) are projected on the ZX plane which is a one-dimensional plane. Intermediate data represented by □ is obtained by the intermediate interpolation of the depth information rt (θ, φ) and r′t + δt (θ, φ), and this intermediate data becomes new depth information rnew.

As this interpolation method, linear interpolation, spline interpolation or the like can be used.

As described above, the adverse effect from the specular reflection component is removed and the interpolated depth information is obtained. Furthermore, the reliability of the coordinates of the depth information is judged based on the flag from the depth information of the image pickup system. By doing so, it is possible to obtain more reliable depth information. In this process, information with low reliability is excluded based on the focal depth information of the imaging lens. Further, information is selected according to the occlusion information.

Next, the integration of the depth information from the contour and the depth information from the stereo image will be described with reference to FIG. FIG. 16 is a diagram showing the contents of integration processing of the depth information from the contour and the depth information from the stereo image.

In the integration processing of the depth information from the contour and the depth information from the stereo image, as shown in FIG.
Depth information obtained from the contour depth information extraction process # 1 (shown in FIG. 16A) and depth information obtained from the stereo image depth information extraction process # 2 (shown in FIG. 16B). Information and simply combined in some cross section,
With this combination, as shown in FIG. 16C, the depth information (indicated by hatched rectangles) obtained in the process # 1 and the depth information (indicated by hatched white circles) obtained in the process # 2 are shown in the cross section. ) And are simply arranged.

As is apparent from FIG. 16C, it is understood that the depth information 10001 and 10002 are located outside the contour line, and the depth information 10001 and 10002 are obtained.
Since it is considered that 2 is located outside the contour line due to some error, the depth information 10001 and 10002 are not adopted. On the other hand, since the depth information 10003 is located outside the contour line, it is adopted as new depth information, and the depth information 1004 is deleted. This is based on the property that in general, in the depth information from the contour, the convex portion can be properly extracted, but the concave portion cannot be properly extracted.

Therefore, it is possible to remove information located outside the depth information of the contour line, adopt information that can express the concave portion from the depth information of the contour line, and perform processing for connecting the information from the contour and the information from the stereo image. The integrated processing including the adoption of the remaining information to be used is performed, and the processing result is shown in FIG.
As shown in (d), it is possible to obtain smooth depth information that is faithful to the subject.

(Second Embodiment) Next, a second embodiment of the present invention will be described.

The present embodiment has the same structure as the first embodiment, and in this embodiment, a voting process different from that of the first embodiment is performed.

Specifically, in the first embodiment, the voting process of uniformly casting one vote in the voxel as viewed from the inside of the subject is performed, but the voting action from one viewpoint is not necessarily uniform. In the present embodiment, for example, one vote is provided on the side closer to the camera head 1, and two votes and three votes are obtained as the position approaches the pad.
The voting process is performed such that the number of votes cast is weighted according to the distance from the pad, such as voting votes.

By the voting process based on this weighting, the number of votes increased, that is, the number of votes obtained from the subject group is increased. The determination as to whether or not it is a place to vote is performed in the same manner as in the first embodiment.

(Third Embodiment) Next, a third embodiment of the present invention will be described.

The present embodiment has the same configuration as the first embodiment, and in this embodiment, a voting process different from that of the first embodiment is performed.

Specifically, in the first embodiment, one vote is uniformly cast within the voxel as viewed from the inside of the subject, and when it exceeds the set threshold value, the voting process adopted as the vote is performed. The present embodiment adopts a method of performing the processing of the first embodiment step by step. In this method, a voting act is performed for a certain number of viewpoints, and then a threshold is used to select a roughly promising place. Then, votes are cast for the selected range, and this is repeated in stages.

By using the above-mentioned method, the processing in places where there is no prospect is omitted, and the processing speed can be improved.

[0153]

As described above, according to the image information input device of the first aspect, the depth information extraction means puts the depth information into the voxel space from the boundary portion of the subject images picked up from a plurality of viewpoints. The subject is converted into the target image form because the processing for obtaining the subject is performed, the proportions of the subject images existing in the voxel space obtained by the processing are integrated, and the depth information is extracted according to the integration result. It is possible to obtain the depth information that can flexibly correspond to the processing for performing.

According to the image information input device of the second aspect, the depth information extracting means performs the integration of the ratios of the subject images existing in the voxel space by integrating the existence probabilities from the respective viewpoints, Since the depth information that exceeds the set threshold among the integration results is extracted, it is possible to suppress the occurrence of an error due to overcutting as much as possible.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a first embodiment of an image information input device of the invention.

FIG. 2 is a block diagram showing a detailed configuration of the image information input device of FIG.

FIG. 3 is a block diagram showing a detailed configuration of the image information input device of FIG.

FIG. 4 is a block diagram showing the configuration of the system controller of FIG.

FIG. 5 is a diagram conceptually showing depth information extraction processing using polar coordinate voxel space.

FIG. 6 is a diagram conceptually showing a method of extracting depth information from a multi-viewpoint contour image.

FIG. 7 is a diagram conceptually showing a state of voting processing for an 8-viewpoint contour image.

FIG. 8 is a diagram showing a voting situation in the voting process of FIG. 7.

FIG. 9 is a diagram showing a process of extracting depth information from a contour image.

FIG. 10 is a diagram showing point plate matching.

FIG. 11 is a diagram schematically showing calculation of depth information from a stereo image.

FIG. 12 is a diagram showing a flow of processing for integrating depth information from a multi-view stereo image.

13 is a diagram showing the contents of a majority decision process in the depth information integration means 122 of FIG.

FIG. 14 is a diagram schematically showing how depth information is integrated from a multi-view stereo image.

FIG. 15 is a diagram showing an intermediate point interpolation method.

[Fig. 16] Fig. 16 is a diagram illustrating the content of integration processing of depth information from a contour and depth information from a stereo image.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 camera head 4 camera attitude position detection unit 5a, 5b image memory 6 depth information calculation unit 7 two-dimensional image data calculation unit 100L, 100R imaging lens 101L, 101R aperture 102L, 102R image sensor 104L, 104R video signal processing unit 105L, 105R Subject separation unit 220 Image processing unit 250 Recording unit 270 Focus detection unit

 ─────────────────────────────────────────────────── ─── Continued Front Page (72) Inventor Nao Kurahashi 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc.

Claims (2)

[Claims] 1. An image pickup unit for picking up an image of a subject through at least one optical system and outputting the image of the subject, and depth information of the subject is extracted based on the subject image picked up from an arbitrary position by the image pickup unit. In the image information input device including depth information extraction means, the depth information extraction means performs processing for obtaining depth information in a voxel space from a boundary portion of a subject image captured from a plurality of viewpoints,
An image information input device, characterized in that the ratios of the subject images existing in the voxel space obtained by the processing are integrated, and the depth information is extracted according to the integration result. 2. The depth information extracting means integrates the proportions of the subject images existing in the voxel space by integrating the existence probabilities from the respective viewpoints, and is set in the integration result. The image information input device according to claim 1, wherein a depth exceeding a threshold value is extracted as the depth information.