JPH09229648A - Input/output method and device for image information - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image information input/output device which can acquire depth information of an object to be photographed with a high value of utilization from any point of view with a high accuracy. SOLUTION: Among a plurality of pieces of depth information acquired through the second processing, the piece(s) showing more outside than the depth information obtained by the first processing should be removed (10001 of (C) in the attached illustration). The information piece showing inside, i.e., recess, (10003 of (C) in illustration) is adopted, while the depth information due to the first processing concerning approx. the same position (10004 of (C)) is removed. The remaining depth information due to the first processing and the second processing are adopted in such a way that they are tied together, as (C) and (D) in illustration.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a computer graphics (CG) from an image obtained by picking up an image of a subject.
The present invention relates to an image information input / output device that extracts three-dimensional shape information used for various purposes such as synthesizing with a CAD (cad) or a document file.

[0002]

2. Description of the Related Art Conventionally, methods for obtaining a three-dimensional shape of a subject are roughly classified into a passive method and an active method.

A typical passive method is a so-called stereo image method, and this method uses two cameras to perform triangulation. That is, in each image obtained by the left and right cameras, a location where the same object point is found is searched, and the three-dimensional position of the object point is measured from the displacement amount of the position.

On the other hand, typical active methods include an optical radar type range finder for measuring the time until the projected light returns and a slit-shaped optical pattern for projection. There is a slit light projection method or the like that measures a three-dimensional shape from the displacement of the pattern shape reflected on the subject.

By the way, conventionally, it is generally practiced to output / reproduce an image of a subject viewed from an arbitrary viewpoint on a two-dimensional display or the like based on the obtained three-dimensional information of the subject.

With the widespread use of personal computers, it has become possible to capture and edit images taken by electronic cameras. For example, it is already possible to import a landscape image captured by an electronic camera into a plurality of images into a personal computer and arbitrarily process and process the image with application software.

From such a background, it is necessary to acquire highly accurate three-dimensional shape information for various objects according to the purpose. For example, image information generally handled in an office or the like is often output on paper in the end, and the image form that is required to be used is not only a natural image but also a line drawing that represents a subject only with a contour line. . Therefore,
It is desired to capture image information of a subject from all sides (viewpoints) so that it can be freely processed and processed.

[0008]

However, in passive techniques such as, for example, stereo imaging, the measurement information is discrete and may contain noise. Moreover, since the main purpose is to calculate distance information from a specific position, it is not always sufficient to measure the three-dimensional shape itself of the subject.
Therefore, it is difficult to obtain highly accurate and smooth three-dimensional shape information.

On the other hand, in the active method, since it is necessary to irradiate the subject with a laser beam, there is a certain limitation on the type of the subject to be measured, and the measurement itself is complicated and not easy.

Further, in both the passive method and the active method described above, the distance can be measured with high accuracy from an arbitrary viewpoint by flexibly supporting a dynamic imaging method in which an image is taken while moving around the subject. It was difficult to perform (hereinafter also referred to as “depth information extraction”).

Therefore, it is possible to obtain highly accurate depth information from an arbitrary viewpoint without being restricted by the type of subject,
There was room for improvement in the purpose of using it.

The present invention has been made in order to solve the above-mentioned problems of the prior art, and an object thereof is an image capable of obtaining highly accurate and highly useful depth information from an arbitrary viewpoint for any object. To provide an information input / output device.

[0013]

In order to achieve the above object, the image information input / output device according to claim 1 of the present invention is an image information input / output device for capturing an image of a subject and obtaining depth information of each part of the subject. An image of the subject is projected to a plurality of viewpoints, contour measuring means for obtaining depth information of each part of the subject based on contours of the projected plurality of images, and the subject is imaged from two viewpoints, Stereo measuring means for obtaining depth information of each part of the subject from two images of the subject captured from two viewpoints using parallax of the two viewpoints, depth information obtained by the contour measuring means and the stereo And an integration unit that obtains the depth information of each part of the subject based on the depth information obtained by the measuring unit.

In order to achieve the same object, the image information input / output device according to a second aspect of the present invention is the image information input / output device according to the first aspect, wherein the integrating means obtains the contour measuring means for each part of the subject. It is characterized by further comprising a selection means for selecting the depth information obtained and the depth information obtained by the stereo measurement means.

In order to achieve the same object, an image information input / output device according to a third aspect of the present invention is the image information input / output device according to the second aspect, wherein the selection means is the contour measuring means for substantially the same portion of the subject. The depth information obtained by the above and the depth information obtained by the stereo measurement means are removed from the outside.

In order to achieve the same object, an image information input / output method according to a fourth aspect of the present invention is an image information input / output method for picking up an image of a subject and obtaining depth information of each part of the subject. A contour measuring step of projecting to a plurality of viewpoints and obtaining depth information of each part of the subject based on the contours of the projected plurality of images, and the subject is imaged from two viewpoints and imaged from the two viewpoints. The stereo measurement step of obtaining depth information of each part of the subject from the two images of the subject using the parallax of the two viewpoints, the depth information obtained in the contour measurement step, and the stereo measurement step An integration step of obtaining depth information of each part of the subject based on the depth information.

In order to achieve the same object, an image information input / output method according to a fifth aspect of the present invention is such that the image of a subject is projected onto a plurality of viewpoints and coordinate information relating to the outside of the contours of the projected plurality of images. Is removed from the spatial coordinate information expressing each coordinate of the space including the subject, so that the three-dimensional shape of the subject is grasped from the coordinate information remaining in the spatial coordinate information.

Here, the "depth" means the distance of each part of the subject from the viewpoint.

[0019]

Embodiments of the present invention will be described below with reference to the drawings.

An image information input / output device according to an embodiment of the present invention will be described below.

First, the configuration of the image information input / output device will be described with reference to FIGS. 2 to 4, the depth information extraction processing will be described with reference to FIGS. 5 to 13, and finally FIG. The depth information integration process will be described.

FIG. 2 is a block diagram showing an outline of the image information input / output device according to the present embodiment and its surrounding environment.

In the figure, in the image information input / output device 20, in addition to the image pickup head unit (camera head) 1, a camera posture position detection unit 4, an image memory 5, depth information calculation units 6 and 2 are provided.
A processing unit 21 including the three-dimensional image data calculation unit 7 is included.

The image pickup head unit 1 picks up an image of the subject 2 to be detected while moving in an arc shape from the image pickup start position A0 to the image pickup end position An. Left imaging lens 10
The imaging range of 0L is a circular arc 10L, and the imaging lens 10 on the right side
The imaging range of 0R is indicated by the arc 10R. Lighting unit 2
00 illuminates the subject 2 with illumination light according to the imaging environment. In addition, a pad (hereinafter referred to as “back surface”) is provided behind the subject 2.
3) is placed.

The camera posture position detection unit 4 calculates angle information by a sensor such as a gyro and also calculates posture and position information by image processing based on the information obtained from the back surface 3 to determine the posture / position of the image pickup system. Detect the position.

The information on the position at each image pickup point between A0 and An by the image pickup head unit 1 is sent from the camera posture position detection unit 4 to peripheral devices and the like.

The image memory 5 stores the image information obtained by the image pickup head 1 and the position information of the image pickup head unit 1 obtained by the camera posture position detection unit 4.

The depth information calculator 6 calculates the three-dimensional shape of the subject 2 based on the image information and the position information stored in the image memory 5.

The two-dimensional image data calculation unit 7 calculates the three-dimensional information about the three-dimensional shape of the subject 2 calculated as above.
Two-dimensional image information in the image form specified by the user is calculated for the subject 2 viewed from an arbitrary viewpoint.

Next, the processing when using this apparatus will be briefly described.

First, the user attaches the imaging head unit 1 to the subject 2
When a release button (not shown) is operated, the subject 2 is imaged and the first image information is stored in the image memory 5.

Next, the user attaches the imaging head unit 1 to the subject 2
The center is moved from the position A0 to the position An. Position A0
When the camera posture detection unit 4 detects that the position and the imaging direction of the imaging head unit 1 have changed from the position A0 by a predetermined amount while moving from A to An, the second imaging is performed. In the same manner, imaging is sequentially performed up to the nth time.

For example, if images are taken at angular intervals of 3 °, a total of 60 images will be taken when the positions A0 to An are 180 °. The setting of the angular interval is determined from the performance of the gyro and the resolution of the depth information to be obtained. For example, if the detection capability of the gyro is 90 ° / sec, 3 ° / frame speed is set.

The image information of the subject 2 is stored in the image memory 5 together with the amount of displacement of the position of the image pickup head unit 1 with respect to the position A0 and the amount of displacement in the image pickup direction obtained by the camera posture position detector 4. Then, when at least one of the displacement amount of this position and the displacement amount of the direction is larger than a predetermined amount, a warning is given.

This image pickup operation is repeated several times, and the object 2
When a sufficient number of pieces of image information for calculating the depth information are obtained, the user is notified of that fact by an imaging end detector or the like (not shown), and the imaging ends.

Next, the depth information calculation unit 6 calculates the three-dimensional image information of the subject 2 from the image information stored in the image memory 5 and the position information of the image pickup head unit 1 corresponding to each image data. . Two-dimensional image data calculation unit 7
Is the three-dimensional image of the subject 2 in the image form designated by the operation unit 11 connected to the image information input / output device 20, and the two-dimensional image information when the subject 2 is first photographed from the position A0. It is calculated based on the information and displayed on the monitor 8 connected to the operation unit 11.

When the user operates the operation unit 11 and causes the two-dimensional image data operation unit 7 to perform an operation according to the operation, the two-dimensional image information is viewed from an arbitrary viewpoint of the subject 2. You can change the information. Further, by operating the operation unit 11, the image form of the subject 2 displayed on the monitor 8 can be changed.

The printer 9 is connected to the processing section 21 and the operation section 11 and prints out various information. Further, the data composition unit 1000 connected to the image information input / output device 20 and the text data creation unit 1001 includes the image information and text data creation unit 10 sent from the image information input / output device 20.
The document file created and sent by 01 is combined.

FIG. 3 is an overall configuration diagram of the image information input / output device according to the present embodiment, and shows the components of the image pickup head unit 1 and the processing unit 21 of FIG. 2 in detail.

In the figure, the same elements as those in FIG. 2 are designated by the same reference numerals.

The system controller 210 includes an illumination unit 200, a camera posture position detection unit 4, and a zoom control unit 106.
L, 106R, focus control units 107L, 107R,
Aperture control units 108L and 108R, sounding body 97, image sensor drivers 102L1 and 102R1, video signal processing units 104L and 104R, and subject separation units 105L and 105.
R, focus detection unit 270, image processing unit 220, release button 230, external input I / F (interface) unit 76
0, the overlap detector 92 is connected, and the system controller 210 controls all of them.

When the release button 230 is pressed, the optical image from the subject is captured by the image pickup lens 1 including a zoom lens.
00L, 100R, then apertures 101L, 101R, and an image sensor 102L using a CCD or the like.
2, 102R2 exchanges electrical signals. The image sensors 102L2 and 102R2 are connected to the image sensor drivers 102L1 and 102R, respectively.
1 is controlled via

The electric signal is sent to the image sensor 102L.
2, 102R2 connected to the A / D converter 103L, 1
The analog signal is converted to a digital signal by 03R and stored in the memory 83 and the memory 85 connected to the A / D conversion unit 103L, and the memory 73 and the memory 75 connected to the A / D conversion unit 103R.

The digital signals stored in the memories 83 and 85 and the memories 73 and 75 are converted by the video signal processing section 104L and the video signal processing section 104R into video signals composed of luminance signals and chrominance signals of appropriate forms.

The converted video signal is processed by the subject separation unit 105 connected to the video signal processing units 104L and 104R.
The signals of the subject 2 and the signal of the rear surface 3 which are to be extracted as three-dimensional shape information are separated by L and 105R.

Here, the separation is performed as follows, for example.

The rear surface 3 is imaged in advance and its image information is held in a memory, and then the object 2 is arranged in front of the rear surface 3 and the object 2 is imaged together. Then, only the back surface area is separated by performing matching processing and difference processing with the image of only the back surface 3 held in the memory. In addition,
The separation method is not limited to this, and separation may be performed based on color or texture information.

The image signal of the subject 2 separated in this way is extracted by the image processing unit 220 connected to the subject separating units 105L and 105R to extract depth information based on the parameters at the time of image pickup (hereinafter referred to as "first process"). ", Which will be described later in detail), and is recorded in the recording unit 250.

On the other hand, the video signal processing units 104L and 104R
After the video signal including the luminance signal and the color signal obtained in step 1 is subjected to depth information extraction processing using parallax (hereinafter referred to as “second processing”, details will be described later) by the image processing unit 220. Are recorded in the recording unit 250.

Then, the depth information obtained by the first processing and the second processing is integrated by selecting it in the image processing section 220 under the control of the system controller 210, and the final depth information of the entire subject 2. (See below).

Focusing is performed by a focus detecting section 270 connected to the object separating sections 105L and 105R.

The focal length is set as follows.

First, the distance information is represented by the following mathematical formula 1.

[0054]

Z = fb / d where Z is the distance between the viewpoint and the subject 2, f is the focal length, b is the baseline length, and d is the parallax between the two viewpoints. Considering the distance separability determined by parallax as a parameter, the following Equation 2 is obtained.

[0055]

[Formula 2] ∂Z / ∂d = −fb / d ² Therefore, the focal length is given by the following mathematical formula 3.

[0056]

[Formula 3] f = (-d ² / b) · (∂Z / ∂d) Further, the resolution is set from a computer or the like via the external input I / F 760, and the focal length f is set based on this value. It is also possible to do so.

In addition, the display unit 240 is connected to the video signal processing units 104L and 104R and outputs and displays various image information.

The overlap detecting section 92 is connected to the memories 73, 75, 83 and 85.

FIG. 4 is a block diagram showing the configuration of the system controller 210. As shown in the figure, the system controller 210 is a microcomputer 900.
In addition, an image calculation processing unit 920 that performs calculation processing of image information.
And a memory 910 for storing image information are connected by a bus.

The first process will be described below with reference to FIGS.

FIG. 5 is a diagram for explaining the first process in this embodiment.

In the figure, reference numeral 1120 denotes a voxel space in which the space is divided at predetermined coordinate intervals, and three-dimensional coordinates (X, Y,
Each voxel is represented by Z). Voxel space 11
Reference numeral 20 includes the entire spherical object 2, and the position information of each part on the surface of the object 2 is represented by three-dimensional coordinates as depth information from the principal point 1110 of the camera.

Reference numeral 1100 denotes a background separation image obtained by subjecting the obtained image to the separation processing of the object 2 and the back surface 3 by the above-described method, and the object image 1101 and the background image 1
And 102.

Here, it is determined that the background separated image 1100 is a portion representing the subject 2 or a portion representing the back surface 3 in each portion of the image, and, for example, the portion representing the subject 2 after the separation processing. This can be obtained by adding the flag "1" to the image information of "1" and adding the flag "0" to the image information of the portion representing the back surface 3.

The line segments 1130 to 1134 pass through the principal point 1110 from the outline of the separated subject image and contact the subject 2. Points P1, P2, P3, P on the surface of subject 2
4 and P5 are points p1 and p on the contour of the subject image 1101.
2, p3, p4, p5 corresponding to line segment 113
It is connected by 0,1131,1132,1133,1134.

The first processing is performed by removing the outside of the contour of the subject 2 from the voxel space 1120. That is, the outer voxel data is divided into a large number of line segments. This means that, for example, the voxels outside the line segments 1130 to 1134 are treated as not the subject portion, and the three-dimensional shape formed only by the inner voxels is regarded as the subject 2 shape. A specific process is to set a flag for each voxel. For example, if the flag is “0”, the coordinate information of that voxel is set to the voxel space 11
It is performed by removing from the space coordinates of 20 and adopting the flag "1".

FIG. 6 schematically shows the coordinate information removing process by the first process, and is a sectional view of the subject 2. In the figure, for ease of explanation, a case is shown in which the camera for imaging the subject 2 is moved two-dimensionally and processing is performed within the same plane.

In the figure, reference numeral 1210 in (a) is a part of the voxel space (captured in a plane), and the subject 2 as a plane body is present therein.

In this figure, there are four viewpoints to be imaged, and the main points 1200 to 1203 of each camera correspond to it.

The image of the subject 2 has main points 1200-120.
3 through the cross section 1240 of the image sensor 120L2, etc.
To 1243.

Line segments 1220 to 1 that are in contact with the contour of the subject 2
227 is an inner boundary line of the subject 2. Line segment 122
0,1221 passes through the main point 1200, and the line segment 1222,1
223 passes through the main point 1201 and is divided into line segments 1224 and 1225.
Passes through the principal point 1202, and the line segments 1226 and 1227 pass through the principal point 1203.

Here, when the outer part of the subject 2 is removed from the part 1210 of the voxel space by the line segments 1200 to 1207, the intersection points a, b, c, d, e, shown in FIG.
A region 1230 surrounded by f, g, and h remains inside the boundary line. In (b) of the figure, the region subjected to this processing is extracted, and the true region 1231 of the subject 2 is shown as being overlapped.

By performing such processing three-dimensionally,
The three-dimensional contour of the subject 2 can be grasped.

Note that such processing can be performed three-dimensionally by viewing the projected image from the voxel space side. That is, for example, a line segment that passes through the principal point 1110 is calculated from a certain coordinate in the voxel space, and where the point relating to the coordinate is projected on the background separated image 1100 is calculated. A flag may be used to determine whether the image 1101 or the background image 1102 is displayed. Then, this processing is performed for each point from the point (0, ₀ , 0) to (X ₀ , Y ₀ , Z ₀ ) in the voxel space.

This processing is performed by the following arithmetic processing.

First, the coordinates of the principal point 1110 are (a ₀ , b ₀ ,
c ₀ ), and the projection point on the background separated image 1100 corresponding to the voxel space coordinates (X, Y, Z) is (x, y), the projection point (x, y) is given by , Is calculated by the following mathematical formulas 5 and 6.

[0077]

(Equation 4)

[0078]

## EQU5 ## x = fX '/ Z'

[0079]

Y = fY ′ / Z ′ where ω is the twist angle in the direction of rotation from the Y axis to the Z axis with respect to the X axis, and θ is the Y axis with respect to the Z axis to the X axis. Is the twist angle in the direction of rotation, and φ is
With respect to the Z axis, it is the twist angle in the direction of rotation from the X axis to the Y axis direction. These twist angles ω, θ, and φ are parameters indicating the twist relationship between the coordinate system of the principal point 1110 of the camera and the coordinate system of the voxel space 1120.

Next, by referring to the flag of the projection point (x, y), it is judged whether or not the voxel belongs to the inside of the subject 2. When the values of x and y are not integers, the flag is judged from the neighboring area.
Then, the depth information of the subject 2 is extracted into the voxel space 1120 by repeating this processing from multiple viewpoints. The process of this process is shown in FIG. In the figure, the same elements as those in FIG. 5 are designated by the same reference numerals.

By the first processing described above, the contour of the subject 2 can be grasped.

Below, with reference to FIG. 8 to FIG.
The processing will be described.

FIG. 8 is a block diagram showing a procedure for extracting the depth information of the subject 2 by the second processing.

This processing is to obtain depth information from a so-called stereo image. The stereo image 110 is stored in the image memory 5 and includes a left image and a right image.

The stereo image 110 includes the edge extraction unit 11
1 is input, and an edge image in which edges have been extracted is generated by the method described later. With respect to this edge image, the edge image corresponding point extraction unit 113 determines how each pixel in the left and right images has a corresponding relationship, and the information on the corresponding relationship is output to the contradiction elimination processing unit 114. To be done.

On the other hand, the stereo image 110 is also input to the stereo image corresponding point extraction unit 112, and it is determined from the brightness value of the stereo image 110 how each pixel corresponds to each other by a method described later. Then, the information on the correspondence is output to the contradiction elimination processing unit 114.

In the contradiction elimination processing unit 114, the correspondence relationship obtained by the edge image corresponding point extraction unit 113 and the correspondence relationship obtained by the stereo image corresponding point extraction unit 112 are compared to determine whether or not there is a contradiction. To determine. Then, when there is a contradiction between the two, the information is excluded because the correspondence relationship has low reliability. In the judgment at this time,
Both may be appropriately weighted.

The processing result of the contradiction elimination processing unit 114 is output to the occlusion area determination processing unit 115. Here, the occlusion area is determined based on the locations of the corresponding points that have already been obtained and an index that indicates the degree of correlation used in the course of obtaining the corresponding points, such as a residual. this is,
Although a corresponding result is obtained by the corresponding point extraction process, it is performed to further improve the reliability.

A correlation coefficient or a residual is used as an index indicating the degree of correlation. For example, when the residual is very large or the correlation coefficient is low, it is determined that the reliability of the correspondence is low. Then, this low-reliability portion is treated as an occlusion area or an area having no correspondence.

In the depth information distribution calculation processing unit 116, the depth information of the subject 2 is calculated based on the obtained correspondence relationship by the principle of triangulation. The triangulation is performed by using the above formula 1.

The stereo image 110 is also input to the feature point extraction unit 117, and the feature points in the background portion are identified. The processing result is output to the correction data calculation processing unit 118, and the imaging parameter, the posture, and the movement relationship are acquired by using the feature points of the background portion. At that time, gyro 880
Information such as an angle change due to the correction data calculation processing unit 118
Is output to

Here, the stereo image corresponding point extraction unit 1
The template matching method will be described as a representative of the corresponding point extraction processing method of 12.

FIG. 9 is a diagram for explaining the template matching method using the stereo image 112.

As shown in the figure, for example, a template image of N × N pixels is cut out from the left image. Then, this is set as a search region (M
-N + 1) Move up ² . Further, the position of the template image that minimizes the residual R (a1, b1) is obtained based on the following formula 7, and the central pixel of the template image of N × N pixels is obtained as the matching point.

[0095]

(Equation 7) Here, (a1, b1) indicates the upper left position of the template image of N × N pixels, I _{R (a1, b1)} (i, j) is a partial image of the right image, and T _L (i, j) is , Corresponding to the template image cut out from the left image.

Next, the edge extraction method by the edge extraction unit 111 will be described. For the edge extraction, a Sobel filter can be used in addition to a technique such as a Robert filter.

According to the Robert filter, the input image is f
If (i, j), g (i, j) of the output image is calculated by the following formula 8 or formula 9.

[0098]

G (i, j) = [{f (i, j) -f (i + 1, j + 1)} ² + {f (i + 1, j) -f (i, j + 1) ) ² ] ^1/2

[0099]

G (i, j) = | {f (i, j) -f (i + 1, j + 1)} | + | {f (i + 1, j) -f (i, j + 1)} | According to the Sobel filter,
After obtaining the matrices fx and fy by
Thus, the index θ ₁ representing the direction of edge strength is calculated.

[0100]

(Equation 10)

[0101]

[Equation 11]

[0102]

[Mathematical formula-see original document] [theta] ₁ = tan < ^-1 > (fy / fx) In this way, binarization processing is performed on the image in which the edge portion is emphasized to extract the edge component. The binarization process is performed using an appropriate threshold value.

FIG. 10 shows the depth information obtained based on the stereo image by the above processing.

In the figure, the same elements as those of FIG. 5 are designated by the same reference numerals and the description thereof will be omitted. In the figure, points Q _{1 to} Q
₇ indicates a point for which depth information is obtained, and points q _{1 to} q
₇ shows a projected point in the background separation image 1100 corresponding to each point Q ₁ to Q _7. In addition, the distance between the centers of the two background separated images obtained from the left and right images is defined as the baseline length B.

Next, the integration processing of depth information obtained from stereo images from multiple viewpoints will be described with reference to FIGS. 11 to 13.

FIG. 11 is a block diagram showing the procedure of depth information integration processing in the second processing.

[0107] For example, the first view point t by the depth information Z ^t shown in the figure, then and so depth information Z ^{t + .DELTA.t} according to a second viewpoint t + .DELTA.t, 1 from the pair of stereo images, depth information 120 from time to time Generated moment by moment.

The depth information from each viewpoint is input to the coordinate system conversion unit 121 and converted into an arbitrary unified coordinate system by the method described later using the position information input from the camera posture position detection unit 4. To be done. As the unified coordinate system, for example, a coordinate system similar to the voxel space as shown in FIG. 5 may be adopted. To convert to a unified coordinate system,
Use the affine transformation or the like to make the Euler angles the same. By performing such conversion, it is possible to easily integrate the depth information obtained from multiple viewpoints.

The depth information integrated into the unified coordinate system is integrated by the depth information integration unit 122 using the information from the occlusion area information transmission unit 123. Details of the processing will be described later.

The "integration" referred to here is obtained by obtaining information relating to each local deviation amount with respect to the depth information 120 of the subject 2 from at least two or more arbitrary viewpoints. Based on, it means that it is the depth information viewed from the same coordinate system, it is determined that the same point is the same,
It is performed by interpolating between the obtained coordinates of each point.

The integrated depth information and the information from the occlusion area information sending section are displayed on the display section 124.

Next, the integration processing by the depth information integration unit 122 will be described with reference to FIG. (A) of the figure shows the first
The depth information Z ^t (X, Y) obtained from the viewpoint t of FIG.

[0113] (b) the depth information obtained by the following second aspect ^{t + δt Z t + δt (} X, Y) depth information when the, viewed from unified direction Z ^{'t +} δt (X , Y).

(C) is a drawing of (a) and (b), showing depth information Z ^t (X, Y) and ^{Z't + δt} (X,
It is locally deviated from (Y) by (i0, j0).

(D) is depth information Z'with respect to (c).
It is drawn by shifting ^{t + δt} (X, Y) by (-i0, -j0). The depth information Z ′ ^{t + δt} (X, Y) shifted is drawn by a dotted line.

Here, the quantities i0 and j0 that determine the shift amount are derived as follows.

When the detected brightness by the camera posture position detection unit 4 and the depth extraction accuracy from the stereo image are highly accurate, the deviation amount Q (i, j) has a small value. Deviation amount Q
(I, j) is calculated by the following Expression 13.

[0118]

(Equation 13) The quantities i, j that minimize the deviation Q (i, j) are obtained, and are defined as the quantities i0, j0.

Shifted by this quantity i0, j0,
From the overlapped depth information as shown in (d), the same point is excluded and the intermediate point is interpolated as described below.

First, the same point exclusion processing will be described.

In this processing, if there is information indicating the same point of the subject 2 in the mutual depth information, either one is adopted and the other is excluded. This also has the significance of reducing the amount of information. Whether or not the compared information indicates the same point is determined by the following formula 14 or formula 15,
It is determined that they are the same when either one of the expressions is satisfied.

[0122]

(14) (x0-x1) ² + (y0-y1) ² + (z0
-Z1) ² <ε1

[0123]

## EQU15 ## a2 (x0-x1) ² + b2 (y0-y1) ²
+ C2 (z0-z1) ² <ε2 where ε1 and ε2 are predetermined reference values, and a2 and b
2, c2 are coefficients. For example, a2 = b2 = 1, and c
By setting 2 = 2, it is possible to more sensitively judge the difference in distance.

Next, the intermediate point interpolation processing will be described with reference to FIG.

In the same figure, in order to simplify the explanation, the depth information of (d) of FIG. 12 is projected on the ZX plane and shown in one dimension.

The white circle in (a) of the figure is the depth information Z.
^t (X, Y), and black circles represent depth information Z ′ ^{t + δt} (X, Y
Depth information Z ′ ^{t + δt obtained} by shifting (Y) by (i0, j0)
(X + i0, Y + j0) is shown.

The interpolation performed here is, for example, an intermediate point interpolation that adopts an intermediate point from mutual depth information. The new depth information Znew obtained by the midpoint interpolation is (b)
Is indicated by a white square.

As the interpolation method, linear interpolation, spline interpolation or the like can be used.

Next, the reliability of each depth information Znew is judged based on the flag obtained from the depth information of the image pickup system. For example, the depth information Znew having low reliability is eliminated by using the depth-of-focus information of the optical system of the imaging system.

Further, the occlusion area information sending section 1
Based on the detection information from 23, each depth information Znew is selected.

The three-dimensional shape of the subject 2 can be grasped by the second processing described above. In particular, there is an advantage that it is easy to obtain depth information about the concave portion of the subject 2.

The process of integrating the depth information obtained in the first process and the depth information obtained in the second process described above will be described below with reference to FIG.

FIG. 1 is a diagram for explaining an integration process by depth information selection in the image information input / output device according to this embodiment.

FIG. 15A is a schematic diagram of the first processing, which is a simplified drawing of FIG. Also,
(B) is a schematic diagram of the second processing, and is a more simplified drawing of FIG. 10.

In order to simplify the description of this integration process, a specific section will be described. In (A) and (B), the specific cross section is shown by hatching, and (C) and (D) show the specific cross section viewed from above (from the Y-axis direction to the central point direction).

(C) is a display in which the shaded squares showing the output result of the first process and the shaded white circles showing the output result of the second process are simply combined and displayed in an overlapping manner. The shaded white circles correspond to the white circles in (B).

This integration processing is performed in the following (1), (2),
Perform according to the procedure of (3).

(1) Of the depth information obtained by the second processing,
Those outside the depth information obtained by the first process are removed.

(2) Of the depth information obtained by the second processing,
The inside of the depth information obtained by the first processing, that is, the one that can express the concave portion is adopted, and the depth information obtained by the first processing regarding substantially the same portion is removed.

(3) The remaining depth information by the first and second processes is adopted so as to connect the two.

For example, (C) depth information 10001,
Focusing on 10002, regarding the part related to this information, the depth information 100 is larger than the depth information obtained by the first processing.
It can be seen that 01, 10002 indicates the outside (projects). Since this phenomenon is considered to be caused by some error, the depth information 10001,100
02 is not adopted and is removed.

On the other hand, like the depth information 10003, the information indicating the inside (recessed) of the depth information by the first process is adopted, and the depth information 10004 by the first process for the same part is removed.

The selection is made in consideration of the characteristics of the first processing and the second processing. That is, in the first process, the extraction accuracy of the convex part of the subject is generally high, and it is difficult to extract the concave part, while in the second process, the concave part of the subject is generally easy to extract. Therefore, as a general rule, by using the result of the first processing for the convex portion and the result of the second processing for the concave portion, highly reliable depth information can be obtained.

The result of such integration processing is shown in (D). By performing this processing three-dimensionally, it is possible to obtain smooth depth information that is faithful to the three-dimensional shape of the subject 2.

Regarding the processing of the recess, in the above example,
The case where the two pieces of depth information 10003 are adjacent to each other has been described, but when the two pieces of information are apart from each other, a method of connecting the information is devised. For example, when the distance between both pieces of information is 2 pixels (cells) or less, they are connected, and when the information is 3 pixels or more, the depth information of the first process is used to connect them.

By connecting in this way, more faithful and smooth depth information can be obtained.

The procedure of the integration process is not limited to the above (1), (2) and (3). For example,
Instead of the procedure (3), the depth information of the first process and the depth information of the second process may be appropriately weighted and interpolated.

As a result, even if the depth information of the second process is discrete (split), it is possible to ensure smooth acquisition of depth information.

Further, as the procedure of the integration process, the following procedure may be adopted instead of (2) among (1), (2) and (3) described above. That is, of the depth information obtained by the second processing, the inside of the depth information obtained by the first processing,
That is, when there is something that can represent the concave portion, the vicinity of the midpoint between the two is adopted as the depth information of the portion. Thereby, the depth information can be made smoother.

As described above, according to the image information input / output device of this embodiment, the depth information of the subject 2 can be obtained by both the first process and the second process, and the characteristics of both can be obtained. By integrating the information in consideration of, it is possible to obtain final depth information that is faithful to the three-dimensional shape of the subject 2 and is smooth. Also, with highly accurate depth information from all sides, it is possible to obtain a highly usable image form according to the purpose.

In the present embodiment, the first process and the second process are performed using the orthogonal space coordinates.
The present invention is not limited to this, and it may be performed using curvilinear coordinates such as polar coordinates or cylindrical coordinates.

[0152]

As described above, according to the first aspect of the present invention.
According to the image information input / output device of the present invention, the image of the subject is projected onto the plurality of viewpoints by the contour measuring means, depth information of each part of the subject is obtained based on the contours of the projected plurality of images, and the stereo is obtained. The measuring means images the subject from two viewpoints, depth information of each part of the subject is obtained from two images of the subject captured from the two viewpoints by using parallax of the two viewpoints, and the contour is obtained. Since the depth information of each part of the subject is obtained by the integration unit based on the depth information obtained by the measuring unit and the depth information obtained by the stereo measuring unit, high accuracy from any viewpoint is obtained for all subjects. It is possible to obtain smooth depth information and increase the utility value of three-dimensional shape information.

According to the image information input / output device of the second aspect of the present invention, the depth information obtained by the contour measuring means and the depth information obtained by the stereo measuring means are discarded for each part of the subject. Since depth information is selected by the selection means, it is possible to obtain more reliable depth information that is faithful to the subject shape.

According to the image information input / output device of the third aspect of the present invention, the depth information obtained by the contour measuring means and the depth information obtained by the stereo measuring means for substantially the same part of the subject are displayed. Outer ones are removed by the selection means, so that it is possible to obtain more reliable depth information that is faithful to the subject.

According to the image information inputting / outputting method of the fourth aspect of the present invention, the image of the subject is projected at a plurality of viewpoints in the contour measuring step, and each of the subject is projected based on the contours of the projected plurality of images. Depth information of the part is obtained, the subject is imaged from two viewpoints in a stereo measurement step, and each part of the subject is calculated from two images of the subject captured from the two viewpoints using parallax of the two viewpoints. Depth information of each object is obtained in the integration step based on the depth information obtained in the contour measurement step and the depth information obtained in the stereo measurement step. With respect to, it is possible to obtain highly accurate and smooth depth information from an arbitrary viewpoint, and to enhance the utility value of the three-dimensional shape information.

According to the image information input / output method of the fifth aspect of the present invention, the image of the subject is projected onto a plurality of viewpoints, and the coordinate information on the outside of the contours of the projected plurality of images indicates the subject. Since the three-dimensional shape of the subject is grasped from the coordinate information left in the spatial coordinate information by removing the three-dimensional shape of the subject from the spatial coordinate information expressing each coordinate of the space including the three-dimensional shape of the subject, Easy to understand in the configuration.

[Brief description of drawings]

FIG. 1 is a diagram for explaining integration processing by selecting depth information in an image information input / output device according to an embodiment of the present invention.

FIG. 2 is a block diagram showing an outline of an image information input / output device according to the same embodiment and its surrounding environment.

FIG. 3 is an overall configuration diagram of an image information input / output device according to the same embodiment.

FIG. 4 is a block diagram showing a configuration of a system controller according to the apparatus.

FIG. 5 is a diagram for explaining a first process in the same embodiment.

FIG. 6 is a sectional view of a subject relating to coordinate information removal processing by the first processing.

FIG. 7 is a diagram showing a process of a first process.

FIG. 8 is a block diagram showing a procedure for extracting depth information of a subject by the second processing.

FIG. 9 is a diagram for explaining a template matching method in the second processing.

FIG. 10 is a diagram for explaining a second process.

FIG. 11 is a block diagram showing a procedure of depth information integration processing in the second processing.

FIG. 12 is a diagram for explaining integration processing by a depth information integration unit in the second processing.

FIG. 13 is a diagram for explaining an intermediate point interpolation process in the second process.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Imaging head section 2 Subject 3 Rear surface 4 Camera attitude position detection section 5 Image memory 6 Depth information calculation section 7 Two-dimensional image data calculation section 100R Imaging lens 100L Imaging lens 102R2 Image sensor 102L2 Image sensor 104R Video signal processing section 104L Video signal processing Unit 105R Subject Separation Unit 105L Subject Separation Unit 210 System Controller 220 Image Processing Unit 900 Microcomputer 920 Image Calculation Processing Unit 116 Depth Information Distribution Calculation Processing Unit 122 Depth Information Integration Unit

 ─────────────────────────────────────────────────── ─── Continuation of front page (72) Inventor Motohiro Ishikawa 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc.

Claims (5)

[Claims] 1. An image information input / output device for capturing an image of a subject to obtain depth information of each part of the subject, wherein the image of the subject is projected onto a plurality of viewpoints, and the image is projected based on the contours of the plurality of projected images. A contour measuring unit that obtains depth information of each part of the subject, and the subject is imaged from two viewpoints, and the two images of the subject captured from the two viewpoints are used to detect the subject using the parallax of the two viewpoints. Stereo measurement means for obtaining depth information of each part, and integration means for obtaining depth information of each part of the subject based on the depth information obtained by the contour measurement means and the depth information obtained by the stereo measurement means. An image information input / output device comprising: 2. The integrating means comprises a sorting means for sorting depth information obtained by the contour measuring means and depth information obtained by the stereo measuring means for each part of the subject. The image information input / output device according to claim 1, characterized in that: 3. The selection means removes the depth information obtained by the contour measurement means and the depth information obtained by the stereo measurement means, which are outside, with respect to substantially the same part of the subject. The image information input / output device according to claim 2. 4. An image information input / output method for capturing an image of a subject and obtaining depth information of each part of the subject, wherein the images of the subject are projected onto a plurality of viewpoints, and the image is projected based on the contours of the plurality of projected images. A contour measuring step of obtaining depth information of each part of the subject, and the subject is imaged from two viewpoints, and from the two images of the subject imaged from the two viewpoints, the parallax of the two viewpoints is used to detect the subject. A stereo measurement step of obtaining depth information of each part, and an integration step of obtaining depth information of each part of the subject based on the depth information obtained by the contour measurement step and the depth information obtained by the stereo measurement step. An image information input / output method comprising: 5. An image of a subject is projected to a plurality of viewpoints, and coordinate information outside the contours of the plurality of projected images is removed from spatial coordinate information representing each coordinate of a space including the subject. According to the image information input / output method, the three-dimensional shape of the subject is grasped from the coordinate information left in the spatial coordinate information.