JPH06265322A - Method for detecting parallax error from a plurality of images - Google Patents

Abstract

PURPOSE:To detect the parallax error of the entire region of an image by estimating the corresponding point at an occlusion part even if the occlusion part is generated. CONSTITUTION:In S11, the corresponding points of a plurality of images obtained by a plurality of image-pickup systems are extracted to obtain a parallax line. In S12, it is judged whether the parallax line includes an occlusion part or not. In S13, it is judged whether each line which is cut by the occlusion part indicates a same object or not when the occlusion part is included. In S15, the occlusion part is smoothly interpolated for connecting the parallax line smoothly when it is judged that each line indicates the same object. On the other hand, in S14, only the lines indicating the objects in front of each image pick-up system are extended and lines are interpolated when it is judged that each line does not indicate the same object. In this case, the parallax line at the occlusion part is not connected smoothly.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method of extracting corresponding points from a plurality of images picked up by a plurality of image pickup systems, that is, a parallax detecting method.

[0002]

2. Description of the Related Art Conventionally, there are a template matching method and a collaborative algorithm based on visual processing as typical methods of extracting corresponding points from a plurality of images picked up by a plurality of image pickup systems.

The template matching method considers compound-eye imaging having a left imaging system and a right imaging system, and if the left imaging system captures a left image and the right imaging system captures a right image, for example, a left image. A corresponding point is determined by considering a template surrounding a certain point in the inside and comparing the similarity in the right image with the image of the template. There are two methods for comparing the similarities.

(A) SSDA method (Sequential Similarity)
Detection Algorithm) In the SSDA method, as shown in the following equation (1), for all pixels on the epipolar line of the left image and all pixels on the epipolar line of the right image, the image in the template of the left image The difference between the pixel value in the inside and the pixel value in the right image to be searched for is added, and the coordinate at which the obtained sum E (x, y) becomes the minimum value is used as the coordinate of the corresponding point.

[0005]

[Equation 1] In the SSDA method, if the sum of the differences in the pixel values being calculated becomes larger than the minimum value at other coordinates calculated up to now, the calculation may be stopped and the coordinates may be moved to the next coordinate.
The calculation time can be shortened by eliminating extra calculation.

(B) Correlation Method The correlation method is to obtain the cross-correlation between the pixel value in the image in the template of the left image and the pixel value in the right image to be searched, as shown in the following equation (2). Then, the correlation value ρ (x, y) is obtained in accordance with, and the coordinate at which the obtained correlation value ρ (x, y) becomes the maximum value is used as the coordinate of the corresponding point.

[0007]

[Equation 2] In the normalized cross-correlation shown in equation (2), the maximum value is "1".

Next, the cooperation algorithm will be described.

The cooperative algorithm proposed by David Marr is an algorithm for obtaining a parallax line using the following three rules (D. Marr, Vision: A Computatio
nalInvestigation into the Human Representation and
Processing of Visual Information, WH Freeman &
Co., San Francisco Calif., 1982). Rule 1 (Compatibility) ... Black dots can only match black dots. Rule 2 (Uniqueness) ... Almost always one black dot in one image can match the only black dot in the other image. Rule 3 (Continuity) The parallax of matching points changes smoothly in almost all areas.

According to Marr, the corresponding point extracting device for realizing the cooperative algorithm is a network in which many processing devices are connected in parallel to each other,
A small processor is placed at each intersection or node of the graph shown in FIG. If the node represents a correct pair of black dots, the processor placed at the node will eventually have the value "1", while if it represents an incorrect pair (false target), the process The device has the value "0".

From Rule 2, only one correspondence is allowed along a horizontal or vertical line. Thus, all processors placed at the nodes along each horizontal or vertical line are constrained to each other. This is the result of competition along each line,
This is because only one processor survives as a value "1" and all other processors have a value "0" to satisfy the rule 2.

According to Rule 3, the correct pair is likely to lie along the dashed line, so that an excitatory coupling is inserted between the processors aligned in this direction. This gives each local processing device a structure as shown in FIG. That is, the inhibitory coupling is applied to the processing devices along the horizontal line 201 and the vertical line 202 corresponding to the line of sight from both eyes, and the excitability is applied to the processing device along the oblique line 203 corresponding to the line with constant parallax. Apply a bond. This algorithm can be applied to two-dimensional images. In that case, as shown in FIG. 18C, the excitatory connection is applied to the two-dimensional minute neighborhood 204 with a constant parallax while the inhibitory connection remains unchanged.

In such a corresponding point extracting device, the left image and the right image are respectively taken, and "1" is given where the two black points match (all including the false target), and "0" is given for all the others. As a result, the network of the processing device is loaded and then the network is run. Each processor sums "1" s in the excitatory neighborhood and "1" s in the inhibitory neighborhood.
Then, after applying an appropriate weight to one of the totals, the resulting numerical values are subtracted. If the result exceeds a certain threshold then the processor is set to the value "1", otherwise the processor is set to "0". Formally, this algorithm is represented by the iterative relationship shown in equation (3) below.

[0014]

[Equation 3] here,

[0015]

[Outer 1] Shows the state of cells corresponding to position (x, y), parallax d, and time t in the network shown in FIG. S (x, y, d) is the local excitability neighborhood, O
(X, y, d) is near inhibitory. ε is the inhibition constant. σ is a threshold function. The initial state C ^o contains all possible pairs containing false targets within the defined parallax range. Here it is added each iteration (though it need not be, but this will cause the algorithm to converge faster).

[0016]

However, in the above-mentioned conventional corresponding point extraction method, when a three-dimensional object is imaged by a plurality of image pickup systems, one of the image pickup systems takes an image, while the other image pickup system takes an image. There is a problem that a part that is not imaged, that is, an occlusion part may occur, and in this occlusion part, corresponding points are not extracted, that is, parallax cannot be detected.

Therefore, an object of the present invention is to provide a parallax detection method from a plurality of images, which can detect the parallax in the entire image by estimating the corresponding points in the occlusion part even if the occlusion part occurs.

[0018]

In order to achieve the above object, a method of detecting parallax from a plurality of images according to the present invention provides a method for obtaining a single image from a plurality of images obtained by imaging the same object with a plurality of imaging systems. Extracting a corresponding point in another image with respect to an arbitrary point in an image, in a parallax detection method from a plurality of images, if there is a part that becomes an occlusion part without extracting the corresponding point, the corresponding It is characterized in that the shape of the object is estimated based on the shape of the line or the surface representing the parallax of the portion where the points are extracted, and the occlusion section is interpolated.

Further, it is judged from the shape of the line or surface representing the parallax whether or not the object is a single object, and if it is judged that the object is a single object, the occlusion part is made into a smooth line. If it is determined that the object is a plurality of objects, the shape of a line or a surface representing the parallax of the object existing in front of the imaging system is extended to interpolate the occlusion part. May be. .

The shape of the line or surface representing the parallax may be the curvature of the line or surface representing the parallax.

[0021]

In the present invention configured as described above, when the corresponding points cannot be extracted and there is a part which becomes the occlusion part, a line or a plane representing the parallax of the part where the corresponding points are extracted is present. By estimating the shape of the object based on the shape of, and performing the interpolation processing of the occlusion part, the corresponding point in the occlusion part is extracted even if the occlusion part where the corresponding point cannot be extracted exists, and the overlap part of each image is It is possible to extract corresponding points in all areas, that is, detect parallax.

The shape of the object is estimated by determining whether or not there is one imaged object based on the shape of the line or surface representing the parallax of the object, and when there is only one object, the occlusion part is a smooth line. This can be done by interpolating, and in the case of a plurality of objects, the occlusion part is interpolated by extending the shape of the line or surface representing the parallax of the object existing in front of the imaging system.

[0023]

Embodiments of the present invention will now be described with reference to the drawings.

(First Embodiment) FIG. 1 is a flowchart of a first embodiment of a method for detecting parallax from a plurality of pixels according to the present invention. The parallax detection method of the present embodiment will be briefly described. First, corresponding points of each image are extracted from a plurality of images obtained by a plurality of imaging systems by the corresponding point extraction method described in the related art, and will be described later. A parallax line is obtained (S11). And
It is determined whether or not this parallax line includes a blank portion, that is, an occlusion portion (S12). If no occlusion portion is included, no processing is performed. If an occlusion portion is included, the occlusion portion is used for division. It is determined whether or not the respective lines thus formed represent the same object (S13). If it is determined that they represent the same object, the occlusion part is filled in by smooth interpolation to connect the parallax lines smoothly (S1).
5). On the other hand, if it is determined that they do not represent the same object, only the line representing the object in front of each imaging system is extended to interpolate the lines (S14). At this time, the parallax lines in the occlusion part are not connected smoothly.

Corresponding point extraction, that is, parallax detection, is performed in the occlusion section based on the above-described procedures. Below, each procedure mentioned above is demonstrated in detail.

First, the concept of parallax lines will be described.

In this embodiment, for example, as shown in FIG.
Consider a compound-eye imaging device including a left imaging system 25 and a right imaging system 26, which are arranged so that their optical axes L _L and L _R are parallel to each other. Each of the image pickup systems 25 and 26 is provided with a sensor (not shown) such as a CCD as an image pickup element. In order to simplify the description, the left image 21 and the right image 22 obtained by the sensors of the image pickup systems 25 and 26 are shown at the positions between the image pickup systems 25 and 26 and the subject 30, and the left and right images are shown. It is assumed that the displacements on 21 and 22 are only one-dimensional displacements in the x direction. Unless otherwise specified, the following figures are also followed.

In FIG. 2, the epipolar surface 2 is a plane formed by connecting three points P on the object 20, the lens center 23 of the left imaging system 25 and the lens center 24 of the right imaging system 26.
A line whose epipolar plane 29 crosses the left image 21 is called a left epipolar line 27, and a line which crosses the right image 22 is called a right epipolar line 28.

Now, the optical axes L _L and L of the image pickup systems 25 and 26 are used.
_Since a compound eye image pickup device in which _R is parallel is considered, when the planar object 32 is imaged by this compound eye image pickup device as shown in FIG. 3, the planar object 32 is imaged in each image 21, 22 without distortion. Therefore, the left and right epipolar lines 27 and 28 are obtained by reading the respective images 21 and 22 in the x direction.

On the other hand, as shown in FIG.
6 optical axis L_L, L_RAre not parallel, that is, each optical axis
L_L, L_RIn the plane including
Line and each light perpendicular to the line connecting the lens centers 23 and 24 of
Axis L_L, L_RWith a certain angle (convergence angle)
When the flat subject 32 is imaged in different states,
L _L, L_REven if the images are taken in the same way as when the
The subject 32 has a trapezoidal distortion in each of the images 21 and 22.
It is imaged. In this case, the congestion of each imaging system 25, 26
Since the corners are known, the image is rearranged by the linear transformation and the compound eye
Optical axis L of imaging device_L, L_RGet the image when are parallel
It As a result, the left and right epipolar lines 27, 28
Obtained by reading the x-direction of the repositioned image
Be done.

A screen obtained by arranging the left and right epipolar lines 27 and 28 obtained as described above as shown in FIG. 5 is called a parallax screen 51. That is, the left epipolar line 27 is the horizontal axis, the right epipolar line 28 is the vertical axis, and the left and right epipolar lines 27,
The left and right epipolar lines 27, 28 are arranged so that the left ends 27a, 28a of the 28 correspond to the lower left point of the parallax screen 51.
The screen obtained by arranging at right angles is the parallax screen 5.
It is 1.

Here, as shown in FIG. 6, a compound eye image pickup device in which the optical axes L _L and L _R of the left and right image pickup systems 25 and 26 are parallel to each other is used to form a flat plate on the background of the semi-cylindrical object 61 in plan view. Ideally, by capturing the landscape in which the object 62 is arranged and plotting the corresponding points of the left and right epipolar lines 27 and 28 respectively obtained from the left and right images on the parallax screen 51 shown in FIG. Can obtain a line as shown in FIG. This line is called a parallax line 71. That is, the parallax line 71 indicates the corresponding points of the left and right epipolar lines 13 and 14, and also indicates the parallax.

The parallax line 71 will be described in more detail below. Left and right epipolar lines 13 and 14
In the place where the parallax is constant above, the parallax line 71 on the parallax screen 51 is a straight line rising 45 degrees to the right. By the change of the parallax line 71, the change of parallax in the left and right images can be known. This will be described with reference to FIG. FIG. 8 (a)
As shown in, the lens center 2 of each of the left and right imaging systems 25 and 26
Positions where the depths from 3 and 24 are constant are represented by the same symbols ☆, ★, ○, and ◎, respectively, and when they are plotted on the parallax screen 51 shown in FIG. The comrades are plotted on a straight line rising 45 to the right. In addition, as shown by the alternate long and short dash line in FIG. 8A, the parallax line obtained when an image of a convex object 81 protruding midway from the star position to the star position is captured is shown in FIG. As indicated by the alternate long and short dash line, it starts from the parallax line indicated by *, moves to the parallax line indicated by * on the way, and finally returns to the parallax line indicated by *. Furthermore, FIG.
From (a) and (b), it can be understood that the diagonal line rising to the right of the parallax screen 51 indicates a parallax line whose depth from each lens center 23, 24 is infinite.

As described above, the parallax line 71 indicates the depth from each lens center 23, 24. Therefore, from the point A at the lower right corner in the parallax screen 51 shown in FIG. 7 to the upper left corner in the parallax screen 51, the parallax line 7 is drawn.
Looking at 1, the shape represents the actual object arrangement of the three-dimensional landscape. That is, the parallax line 71 makes it possible to estimate the object arrangement of a three-dimensional scene.

Now, the objects 61 and 62 shown in FIG. 6 are picked up by the left and right image pickup systems 25 and 26, and the corresponding points of the obtained left and right images are determined by the template matching method shown in the prior art (separately). As a result of the extraction, as shown in FIG. 9, in the parallax screen 51, two blank areas, that is, three lines 9 divided by the occlusion section 91 are obtained.
It is assumed that 2, 93, and 94 are obtained. These two occlusion parts 91 are imaged by the left imaging system 25 but not by the right imaging system 26, and by the right imaging system 26 but not by the left imaging system 25, and corresponding points cannot be extracted. It is a place. The method of extracting corresponding points in each occlusion unit 91 will be described below.

In FIG. 9, when the obtained lines 92, 93, and 94 are viewed from the point B at the lower right corner of the parallax screen 51, the line 93 can be seen at the forefront. Let's consider the meaning of this line. For example, as shown in FIG. 10A, when a landscape in which a flat object 102 is arranged on the background of an object 101 having a rectangular shape in plan view is imaged by the left and right imaging systems 25 and 26, ideally, The parallax line 103 drawn in the parallax screen 51 of FIG. 10B is obtained. Further, as shown in FIG. 11A, when a landscape in which a flat object 112 is arranged on the background of a triangular object 111 in plan view is imaged by the left and right imaging systems 25 and 26, ideally, Figure 11
(B) Parallax line 113 drawn in the parallax screen 51
Is obtained. From these results, it can be said that the shape of the parallax line appearing in the parallax screen 51 is similar to the shape of the imaged object, and the line 93 shown in FIG. 9 images the object having an arcuate shape. It can be estimated that it was obtained by the above.

Here, if the assumption that "the parallax on the visible surface of one object continuously changes" is defined, this occlusion of each occlusion part 91 generated in the parallax screen 51 shown in FIG. It is understood that it is necessary to connect the connecting portions with the lines 92, 93, 94 with lines that are as smooth as possible. Further, an extension line from the line 92 and the line 9 obtained by estimating the corresponding points in the occlusion section 91 between the line 92 and the line 93 in FIG.
The extension lines from 3 always have intersections, but from the above assumption, if the line 92 and the line 93 are obtained from the same object, the extension lines must be connected smoothly. On the other hand, the extension lines must not be connected smoothly unless they are obtained from the same object. Whether the objects are the same or not can be determined from the shapes of the lines 92 and 93. The same applies to the occlusion portion 91 between the line 93 and the line 94.

From the above, it can be considered that the lines showing all the corresponding points including the corresponding points of each occlusion section 91 in FIG. 9 are the lines for each object represented. In addition, the line can be graphed and displayed as a function on the parallax screen 51. That is, the parallax line 71 (see FIG. 7) is considered as a function having the pixel number of the left epipolar line 27 as an input value and the pixel number of the right epipolar line 28 as an output value. Then, there is a function for each line representing each object, the function of the parallax line 71 indicating all corresponding points including the corresponding points of the occlusion unit 91 is F, and each function representing each object is F ₁ , F ₂ , ..., F
_{Letting k} be F = F ₁ + F ₂ + ... + F _k (4) Each function F ₁ , F in this equation (4)
_2, ..., F _k are obtained by determining the object represented by the parallax line 71.

A method of obtaining the parallax line 71 as the interpolated function F will be described below with reference to FIG. To obtain this function F, there are several parameters. The shape of the obtained line is the occlusion part 91.
Is the size of the gap. Here, as an example, the curvature ρ of the line is used as a parameter representing the shape of the line. For the sake of simplicity this time, it will be decided whether or not the objects are the same from the shape of the line only by the curvature ρ of the line. The curvature of the line 93 is ρ ₉₃ , the curvature of the line 94 is ρ ₉₄ ,
The function representing the line 93 is F ₉₃ , and the function representing the line 94 is F ₉₄ . From FIG. 9, since the curvature ρ ₉₃ of the line ₉₃ and the curvature ρ ₉₄ of the line ₉₄ are different, it is determined that the function F ₉₃ representing the line 93 and the function F ₉₄ representing the line ₉₄ are different. Therefore, the function F ₉₃ and the function F ₉₄ are not connected smoothly. Further, when viewed from the point B, it is determined that the line 93 is present in front of the line 94 and the object represented by the line 93 is present in front of the object represented by the line 94. The occlusion part 91 between the line 94 and the line 94 interpolates by extending the shape of the line 93. The interpolated line and the line 94 are not connected smoothly. Similarly, in the occlusion portion 91 between the line 92 and the line 93, interpolation is performed by extending the shape of the line 93, and the interpolated line and the line 92 are not connected smoothly. As a result, the parallax line 71 shown in FIG. 7 is obtained.

As described above, the occlusion section 9
By estimating the actual object arrangement of the three-dimensional scene based on the sizes of the gaps between the lines 92, 93, 94 and the occlusion part 91 obtained by other than 1, and detecting the entire parallax line 71, Corresponding point extraction, that is, parallax detection can be performed over the entire area of the images captured by the imaging systems 25 and 26.

Further, as described above, in the present embodiment, in order to make the explanation easy to understand, each of the left and right images 21, 2 is
Although the deviation on 2 exists only in the x-axis direction of the sensor, the one-dimensional deviation is considered, but the invention is not limited to this. For example, in the case of an image having a two-dimensional shift, a one-dimensional shift in the x direction is detected by the method described above, and then a signal is read in the y direction of the sensor to detect the shift in the y direction as in the x direction. However, a two-dimensional shift can also be detected by combining the shifts in the x and y directions. As another method of detecting the two-dimensional shift, the left image 21 and the right image 22 obtained by the left and right imaging systems 25 and 26 are arranged as shown in FIG. When the corresponding points obtained by using the corresponding point extraction method such as the template matching method shown in the related art are plotted, a parallax plane having parallax lines as planes can be obtained. Then, based on this parallax surface, it is also possible to detect the parallax of the occlusion part by the same procedure as the above-mentioned procedure. In addition, as an improved version of this method, a plurality of pairs of epipolar lines are used instead of the entire image, and a parallax solid 121 is similarly configured to generate a parallax plane, and similarly parallax of an occlusion part is detected. It is also possible to do so.

Further, as shown in FIG. 13, the left imaging system 2
5 and the right imaging system 26 are arranged so that the vergence angles are directed outwards from each other, and when the overlapping part 131 that can be imaged by both the left imaging system 25 and the right imaging system 26 is only a part of the imaging range, It is possible to rearrange the epipolar lines and similarly detect the parallax of the occlusion part in the overlap part 131.

When the left and right image pickup systems 25 and 26 have a convergence angle, the parallax line at a position where the depth from the lens centers 23 and 24 is constant changes. For example, as shown in FIG. 13, the object imaged when the vergence angle is outward is the lens center 2
If the depths from 3 and 24 exist at infinity, an arc-shaped parallax line 1 as shown by a broken line in FIG.
41 is obtained. On the contrary, when the vergence angle is inward, an arc-shaped parallax line 141 'as shown by a chain line is obtained at a position symmetrical to the parallax line 71 when there is no vergence angle. Parallax lines 141 and 1 when a convergence angle is attached
The curvature of 41 'changes according to the magnitude of the angle of convergence. Therefore, if the state of the parallax lines when the convergence angles are respectively obtained in this way is known, the left and right imaging systems 25, 26
It is possible to detect all parallax in the overlap part 131 by performing the same processing as when there is no vergence angle without rearranging the image and epipolar line even when the vergence angle is attached to .

(Second Embodiment) Now, based on the corresponding point extraction method such as the template matching method shown in the prior art,
As a result of extracting the corresponding points of each image obtained by the two imaging systems and plotting them on the parallax screen, five lines 1 divided by four occlusion units 151 as shown in FIG.
It is assumed that the parallax screen 51 in which 52, 153, 154, 155, and 156 are drawn is obtained. Then, as shown in FIG.
Each line 152, 153, 154, 155, 156 is connected while extending its length while maintaining its curvature and connecting smoothly. As a result, in FIG. 16, the line 152 and the line 156 are connected, and the line 153 and the line 155 are connected, so that the two regions 161 and 162 are connected.
To form. Areas 161 and 162 respectively represent objects arranged in a three-dimensional scene.

In this way, each line 152, 153, 1
54, 155, 156 shapes and occlusion section 151
Of the lines 152, 153,
Each object can be separated by interpolating 154, 155, and 156. The parallax line 171 as shown in FIG. 17 is obtained by performing the hidden line processing on the parallax screen 51 shown in FIG. 16 from the point A at the lower right end of the parallax screen 51. Accordingly, the parallax of each image obtained by the two image pickup systems can be detected.

In each of the embodiments described above, the parallax detection processing for two images obtained by the two image pickup systems has been described, but the number of image pickup systems is not limited to this.
There may be three or more.

[0047]

As described above, according to the present invention, the shape of an object is estimated based on the shape of a line or a surface representing the parallax of the part where the corresponding points are extracted, and the interpolation processing of the occlusion part is performed. By performing the above, it becomes possible to extract corresponding points in the occlusion part, that is, to detect parallax, which occurs when an object is imaged by a plurality of imaging systems. it can.

[Brief description of drawings]

FIG. 1 is a flowchart of an embodiment of a parallax detection method from a plurality of images according to the present invention.

FIG. 2 is a diagram for explaining epipolar lines of left and right images obtained when a subject is imaged by a compound-eye imaging device having two left and right imaging systems.

FIG. 3 is a diagram showing left and right images obtained when a planar subject is imaged by a compound eye imaging device in which optical axes of left and right imaging systems are arranged in parallel.

FIG. 4 is a diagram showing left and right images obtained when a flat subject is imaged by a compound eye imaging device in which each of the left and right imaging systems has a convergence angle.

FIG. 5 is a diagram showing a parallax screen composed of left and right epipolar lines.

FIG. 6 is a diagram showing the arrangement of each object and each of the left and right imaging systems when capturing an image of a landscape in which a flat object is arranged on the background of a semicylindrical object in plan view.

7 is a diagram of a parallax screen showing ideal parallax lines obtained by imaging the two objects shown in FIG.

FIG. 8 is a diagram for explaining changes in the parallax line with respect to the depth from the lens center of each of the left and right imaging systems, and FIG.
Is a diagram showing the correspondence to the left and right epipolar lines depending on the positions with respect to the left and right imaging systems, and FIG. 11B is a diagram of the parallax screen obtained in FIG.

FIG. 9 is a diagram of a parallax screen before interpolation in which parallax lines are interrupted by an occlusion unit.

FIG. 10 is a diagram for considering the meaning of lines on the parallax screen shown in FIG. 9, and FIG. 10 (a) is an image of a landscape in which a flat object is arranged on the background of an object having a rectangular shape in plan view. FIG. 2B is a diagram showing the arrangement of each object and each of the left and right imaging systems when
[Fig. 6] is a diagram of a parallax screen showing ideal parallax lines obtained by imaging the two objects shown in Fig. 14A.

11 is a diagram for considering the meaning of lines on the parallax screen shown in FIG. 9, and FIG. 11A is an image of a landscape in which a flat object is arranged on the background of a triangular object in plan view. FIG. 2B is a diagram showing the arrangement of each object and each of the left and right imaging systems when
[Fig. 6] is a diagram of a parallax screen showing ideal parallax lines obtained by imaging the two objects shown in Fig. 14A.

FIG. 12 is a diagram for explaining an example of a method for detecting a two-dimensional shift between left and right images.

FIG. 13 is a diagram showing the arrangement of left and right imaging systems when the convergence angle is outward.

FIG. 14: When there is no vergence angle, when the vergence angle is outward,
FIG. 11 is a diagram of a parallax screen showing a parallax line at infinity when the vergence angle is inward.

FIG. 15 is a second method of detecting parallax from a plurality of images according to the present invention.
FIG. 8 is a diagram of a parallax screen showing parallax lines before interpolation obtained by imaging a certain object, for explaining the embodiment.

FIG. 16 is a diagram of a parallax screen in which each line shown in FIG. 15 is extended and each line is interpolated.

17 is a diagram of a parallax screen showing ideal parallax lines obtained by performing shading processing on the parallax screen shown in FIG.

FIG. 18 is a diagram for explaining a cooperation algorithm.

[Explanation of symbols]

20 subject 21 left image 22 right image 23, 24 lens center 25 left imaging system 26 right imaging system 27 left epipolar line 27a, 28a left edge 28 right epipolar line 29 epipolar surface 32 plane subject 51 parallax screen 61, 62, 81, 101, 102, 111, 112
Objects 71, 103, 113, 141, 141 ', 171
Parallax lines 91, 151 occlusion parts 92, 93, 94, 152, 153, 154, 155,
156 Line 131 Overlap part 161, 162 Region L _L , L _R Optical axis

Claims (3)

[Claims] 1. A plurality of images, each of which is obtained by imaging the same object with a plurality of imaging systems, extracts a corresponding point in another image with respect to an arbitrary point in one image. In the parallax detection method, when there is a part that becomes an occlusion part without being able to extract the corresponding points, the shape based on the shape of the line or surface representing the parallax of the part where the corresponding points are extracted is A parallax detection method from a plurality of images, characterized by estimating the shape of an object and performing interpolation processing of the occlusion part. 2. It is determined whether or not the object is a single object from the shape of the line or surface representing the parallax, and if the object is a single object, the occlusion part is a smooth line. When it is determined that the object is a plurality of objects, the shape of the line or the surface representing the parallax of the object existing in front of the imaging system is extended to interpolate the occlusion part. A method for detecting parallax from a plurality of pixels according to claim 1. 3. The shape of the line or surface representing the parallax is
The line or surface curvature representing the parallax is used.
Alternatively, the parallax detection method from a plurality of images described in 2.