JP2000074665A - Device and method for generating distance image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a distance image which indicates a distance between a camera and an object to be imaged by adjusting an image based on detected information of a zoom camera under a zoom condition and using an image converted with appropriate zoom scale and the other image. SOLUTION: The zoom condition of a standard camera 2 is detected by a detecting mechanism such as a potentiometer, a sensor signal corresponding to the zoom condition is sent to a sensor signal detecting part 10, and the zoom scale, an image center, and lens distortion to output voltage are determined with conversion tables or the like. A distance image generating part 12 initializes and optimizes the zoom scale, an optimal evaluation value, and a zoom scale optimum value, and conducts scale conversion of a reference image using the zoom scale, the image center, and lens distortion. The picture of the reference image is scaled down so as to become the identical size to the picture of the reference image used in calibration. The distance image generating part 12 conducts distance image generating process and evaluation value computing process based on the reference image after scale conversion.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a distance image generating apparatus and method for generating a distance image indicating a distance between a camera that captures an image of an object using a zoom function and the object.

[0002]

2. Description of the Related Art As a method of measuring a distance in a three-dimensional space, there is a method of measuring a distance from a certain camera to an object to be imaged by using a so-called stereo method for an image taken by using a plurality of cameras.

The distance between the reference camera and the object to be imaged measured by the stereo method connects the optical center a of the reference camera in a three-dimensional space and a point nb on the image plane A as shown in FIG. It is calculated using the line of sight formed by connecting the point P on the line of sight, the optical center b of the detection camera, and the point nd on the imaging plane B of the detection camera, and the positional relationship between the two cameras. And in the actual distance measurement,
By searching on the epipolar line L on the detection image captured on the imaging surface B of the detection camera, a point on the detection image corresponding to a point on the reference image captured on the imaging surface A of the reference camera is calculated. I have. What is epipolar line L?
A plane defined by the point P, the optical center a of the reference camera, and the optical center b of the detection camera, and an intersection line with the imaging plane B of the detection camera. The projection point nd necessarily exists on the epipolar line L.

When a point on a detected image corresponding to a point in a reference image is detected, as shown in FIG.
Is compared with the detected images n _d1 , n _d2 ,..., N _d6 on the epipolar line L to search for the region having the highest similarity in the detected image. Done. The evaluation value for comparing the similarity is generally obtained by detecting a correlation between the reference image and the detected image.

[0005] For example, when a small area around the projection point nb is used as a template, an evaluation value is calculated by the following equation.

[0006]

(Equation 1)

Where I (x) is the luminance value of the reference image,
I ′ (x ′) is the luminance value of the detected image.

As described above, in the distance measurement by the stereo method, the accuracy of detecting the point nd on the detection image corresponding to the point nb is greatly related to the accuracy of the distance measurement. When doing so, there are many factors that degrade the accuracy of the distance measurement. For example, when an image has a uniform plain pattern, when distance information is missing due to blockage, noise caused by individual differences between cameras, and the like.

Therefore, when distance measurement is performed by the stereo method, many measures have been taken in order to cope with the above-mentioned deterioration in accuracy. One of the proposed multi-view stereo systems with multiple cameras arranged is to reduce the effects of occlusion by applying a stereo method by changing the combination of the reference camera and the detection camera. The above-described deterioration of the distance measurement accuracy has been suppressed.

[0010]

However, when a camera having a zoom function is used as a reference camera constituting the above-mentioned multi-view stereo system, lens parameters are measured with high accuracy in response to changes in zoom state. The inability to do so causes a deterioration in distance measurement accuracy. In particular,
Conventional cameras cannot measure a zoom scale with high accuracy in response to a change in zoom. Therefore, when performing distance measurement with high accuracy in a conventional multi-view stereo system, the zoom function of the reference camera cannot be effectively utilized, and the viewing angle of the reference camera must be fixed and used.

Here, in the multi-view stereo system, it is necessary to perform calibration in advance and define the relationship between the reference camera and each detection camera by a projective transformation matrix. Therefore, when generating a distance image by changing the zoom state of the reference camera, an algorithm for performing calibration for each zoom state, that is, for each zoom scale, or an algorithm for generating a distance image while converting the scale of a captured image is used. It is possible to build. In a multi-lens stereo system employing such a method of converting a scale, it is necessary to obtain a zoom scale with high accuracy in order to improve the accuracy of distance measurement as described above, but there are the following factors. Because it was difficult.

First, there is no measuring means or sensor for detecting the zoom scale of the reference camera with high accuracy.

Secondly, in zooming in and zooming out operations, the zoom adjustment mechanism for adjusting the zoom scale of the reference camera is mechanically played (backlash).
Exists.

Third, the lens distortion changes as the zoom state of the reference camera changes.

Actually, the first, second, and third factors occur in a complex manner, and it becomes more difficult to estimate the zoom scale with high accuracy.

On the other hand, in the conventional method, the zoom scale obtained from the output voltage of the potentiometer including the measurement error is used as it is, and the lens parameters such as the scale are not optimized. Had to be forced.

Further, when the zoom of the reference camera is operated,
In some cases, it may be necessary to re-adjust the focus on the reference image. In such a case, it is desirable to adjust the focus ring to generate a clearer reference image before generating the distance image. However, when the focus is adjusted, the visual field size in the reference image often changes, resulting in a zoom scale different from the actual zoom scale, which causes deterioration in accuracy when generating a distance image.

Accordingly, the present invention has been proposed in view of the above-described circumstances, and uses a zoom function of a reference camera when generating a range image to accurately estimate a zoom scale. It is an object of the present invention to provide a distance image generating apparatus and method capable of generating a highly accurate distance image.

[0019]

A distance image generating apparatus according to the present invention for solving the above-mentioned problems includes an image pickup means comprising at least one camera having a zoom function and comprising a plurality of cameras, and a camera having a zoom function. Zoom state detecting means for detecting a zoom state and calculating at least a zoom scale;
Scale conversion means for changing the zoom scale of an image captured by any one of the cameras, and an image on which scale conversion has been performed by the scale conversion means and a point on the image on which scale conversion has not been performed corresponding to a point on the image. A zoom scale detecting means for generating an evaluation value indicating the degree of similarity and detecting an appropriate zoom scale, an image at the zoom scale detected by the zoom scale detecting means, and an image not subjected to scale conversion. And a distance image generating means for generating a distance image indicating a distance between any one of the cameras and the object to be imaged.

According to such a distance image generating apparatus, even if the zoom state of the camera changes, the zoom state is detected by the zoom state detecting means, and the image is adjusted accurately based on the detected information. A zoom scale is obtained, and a distance image indicating a distance between the camera having the zoom function and the imaging target is generated using the image converted with the appropriate zoom scale and another image.

Further, the distance image generating method according to the present invention comprises:
Among a plurality of cameras including one or more zoom cameras that generate an image by capturing an image of an object to be captured using a zoom function, a zoom state of the zoom camera is detected, at least a zoom scale is calculated, and a periphery of the zoom camera is calculated. A first step of generating an image by another camera disposed in the camera, and changing the scale of the zoom image or the image generated by the other camera within a predetermined range, and A second step of comparing a point corresponding to a point on an image generated by the other camera to generate an evaluation value indicating similarity, and using the zoom image and an image generated by the other camera. A third step of adjusting the zoom scale based on the evaluation value generated by the other camera, and the other camera corresponding to a point on the zoom image on the adjusted zoom scale. In detecting and corresponding points on the generated image, characterized in that a fourth step of generating a distance image representing a distance between the imaging object and the zoom camera.

According to such a distance image generating method, even when the zoom state of the zoom camera is changed and a zoom image is taken, the zoom state is detected and the accurate zoom obtained from the detection information is detected. The first image is adjusted using the scale, and the first image is adjusted using the first image and the second image.
Then, a distance image indicating the distance between the image and the object to be imaged is generated.

[0023]

Embodiments of the present invention will be described below in detail with reference to the drawings.

Range image generating apparatus 1 to which the present invention can be applied
Has a reference camera 2 and a detection camera 3 arranged around the reference camera 2 as shown in FIGS.

As shown in FIG. 1, the reference camera 2 is arranged at a substantially central position among a plurality of cameras, and captures an object to be imaged to generate a grayscale image (hereinafter, referred to as a reference image). . The reference camera 2 has a zoom mechanism for increasing or decreasing a viewing angle by zooming when capturing an image of an imaging target. The zoom state of the zoom mechanism is detected by a detection mechanism including, for example, a potentiometer. The potentiometer generates an output voltage according to the zoom state of the zoom mechanism, encodes the output voltage, and generates a sensor signal.

As a detection mechanism for detecting the zoom state, not only a potentiometer but also a rotary encoder or the like which is used by connecting to a lens in the reference camera 2 from the outside can be used.

The detection camera 3 is a camera arranged around the reference camera 2, and picks up an image of an object to be picked up and generates a gray-scale image (hereinafter referred to as a detected image).

As shown in FIG. 3, the reference image B and the detection images D1 to D4 taken by the reference camera 2 and the detection camera 3 are the same as the object to be imaged because the reference camera 2 has a zoom function. Even when an object is imaged, the pattern of the reference image B becomes the reference image B 'in the figure and becomes larger depending on the zoom state of the reference camera 2.

As shown in FIG. 2, the distance image generating apparatus 1 includes a sensor signal detecting section 10 for detecting a zoom state of the reference camera 2, a reference image picked up by the reference camera 2 and the detection camera 3, A frame memory 11 in which an image is stored, and a distance image generation unit 12 that generates a distance image using the reference image and the detected image from the frame memory 11
And

The sensor signal detecting section 10 is connected to the reference camera 2 and receives a sensor signal from the above-described detecting mechanism. The sensor signal detection unit 10 obtains the relationship between the output voltage Vn and the zoom scale of the reference camera 2 when a potentiometer is used, for example, using the input sensor signal. Here, the zoom scale is expressed as a ratio of the same object size in the reference image when the zoom state changes with respect to the object size in the reference image at an output voltage V _calib described later. More specifically, the zoom scale determines a corresponding point in the reference image at each output voltage Vn with respect to a feature point on the reference image at the output voltage V _calib by a so-called image matching method or the like, and _obtains the reference image at the output voltage V _calib . It is obtained by obtaining the positional relationship between the above feature point and the corresponding point on the reference image at the output voltage Vn.

As shown in FIG. 4, the relationship between the output voltage Vn and the zoom scale S is such that as the output voltage increases, the zoom scale increases, that is, the image enlargement ratio increases. “V _calib ” in the figure _indicates a zoom when calibration is performed to obtain a point of the detected image corresponding to a point on the reference image when the reference camera 2 and the detection camera 3 are positioned and an image of the same object is taken. This is an output voltage value corresponding to the scale S _calib (= 1).
Zoom Scale _{S c} alib corresponding to the output voltage V _calib is used as a reference zoom scale, the output voltage V _0, V _1,
... When Vn deviates in the positive direction with respect to the output voltage V _calib , the reference image is zoomed in. When _Vn deviates in the negative direction with respect to the output voltage V _calib , zoom-out is indicated. . That is, FIG.
Indicates that the zoom scale zooms in to S ₀ , S ₁ ,..., Sn when the output voltages become V ₀ , V ₁ ,. In addition, since mechanical backlash exists in the zoom operation in the reference camera 2, the relationship between the output voltage Vn and the zoom scale is obtained separately for zooming in and zooming out.

In order to obtain the relationship between the output voltage Vn and the zoom scale S in FIG. 4, the process is stopped by changing the zoom scale of the reference camera 2 and plotting the zoom scale with respect to the output voltage at each point. .

Here, the calibration is performed by setting the viewing angle between the reference camera 2 and the detection camera 3 to a predetermined angle.

The sensor signal detecting section 10 prepares the relationship between the output voltage and the zoom scale shown in FIG. 4 as a table, for example, and detects the sensor signal from the reference camera 2 so that the zoom scale of the reference camera 2 can be adjusted. Is generated and output to the distance image generation unit 12.

The relationship between the output voltage Vn and the zoom scale S may be prepared and prepared as a table of the relationship between the zoom scale and the image center and the relationship between the zoom scale and the lens distortion. .

Further, the sensor signal detection unit 10 detects the center of the image of the reference camera 2 (X
c, Yc) and a lens distortion signal indicating the lens distortion κ of the reference camera 2 are generated and output to the distance image generation unit 12. In addition, the sensor signal detection unit 10
As in the case of the zoom scale described above, the sensor signal and
The relationship with the image center and the relationship with the lens distortion are prepared as a table, for example, and a sensor signal from the reference camera 2 is detected to generate an image center signal and a lens distortion signal. Output.

These relationships may be created as a table as a relationship between the zoom scale and the center of the image and a relationship between the zoom scale and the lens distortion.

When a rotary encoder is used as a detection mechanism built in the reference camera 2, the sensor signal detection unit 10 counts the number of pulses output from the rotary encoder as a sensor signal, Obtain the zoom scale, image center and lens distortion.

The frame memory 11 is connected to the reference camera 2 and each detection camera 3, and receives a reference image and a detection image. The frame memory 11 performs A / D conversion on the input reference image and the detected image and performs a distance image generation unit 12
Output to

The distance image generation unit 12 uses the scale signal, the image center signal, and the lens distortion signal sent from the sensor signal detection unit 10 to generate the reference camera 2 and the object to be imaged in accordance with the zoom state of the reference camera 2. Then, a distance image indicating the distance is generated.

The distance image generating unit 12 is configured to generate a distance image on the basis of each detection point corresponding to a point on the reference image when the reference camera 2 and each detection camera 3 have substantially the same viewing angle. Perform calibration to find points on the image. Thereby, the distance image generation unit 12 can obtain a corresponding point on the detected image corresponding to a point on the reference image when the imaging target is imaged.

The distance image generating apparatus 1 configured as described above
Performs the processing shown in the flowchart of FIG. 5 when generating the distance image.

First, in step S1, the zoom state of the reference camera 2 is detected by the detection mechanism. Then, the detection mechanism generates a sensor signal according to the zoom state, and outputs the sensor signal to the sensor signal detection unit 10.

In step S2, the sensor signal detector 10
When a potentiometer is used as a detection mechanism, for example, a zoom scale Sv, an image center (Xc, Yc), and a lens distortion κ with respect to the output voltage Vn are obtained by referring to a conversion table or the like to obtain a distance image generation unit. 12 is output.

In step S3, the distance image generation unit 1
2 performs an initialization process on the zoom scale S, the optimum evaluation value E _min , and the zoom scale optimum value S _{opt in} order to optimize the zoom scale when the distance image is generated. That is, the initialization process in step S3 is a prerequisite process for optimizing a zoom scale S described later to obtain a zoom scale optimized value S _opt . In addition,
The optimum evaluation value E _min is determined by using a sum of squared absolute distance (SSAD) obtained when a distance image is generated.

Then, in the distance image generation unit 12, S
_{When range} is a constant that specifies a scale range to be subjected to optimization processing described later, by converting to a zoom scale obtained by subtracting “S _range ” from the zoom scale Sv whose value is adjusted by the optimization processing described later. initializing, the zoom scale optimum value S _{op t} in step S3 is "S". Further, the distance image generation unit 12
An optimum evaluation value E _min indicating a similarity between a point on the reference image and a point on the detected image corresponding to the point on the reference image is set to a predetermined value E.

In step S4, the distance image generation unit 1
2 is the zoom scale S obtained by performing the initialization processing in step S3, and the image center (X
c, Yc) and the scale distortion of the reference image using the lens distortion κ, thereby converting the reference image B shown in FIG. 6A into the reference image B ′ shown in FIG. 6B. I do. Note that the detected images D1 to D4 are not subjected to scale conversion.

That is, when performing the scale conversion processing in step S4, the distance image generation unit 12 performs the processing in steps S11 to S13 as shown in FIG.

First, in step S11, the lens distortion κ at the zoom scale Sv and the image center (Xc, Y
Remove the distortion of the reference image using c). As a result, as shown in FIG. 8A, distortion is removed from the reference image as compared with before the distortion is removed. At this time, the distance image generation unit 12
Removes the distortion of the reference image assuming that the distortion of the reference image changes in proportion to the cube of the distance from the center of the image.

In step S12, the distance image generation unit 12 performs scale conversion of the reference image. When performing the scale conversion, the distance image generation unit 12 uses the zoom scale S and the image center (Xc, Yc) of the reference image from which the distortion has been removed in step S11. At this time, the distance image generation unit 12 performs scale conversion using a conversion formula represented by Xi = Xc + (Xi−Xc) / S Yi = Yc + (Yi−Yc) / S. Here, (Xi, Yi) is the coordinates of the ith pixel in the XY direction on the reference image.

As a result, as shown in FIG. 8B, the pattern of the reference image is reduced as compared with that before the scale conversion, and has the same size as the pattern of the reference image used at the time of calibration.

In step S13, the distance image generation unit 12 _sets the image center (X
c, Yc) and the lens distortion κ, distortion is added to the reference image scale-converted in step S12. As a result, as shown in FIG. 8C, the reference image is converted into a reference image captured in a zoom state at the time of calibration.

Returning to FIG. 5, in step S4, the distance image generation unit 12 performs the distance image generation processing and the evaluation value calculation processing using the reference image after the scale conversion processing. At this time, a distance image is generated by using a relationship between a point on the reference image and a corresponding point on the detected image created as a table by calibrating in advance, and an evaluation value E is calculated using the following equation. .

[0054]

(Equation 2)

Where I (x) is the luminance value of the reference image,
I ′ (x ′) is the luminance value of the detected image.

At this time, steps S11 to S1
By performing the processing shown in FIG. 3 on the reference image, the effective area of the distance image becomes smaller than the reference image captured by the reference camera 2. Therefore, when calculating the evaluation value E using the reference image subjected to the scale conversion processing, the effective area of the reference image captured by the reference camera 2 is considered. That is, by performing the scale conversion process, the number of pixels forming a predetermined region of the reference image captured by the reference camera 2 and the number of pixels of the reference image corresponding to the predetermined region after the scale conversion is performed Therefore, when the evaluation value E is obtained by the above equation, an average of the evaluation values E in an arbitrary region is used as the evaluation value E in each small region.

Next, in step S5, the distance image generator 12 performs a zoom scale optimization process. That is, in step S5a, the distance image generation unit 12
The optimum evaluation value E _min determined in step S3 is compared with the evaluation value E calculated in step S4, and the optimum evaluation value E _min is determined.
When it is determined that the evaluation value E is smaller than _min , the process proceeds to step S5b, and when it is determined that the evaluation value E is not smaller than the optimum evaluation value E _min , the process proceeds to step S6.

[0058] In step S5b, updates the evaluation value E that is selected in the above step S5a as the optimum evaluation value E _min, and updates the zoom scale S in the converted optimum evaluated value E _min to optimal values S _opt.

In step S6, the distance image generation unit 1
2 is the zoom scale S at the time when the processing shown in steps S4 and S5 is performed, and “Sv +
Compare the size of “S _range ” with “S ≧ Sv + S _range ”
When it is determined, the process proceeds to step S7, where “S ≦ Sv +
When it is determined that "S _range ", the process proceeds to step S8, where the zoom scale S is incremented to "S + S _step ".

That is, the distance image generation unit 12 outputs
The processing of steps S4 to S6 described above is repeated for each S _step within the _{range of} ≦ S ≦ + S _range (S _start ≦ S ≦ S _end ). That is, the scale conversion of the reference image is performed in step S4, and the evaluation value E obtained from the scale-converted reference image and the detected image is compared with the optimum evaluation value _Emin. Evaluation value E _min
Is updated to the evaluation value E, and the optimum value S _opt is updated to the zoom scale S, thereby detecting an optimum zoom scale within a predetermined zoom scale range.

The range image generation unit 12 calculates ± S
When determining the optimum value S _opt of the zoom scale zoom scale is changed in S _{s tep} units within the _range, for example, as shown in FIG. 9, actually obtained zoom scale S
By extracting a plurality of points near the minimum of the evaluation value E with respect to _{start to} S _end and interpolating these points, the optimum value S _{opt is obtained.}
May be determined. Further, the distance image generation unit 12 may extract a plurality of points in the vicinity of the minimum of the evaluation value E, and determine the optimum value S _opt by approximating the points with a quadratic function.

In step S7, the distance image generation unit 1
2 generates a distance image at the optimum value _Sopt obtained in step S5 described above. At this time, the distance image generation unit 12
In, performs scaling processing shown in step S11~ step S13 using the optimum value S _opt of the zoom scale, to generate a distance image using the reference image and the detected image subjected to the scale conversion process. When outputting the generated distance image, the distance image generation unit 12 performs inverse conversion on the zoom scale, lens distortion, and image center, and outputs the zoom scale and lens when the reference camera 2 captures an image of the imaging target. The transformation is performed so as to be centered on the distortion and the image.

The distance image generating apparatus 1 detects the zoom state of the reference camera 2 by the sensor signal detecting unit 10 to obtain the relationship between the output voltage Vn, the zoom scale, the lens distortion and the image center, and uses this relationship. By performing scale conversion of the reference image to obtain the evaluation value E,
The optimum value S _opt of the zoom scale can be obtained in a predetermined range ± S _range .

Therefore, according to the range image generating apparatus 1, even when a camera having a zoom function is used as the reference camera 2, the optimum value S _{opt of the} zoom scale is obtained.
, A highly accurate distance image can be generated. Therefore, according to the distance image generation device 1, even if the lens parameter cannot be measured with high accuracy in response to a change in zoom, the distance measurement accuracy can be improved by _obtaining the optimum value S _opt . Therefore, according to the distance image generation device 1, it is possible to effectively utilize the zoom function of the reference camera 2 and generate a highly accurate distance image.

Next, another embodiment of the range image generating apparatus 1 to which the present invention can be applied will be described. Note that the same processes (steps) as those in the above-described embodiment are denoted by the same step numbers, and detailed descriptions thereof are omitted.

In the distance image generating apparatus 1 described above, the reference image is converted into a zoom scale equivalent to that of the detected image by performing a scale conversion process on the reference image to obtain the optimum value S.
An example in which _opt is obtained and a distance image is generated has been described. However, the distance image generation device 1 according to another embodiment is configured as shown in FIG.
Are subjected to a scale conversion process so that the detected images D1 to D4 shown in FIG. 7 have the same zoom scale as the reference image B as shown in FIG.

That is, in this embodiment, as shown in FIG. 11, the scale conversion processing in step S4 in FIG.
The distortion of the detected image is removed using the lens distortion on the zoom scale at the time of calibration, which is obtained in advance. As a result, as shown in FIG. 12A, distortion is removed from the reference image as compared to before the distortion is removed.

Next, in step S22, the scale conversion of the detected image is performed by the distance image generator 12 using the zoom scale Sv of the reference image and the image center (Xcv, Ycv). As a result, the pattern of the detected image is as shown in FIG.
As shown in the figure, the size is the same as the picture of the reference image.

Next, in step S23, using the lens distortion κ of the reference image in the zoom scale Sv and the center of the detected image (Xcv, Ycv), the detected image, which has been scale-converted in step S22, is subjected to the scale conversion. Is added by the distance image generation unit 12. As a result, as shown in FIG.
Distortion is added to a detected image having the same size as the reference image.

In this embodiment, the above-described step S4
Similarly to the above, the distance image generation processing and the evaluation value calculation processing are performed by the distance image generation unit 12 using the detected image after the scale conversion processing and the reference image. At this time, the distance image generation unit 12 generates a projective transformation matrix indicating the correspondence between the reference image and the detected image according to the zoom scale Sv of the reference image, and maps a point on the reference image to a corresponding point on the detected image. A point is obtained, a distance image is generated, and an evaluation value E is calculated using the following equation.

[0071]

(Equation 3)

Where I (x) is the luminance value of the reference image,
I ′ (x ′) is the luminance value of the detected image.

Next, the distance image generation unit 12 performs a zoom scale optimization process in the same manner as in step S5 described above. That is, the distance image generation unit 1
2 compares the evaluation value E calculated in the optimum evaluation value E _min and step S4, and converts the optimum evaluation value E _min, and the optimum value S _opt zoom scale S in the converted optimum evaluated value E _min Convert.

In step S6, the distance image generation unit 1
2 sets the zoom scale of the detected image to a predetermined range ± S _range by steps S6 and S8 as described above.
To obtain the optimum evaluation value E _min and the optimum value S _opt ,
A distance image is generated using the detected image of the zoom scale at the optimum value S _opt and the reference image.

The distance image generating apparatus 1 according to such another embodiment generates a projective transformation matrix for each S _step and generates a distance image and an evaluation value E when performing the processing corresponding to step S4. Therefore, although the processing time increases as compared with the above-described embodiment, a more accurate projection transformation matrix can be generated, and a highly accurate distance image and the evaluation value E can be generated.

The distance image generating apparatus 1 according to the other embodiment is different from the above embodiment in that steps S4 and S4 are performed.
In contrast to the necessity of performing the scale conversion in step S7, the scale conversion process of the detected image is performed in a step corresponding to step S4, and the process proceeds to step S7 using the scale-converted detected image and the reference image. Corresponding processing can be performed.

Further, in the distance image generating apparatus 1 according to another embodiment, the above-described embodiment converts the reference image into a zoom scale at the time of calibration in step S4, and performs the above-described processing in steps S4 to S6. Therefore, the number of pixels in the predetermined area of the reference image corresponding to the detected image when the distance image and the evaluation value E are generated in step S4 is S
_Since the zoom scale of the detected image is converted by the process corresponding to step S4 without changing the zoom scale of the reference image for the change in each _step , it is always optimized by using the evaluation value in a certain area of the reference image. The value S _opt can be determined.

The distance image generating apparatus 1 to which the present invention is applied can generate a distance image by optimizing the zoom scale in the same manner as described above, even if the detection camera 3 has a zoom function. . However, in this case, the sensor detection signal detection unit 10 is connected to each detection camera 3, and the relationship between the sensor signal and the zoom scale of the detection image, the relationship between the sensor signal and the lens distortion of the detection image, the sensor signal and the detection image It is necessary to prepare in advance the relationship with the image center as in the case of the reference image. Further, the distance image generation device 1
The scale conversion processing shown in FIG. 7 and / or FIG. 11 is performed on the reference image and / or the detected image by the distance image generation unit 12 so that the scale of the reference image and the detected image is substantially the same, and the reference image and the detected value Sv are obtained. S within ± S _{range
By obtaining} an evaluation value E for each _step , an optimum value S _opt is obtained to generate a distance image. Thereby, the distance image generation device 1 can generate not only a distance image indicating the distance between the reference camera 2 and the imaging target object but also a distance image indicating the distance between the detection camera 3 and the imaging target object.

In the description of the present invention, an example has been described in which a range image is generated even when the zoom state of the reference camera or the detection camera changes. However, the above-described zoom scale optimization method uses the reference camera 2 or the detection camera. The present invention can also be applied to a change in image size that occurs at the time of fine adjustment of focus, which is often performed after the zoom operation of the camera 3 is completed.

That is, the distance image generating apparatus 1 includes, for example, a reference image B and detection images D1 to D1 as shown in FIG.
In D4, only the reference image B is converted into a reference image B 'as shown in FIG. 13 (b) by performing a zoom scale conversion, and focus adjustment processing is further performed on the reference image B' to obtain the reference image B 'in FIG. 13 (c). 13C. At this time, the reference image B ″ shown in FIG.
The size is changed as compared with the reference image B ′ shown in FIG.

On the other hand, when the optimum value _Sopt of the zoom scale is obtained by changing the zoom scale in S _step units within the _range of ± S _range , for example, as shown in FIG.
The optimum value S _opt is determined according to the optimum evaluation value E _min for the actually obtained zoom scales S _{start to} S _end , and the zoom scales S _{start ′ to} S _end for the reference image B ″ after the focus is adjusted. _According to the optimal evaluation value E _{min for '
Find the} optimal S _{opt '} . Accordingly, the distance image generation device 1 generates a distance image of the image by changing the zoom scale, and even when the zoom scale of the image changes by performing focus adjustment, the optimum zoom scale is S _{opt ′.} And a distance image can be generated.

[0082]

As described above in detail, the range image generating apparatus according to the present invention includes one camera having a zoom function.
An imaging unit having a plurality of cameras, a zoom state detection unit that detects a zoom state of a camera having a zoom function and calculates at least a zoom scale, and changes a zoom scale of an image captured by any of the cameras. A scale conversion unit, and generates an evaluation value indicating a similarity by comparing the image on which scale conversion has been performed by the scale conversion unit and a point on an image on which scale conversion has not been performed corresponding to a point on the image. A zoom scale detecting means for detecting an appropriate zoom scale, an image on the zoom scale detected by the zoom scale detecting means, and an image on which no scale conversion has been performed, using one of the camera and the imaging object. And a distance image generating means for generating a distance image indicating the distance of the camera. Even if, to detect the zoom state,
Estimate accurately so that it becomes the appropriate zoom scale,
A highly accurate distance image can be generated.

Further, the distance image generation method according to the present invention
Among a plurality of cameras including one or more zoom cameras that generate an image by capturing an image of an object to be captured using a zoom function, a zoom state of the zoom camera is detected, at least a zoom scale is calculated, and a periphery of the zoom camera is calculated. A first step of generating an image by another camera disposed in the camera, and changing the scale of the zoom image or the image generated by the other camera within a predetermined range, and A second step of comparing a point on the image generated by the other camera corresponding to the point to generate an evaluation value indicating the degree of similarity, and using a zoom image and an image generated by the other camera. A third step of adjusting the zoom scale based on the generated evaluation value, and the other camera corresponding to a point on the zoom image on the adjusted zoom scale. A fourth step of detecting a corresponding point on the formed image and generating a distance image indicating a distance between the zoom camera and the imaging target, so that a zoom state of the zoom camera is changed to capture a zoom image. Even when the zooming is performed, the zoom state is detected, the zoom scale is accurately estimated, and a high-precision indicating the distance between the zoomed image and the imaging target is obtained using the zoomed image and an image generated by another camera. It is possible to generate an accurate distance image.

[Brief description of the drawings]

FIG. 1 is a perspective view showing an example of the arrangement of a reference camera and a detection camera constituting a range image generation device to which the present invention can be applied.

FIG. 2 is a configuration diagram showing a range image generation device to which the present invention can be applied.

FIG. 3 is a diagram for comparing and explaining a reference image and a detected image when a reference image is captured while changing the zoom state of the reference camera.

FIG. 4 is a diagram illustrating a relationship between an output voltage generated by a sensor signal detection unit and a zoom scale of a reference image.

FIG. 5 is a flowchart illustrating a process of a distance image generation unit when a distance image is generated by a distance image generation device to which the present invention can be applied.

6A and 6B are diagrams for explaining that a scale conversion process is performed on a reference image. FIG. 6A illustrates a reference image and a detected image before the scale conversion process, and FIG. 6B illustrates a reference image after the scale conversion process. And a detected image.

FIG. 7 is a flowchart illustrating a process of performing a scale conversion process on a reference image.

8A is a diagram showing a reference image on which a scale conversion process is performed according to the flowchart shown in FIG. 7; FIG.
Shows reference images before and after distortion removal, (b) shows reference images before and after scale conversion, and (c) shows reference images before and after distortion addition.

FIG. 9 is a diagram showing evaluation values when the zoom scale is changed within a predetermined range ± S _range .

10A and 10B are diagrams for explaining that a scale conversion process is performed on a detected image. FIG. 10A illustrates a detected image and a reference image before the scale conversion process, and FIG. 10B illustrates a detected image after the scale conversion process. And a reference image.

FIG. 11 is a flowchart illustrating a process of performing a scale conversion process on a detected image.

12 is a diagram showing a reference image on which a scale conversion process is performed according to the flowchart shown in FIG. 11;
(A) shows a detected image before and after distortion removal, (b) shows a detected image before and after scale conversion, and (c) shows a detected image before and after distortion addition.

13A shows a reference image and a detected image before a scale conversion process, FIG. 13B shows a reference image and a detected image after a scale conversion process, and FIG. 13C shows a reference image and a detected image after focus adjustment. It is a figure showing an image.

FIG. 14 is a diagram illustrating an evaluation value when the zoom scale is changed within ± S _range within a predetermined range, and an evaluation value according to the zoom scale changed by performing focus adjustment.

FIG. 15 is a diagram for explaining that a point on the detected image corresponding to a point on the reference image is searched for and detected from the epipolar line.

FIG. 16 is a diagram illustrating a relationship between a reference image and a detected image when a point on the detected image corresponding to a point on the reference image is searched for from the epipolar line and detected.

[Explanation of symbols]

Reference Signs List 1 distance image generation device, 2 reference camera, 3 detection camera, 10 sensor signal detection unit, 12 distance image generation unit

Claims (12)

[Claims] 1. An image pickup means comprising one or more cameras having a zoom function and comprising a plurality of cameras, and a zoom state detection means for detecting a zoom state of the camera having a zoom function and calculating at least a zoom scale. Scale conversion means for changing the zoom scale of an image captured by the camera, and a point on the image on which scale conversion has been performed by the scale conversion means and a point on the image which has not been subjected to scale conversion corresponding to a point on the image. A zoom scale detecting unit that generates an evaluation value indicating the degree of similarity and detects an appropriate zoom scale, an image at the zoom scale detected by the zoom scale detecting unit, and an image that has not been subjected to scale conversion. And a distance image generating means for generating a distance image indicating a distance between one of the cameras and the object to be imaged. Distance image generating apparatus according to claim Rukoto. 2. The zoom state detecting means detects a zoom state of a camera having the zoom function and detects an image center and a lens distortion of an image taken, and the scale converting means includes a step of detecting the image center and the lens distortion. The distance image generating apparatus according to claim 1, wherein the scale conversion of the image is performed by using. 3. An image pickup means, comprising: a first camera having a zoom function for picking up an image of an object to be picked up and generating a first image; And a plurality of second cameras for generating a second image by capturing an image of the object to be captured.
The distance image generation device according to claim 1. 4. The zoom scale of the first image detected by the zoom state detection means is converted by the scale conversion means, and the scale-converted first image is compared with the second image. An evaluation value is generated, an appropriate zoom scale is detected by a zoom scale detection unit, and a distance image is generated by the distance image generation unit using the first image and the second image of the appropriate zoom scale. The distance image generation device according to claim 3, wherein 5. The scale of the second image is converted by the scale conversion means, the scaled second image is compared with the first image, an evaluation value is generated, and an appropriate zoom scale is calculated. 4. The distance image generation according to claim 3, wherein the distance image is detected by the zoom scale detection means, and the distance image is generated by the distance image generation means using the second image and the first image of the appropriate zoom scale. apparatus. 6. A scale conversion is performed on each image generated by two or more cameras having a zoom function, and an appropriate zoom scale is detected by a zoom scale detection unit using each image subjected to the scale conversion. The distance image generation device according to claim 1, wherein a distance image indicating a distance from the camera to the imaging target is generated by a distance image generation unit. 7. Among a plurality of cameras including one or more zoom cameras that generate an image by capturing an image of an object to be captured using a zoom function, a zoom state of the zoom camera is detected to calculate at least a zoom scale. A first step of generating an image by another camera arranged around the zoom camera, and changing the scale of the zoom image or the image generated by the other camera within a predetermined range, thereby obtaining the zoom image And a second step of comparing the point on the zoom image with a point on the image generated by the other camera to generate an evaluation value indicating the degree of similarity. A third step of adjusting the zoom scale based on the evaluation value generated using the generated image, and a point on the zoom image at the adjusted zoom scale. A fourth step of detecting a corresponding point on an image generated by the other camera corresponding to the above, and generating a distance image indicating a distance between the zoom camera and the imaging target object. Generation method. 8. The first step detects a zoom state of the zoom camera to detect an image center and a lens distortion of a zoom image on the zoom scale, and the second step includes a step of detecting the image center and the lens distortion. The distance image generation method according to claim 7, wherein the zoom scale of the first image or the second image is changed by using (1). 9. The second step calculates an image center and a lens distortion from the zoom state detected in the first step, and removes distortion of the zoom image or an image generated by the other camera. The scale conversion is performed by changing the scale of the zoom image or the image generated by the other camera, and a lens distortion corresponding to the scale conversion is added to the zoom image, and the lens distortion-added zoom image and the other 9. The distance image generation method according to claim 8, wherein the evaluation value is generated using an image generated by the camera. 10. The method according to claim 1, wherein the second step includes converting a scale of the zoom image, and generating an evaluation value using the zoom image having the converted scale and an image generated by the other camera. The method for generating a distance image according to claim 7. 11. The second step is characterized in that a scale of an image generated by the other camera is converted, and an evaluation value is generated by using the converted image and the zoom image. The method for generating a distance image according to claim 7. 12. The second step is characterized in that scale conversion is performed on each zoom image generated by two or more zoom cameras, and an evaluation value is generated using each scale-converted image. The method for generating a distance image according to claim 7.