JP4957134B2 - Distance measuring device - Google Patents

Description

  The present invention relates to a distance measuring device including stereo imaging means and stereo image processing means.

  As background art of this technical field, for example, there is JP-A-5-130646 (Patent Document 1). In this publication, for the purpose of “automatically correcting an optical axis shift or the like generated in the optical system of the imaging apparatus and obtaining a stereoscopic image under an appropriate imaging condition”, as a solving means, “the above-mentioned problem”. In order to solve the above-described problem, the first invention uses a plurality of imaging devices having an imaging condition setting unit, and is a stereoscopic camera device capable of sensing a stereoscopic image to be observed through a stereoscopic image processing unit. A position detection unit that detects the position of the imaging device, a correction data storage unit that stores an imaging condition correction value, and an imaging that stores the position signal detected by the position detection unit in the correction data storage unit A positional deviation correction unit that corrects with a condition correction value, and performs positional deviation correction control of the imaging apparatus so as to obtain an appropriate imaging condition with reference to the imaging condition correction value. The second invention is the imaging condition setting. In a stereoscopic camera device that uses a plurality of imaging devices having a unit and can sense a stereoscopic image to be observed through a stereoscopic image processing unit, the imaging condition setting unit detects a position of the imaging device An image storage unit for storing a stereoscopic image captured by each imaging device, an image analysis unit for comparing and calculating image color information or density information from the image storage unit, and an imaging condition correction value from the calculation result A correction value calculation unit to be created and stored; and a position shift correction unit that corrects the position signal detected by the position detection unit with the imaging condition correction value of the correction value calculation unit, and is sampled by the image analysis unit. The imaging condition correction value is created and updated based on the captured image data, and the positional deviation correction control of the imaging apparatus is performed so that the appropriate imaging condition is obtained at each imaging with reference to the imaging condition correction value. Is. It discloses a technique that.

JP-A-5-130646

  In the above-mentioned Patent Document 1, in a stereo camera or a multi-camera having a plurality of imaging units for performing stereoscopic viewing, the optical axis shift or zoom amount difference that occurs between the cameras when the zoom changes is corrected. It is possible to gain a sense. However, in order to perform distance measurement with a stereo camera, it is necessary not only to correct the relative shift amount between the cameras but also to acquire the value of the focal length of the camera at that zoom.

  Therefore, in a distance measurement device using stereo image processing, when the zoom changes, the amount of change in the focal length is automatically calculated to update the focal length value, and there is a zoom shift between the cameras. Therefore, it is necessary to continue accurate distance measurement even if the zoom changes by automatically correcting this.

  An object of the present invention is to improve the accuracy of distance measurement in a distance measuring device.

  The above object is achieved by the present invention described in the claims.

  An outline of one aspect of the present invention is to detect the amount of zoom change in each of a plurality of image pickup means, and when they are different, correct the zoom position so that they match, and output images from the plurality of image pickup means The distance is measured by performing stereo image processing on the information.

  According to the present invention, accurate distance measurement is possible even when the zoom is changed in the stereo distance measuring apparatus.

  Problems, configurations, and effects other than those described above will become apparent from the following description of embodiments.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a schematic diagram showing a stereo distance measuring system according to a first embodiment of the present invention. In FIG. 1, 0101 is a stereo imaging unit, 0102 is a stereo image processing unit, 0103 is a stereo imaging control unit, and 0104 is a zoom change amount detection unit.

  In the stereo distance measurement system, the stereo imaging unit 0101 has two imaging units including a lens group capable of zooming and focusing, a shutter, an iris, an imaging device, AGC, AD, a camera signal processing DSP, a timing generator, and the like. In each imaging unit, imaging is performed by photoelectric conversion, and stereo image data is output. A CCD or CMOS may be used as an image sensor used for photoelectric conversion. In addition, the stereo imaging unit 0101 may be configured using a mirror or a prism, and may have one imaging unit including one imaging element. In addition, the stereo imaging unit 0101 may be configured as a multi-imaging unit that includes three or more imaging units and outputs a plurality of captured images. The stereo image processing unit 0102 performs stereo image processing using the stereo image data output from the stereo imaging unit 0101, and calculates and outputs distance information and information reliability. Stereo image processing performed by the stereo image processing unit 0102 includes calibration processing such as lens distortion correction processing, inter-image magnification correction processing, and parallelization processing, pre-processing such as low-pass filter processing for noise removal, Disparity information is acquired by searching for corresponding points between stereo images using feature calculation processing such as edge detection, block matching such as normalized cross-correlation and sum of absolute differences, and various correlation calculation processing such as space-sweep method. Stereo matching processing, post-processing that removes singularities by rank filter processing, labeling, etc., distance calculation processing that calculates distance information using parallax information, etc., information obtained in the middle of processing, for example, stereo matching Brightness information, parallax information, evaluation values, i.e. performance of template images and matching images when corresponding point search is performed in the processing. And scores, the distribution of evaluation values in the search area can be output. The stereo imaging control unit 0103 determines a zoom change amount and a change direction by a control based on a user operation in an operation unit (not shown), an image recognition result in an image processing unit (not shown), and the like. Controls the lens group that can be focused. Examples of driving the lens group include a stepping motor and an ultrasonic motor. In order to suitably perform stereo image processing, it is desirable to perform control so that the zoom magnification is equal between the two imaging units. However, in practice, the zoom magnification does not always change evenly by controlling the lens position by an equal amount between a plurality of imaging units due to individual variations in lens characteristics and the accuracy of the structure. Therefore, the relationship between the zoom and focus lens control amount and the zoom magnification change amount may be stored in the data table at the time of shipment from the factory, and the zoom magnifications of the two imaging units may be changed by an equal amount during zoom control. At this time, if only some representative relationships are stored in the data table and the lens is controlled only by values not stored in the data table, neighboring values stored in the data table are stored. The amount of stored data may be reduced by performing interpolation using. Further, the stereo imaging control unit 0103 may be provided with a device that detects the amount of zoom change, and may be output to the stereo image processing unit 0102 as zoom control information. However, the zoom control information output from the stereo imaging control unit 0103 may not be accurate enough to be used for stereo image processing. The zoom change amount detection unit 0104 performs image processing on the stereo image data output from the stereo image pickup unit 0101 to detect a change in focal length due to a change in zoom or focus and a difference in focal length between stereo image pickup. The stereo image processing unit 0102 uses the zoom control information acquired from the stereo imaging control unit 0103 and the focal length change information acquired from the zoom change amount detection unit 0104 individually or in combination, and is necessary for distance calculation. Calculate the parameters.

  FIG. 2 is a first diagram illustrating an example of a focal length change amount detection process according to the first embodiment of the present invention. 2 is executed by the zoom change amount detection unit 0104.

  In FIG. 2, 0201_1 is a first captured image before zoom change, 0202_1 is a second captured image before zoom change, 0201_2 is a first captured image after zoom change, and 0202_2 is a second captured image after zoom change. It is an image. However, the first captured image and the second captured image indicate captured images output from the imaging unit of the stereo imaging unit 0101, respectively.

  The zoom change amount detection unit 0104 includes a parallax d1 in the stereo image before the zoom change of the feature point 0203_1 of the stationary object in the first captured image 0201_1 before the zoom change and a space in the space from the camera to the feature point 0203_1 of the still object. The distance Z is acquired from the stereo image processing unit. In the detection of the feature point 0203_1 of the still object before the zoom change, the detection of the still object is performed by comparing the first captured image 0201_1 before the zoom change with the captured image of one frame or a plurality of frames in the past. This can be done by computing. For feature point detection, an edge enhancement filter such as Prewitt or Sobel or a corner extraction filter such as a Harris operator can be used. Also, the parallax d1 in the stereo image of the feature point 0203_1 of the stationary object and the distance Z in the space from the camera to the feature point 0203_1 of the stationary object are not acquired from the stereo image processing unit 0102, but the zoom change amount detection unit You may calculate by 0104. In this case, the detection of the focal length change amount can be performed independently of the stereo image processing, and thus the obtained information on the focal length change amount can be used during the stereo image processing to perform the stereo image processing with high accuracy. it can. Next, the feature point 0203_2 corresponding to the feature point 0203_1 of the stationary object is detected in the first captured image 0201_1 after the zoom change, and the parallax d2 in the stereo image after the zoom change of the feature point 0203_2 is acquired from the stereo image processing unit. To do. The feature points after the zoom change corresponding to the feature points of the stationary object before the zoom change are performed by a feature tracking method using the KLT method or an optical flow, or by autocorrelation before and after the zoom change in the image of the same imaging unit. It can be obtained by evaluating the correlation value. Further, the parallax d2 in the stereo image of the feature point 0203_2 of the stationary object may be calculated by the zoom change detection unit 0104 instead of being acquired from the stereo image processing unit 0102.

  The zoom change amount detection unit 0104 utilizes the fact that the distance Z in the space from the camera to the feature point of the stationary object is unchanged before and after the zoom change, and the parallax d1 in the stereo image before the feature point zoom change Then, the focal length change amount is calculated from the parallax d2 in the stereo image after the zoom change of the feature point and the distance Z from the camera to the feature point. Although FIG. 2 shows a case where feature points of one stationary object are used, a plurality of feature points of stationary objects are detected from a stereo image, and a histogram is created by calculating the amount of change in focal length for each. As a result of evaluation, it is possible to reduce the influence of erroneous detection in detection of a still region and erroneous response in matching processing, and improve the accuracy of calculation of the focal length change amount. In this way, the amount of change in focal length can be calculated by using parallax information between captured images of feature points of a stationary object, and accurate distance measurement can be performed using the correct focal length even when the zoom changes. Is possible.

  FIG. 3 is a second diagram illustrating an example of the focal length variation detection process according to the first embodiment of the present invention. The focal length change amount detection process in FIG. 3 is executed by the zoom change amount detection unit 0104.

  In the focal length variation detection process shown in FIG. 2, the feature points for calculating the focal length variation are limited to the feature points of a stationary object. On the other hand, in this method, a large number of arbitrary feature points are detected from the stereo image, and for each feature point, the parallax in the stereo image before the zoom change, the parallax in the stereo image after the zoom change, and from the camera to the feature point The amount of change in focal length is estimated using the distance in the space. The zoom variation detection unit 0104 detects the correct focal length variation by creating and evaluating a histogram as shown in FIG. 3 with the estimated focal length variation on the horizontal axis and the frequency on the vertical axis. To do. In FIG. 3, A1 is an estimated focal length change amount of a feature point whose distance does not change at the time of zoom change such as a feature point of a stationary object 0301 or a background region (not shown), that is, a correct focal length change amount. A2 is an estimated focal length change amount of a feature group whose distance changes further from the camera when the zoom changes, such as a feature point of the first moving object 0302, and A3 is a zoom change such as a feature point of the second moving object 0303. This is the estimated focal length change amount of the feature group whose distance changes closer to the camera at the time. When the captured image includes many stationary areas such as the background or when there are few moving objects, the frequency of A1 in the histogram is higher than the frequencies of A2 and A3. The distance change amount can be acquired. When creating a histogram, create a histogram that weights the amount of change in focal length estimated from the distance from the center of the image for each feature point, and feature points that deviate from the angle of view when the zoom changes to a wide angle. The influence of may be reduced. In addition, when creating a histogram, a histogram is created by weighting the amount of change in focal length estimated from the correlation calculation result during feature tracking, and errors may occur due to feature points hidden by occlusion and changes in appearance. You may reduce the influence of the tracked feature point.

  In the stationary object detection process, a stationary object is detected at the time when a stereo image before the zoom change is acquired, and a correct focal length change amount is obtained for an object that suddenly moved during the zoom change and the distance changed. Although there is a possibility that it has not been possible, with this method, it is possible to reduce erroneous calculation of the focal length change amount due to the influence of an object whose distance changes when the zoom changes.

  2 or 3, when the stereo control imaging unit 0103 controls the zoom of the stereo imaging unit 0101, the focal length variation calculation process notifies the zoom variation detection unit 0104 of a control start signal and a control end signal. Thus, processing efficiency can be improved by performing processing only for a certain period of time including before and after zooming of the stereo imaging unit 0101. Further, if the stereo image output from the stereo imaging unit 0101 is processed for each frame, and the change amount is output to the stereo image processing unit 0102 only when the focal length is changed, the zoom or zoom can be performed by vibration or impact. It is possible to detect the amount of change in focal length when the focus changes.

  Thus, according to the present embodiment, in the stereo distance measurement processing, even when the zoom of the stereo imaging unit changes, the focal length is calculated using the parallax information before the zoom change and the parallax information after the zoom change of the stationary subject. Accurate distance measurement can be performed by calculating the amount of change and using the correct focal length value.

  As described in the first embodiment, when the zoom is changed in the stereo distance measuring device, it is possible to accurately measure the distance by detecting the amount of change in the focal length and updating the value of the focal length. However, in reality, the zoom magnification does not always change to an equal amount between a plurality of imaging units due to insufficient lens characteristics and structural accuracy. In addition, even when used in zoom control, a data table that can be controlled so that the zoom magnification changes to an equal amount is taken into consideration at the time of shipment from the factory. There are cases where the control data required for the change in the equivalent amount deviates from the stored value of the data table. In some cases, the zoom magnification may change due to vibration or shock caused by contact. In such a case, it may be solved by the following algorithm.

  FIG. 4 is a schematic diagram showing a stereo distance measuring system according to the second embodiment of the present invention. In FIG. 4, 0401_1 is a first imaging unit, and 0401_2 is a second imaging unit.

  The first imaging unit 0401_1 includes a lens, a shutter, an iris, an imaging device, AGC, AD, and the like, and performs imaging by photoelectric conversion and outputs imaging data. The second imaging unit 0401_2 includes a lens, a shutter, an iris, an imaging device, AGC, AD, and the like, and performs imaging by photoelectric conversion and outputs imaging data.

  FIG. 5 is a diagram showing an example of a processing sequence of the stereo distance measuring system according to the second embodiment of the present invention. The focal length change amount calculation processing sequence shown in FIG. 5 is executed by the zoom change amount detection unit 0104.

  In the focal length variation calculation processing sequence in FIG. 5, in ST0501, the focal length variation of the first imaging unit 0401_1 is calculated using monocular image processing. The focal length change amount detection process will be described later. In ST0502, the focal length change amount of the second imaging unit 0401_2 is calculated using monocular image processing. In ST0503, the focal length change amount of the first imaging unit 0401_1 and the focal length change amount of the second imaging unit 0401_2 are output to the stereo imaging control unit 0103. The stereo imaging control unit 0103 compares the focal length change amount of the first imaging unit 0401_1 with the focal length change amount of the second imaging unit 0401_2, and calculates the zoom change amount necessary for correcting the zoom difference. The change direction is determined, and the zoom difference is corrected by controlling the zoom and focus of the first imaging unit 0401_1 or the second imaging unit 0401_2. At this time, if there is a main imaging unit used for recording or monitoring as the imaging unit that controls the zoom, the other imaging unit may be selected so as to match the zoom of the main imaging unit. Further, the two imaging units may be controlled so as to be weighted according to the current zoom magnification and matched with a specific zoom magnification between the zoom magnifications of the two imaging units. In addition, the correction of the zoom magnification shift may be corrected not only by optically correcting the zoom and focus lens positions but also by using an electronic zoom in the imaging unit. In ST0504, the stereo image processing unit 0102 is notified of the focal length change amount of the first imaging unit 0401_1 and the focal length change amount of the second imaging unit 0401_2. The stereo image processing unit 0102 is necessary for distance measurement based on the focal length information before the zoom change, the focal length change amount of the first imaging unit 0401_1, and the focal length change amount of the second imaging unit 0401_2. Calculate the focal length value. Further, the difference in zoom magnification continues to exist in the stereo image obtained after the difference in zoom magnification between the image capturing units is detected in ST0503 until the stereo image capturing control unit 0103 performs the correction process. Become. Therefore, if there is such an error, the stereo image processing unit 0102 may perform magnification correction by electronic zoom based on the information notified in ST0504 as preprocessing of stereo image processing. In this case, even when there is an error in zoom magnification between the imaging units, it is possible to perform stereo image processing such as accurate matching processing, and the accuracy of distance information to be measured is improved. Furthermore, even when the zoom is changing, it is possible to correct the zoom magnification difference between the captured images and continue the highly accurate distance measurement. Note that the image whose magnification is to be corrected may be determined based on the same standard as the imaging unit that performs zoom control in ST0503.

  FIG. 6 is a first diagram illustrating an example of processing for detecting the focal length change amount according to the second embodiment of the present invention. 6 corresponds to ST0501 and ST0502 in FIG. 5, and an image obtained by the first imaging unit 0401_1 and a second imaging unit 0401_2 in the zoom variation detection unit 0104 are obtained. This process is performed for each image.

  In FIG. 6, 0601_1 is a captured image before zoom change, 0601_2 is a captured image after zoom change, and 0602_1 and 0603_1 are feature points in which the distance in the optical axis direction from the imaging unit is equidistant in the captured images 0601_1 before zoom change. The stationary feature points 0602_2 and 0603_2 are the feature points corresponding to 0602_1 and 0603_1 in the captured image 0601_2 after the zoom change, respectively. Now, the relationship between the interval a1 between the feature points 0602_1 and 0603_1 before the zoom change and the interval a2 between the corresponding feature points 0602_2 and 0603_2 after the zoom change is the distance in the optical axis direction in the space from the imaging unit to the feature point. This can be uniquely expressed using Z, the focal length f1 before the zoom change, and the focal length f2 after the zoom change. Therefore, a fixed feature point whose distance from the imaging unit is known and equidistant from the imaging unit is extracted from the image, and the focal length change amount accompanying the zoom change is obtained by using the feature point interval. can do. When the distance Z in the optical axis direction in the space from the imaging unit to the feature point is sufficiently larger than the focal lengths f1 and f2, the interval between the feature points that are equidistant from the imaging unit is substantially proportional to the focal length. By using this approximation, the processing amount may be reduced without using the distance Z in the space from the imaging unit to the feature point. Further, when this recent time can be used, the two feature points can ignore the restriction that the distance in the optical axis direction from the imaging unit is the same distance, and any two points whose distances are known. Therefore, the dependence on the shooting environment is reduced. Although two feature points are used in FIG. 6, accuracy may be improved by using a plurality of feature points of three or more. The stationary feature point detection and the feature point tracking process when the zoom is changed may be performed in the same manner as the first example of the focal length change detection process in the first embodiment. As described above, by using the interval between two or more feature points whose distance in the optical axis direction from the imaging unit is known, it is possible to acquire the focal length change amount at a high speed with a small amount of calculation.

  FIG. 7 is a second diagram showing an example of processing for detecting the focal length change amount according to the second embodiment of the present invention. 7 corresponds to ST0501 and ST0502 in FIG. 5, and the zoom variation detection unit 0104 obtains the image obtained by the first imaging unit 0401_1 and the second imaging unit 0401_2. This process is performed for each image.

  In FIG. 7, 0701 is a captured image before zoom change, 0702 is a captured image after zoom change, and 0703 is an image obtained by enlarging and reducing the captured image 0702 after zoom change by n times in the zoom change amount detection unit 0104 by image processing. It is. The zoom change amount detection unit 0104 performs matching processing between the captured image 0703 after the zoom change and the captured image 0701 before the zoom change, which has been multiplied by n while changing the value of n, and evaluates the correlation value to thereby adjust the focus value. The distance change amount can be acquired from the value of n. In the matching process, a correlation value may be calculated for the entire image, or a correlation value may be calculated for an arbitrarily sized block near the center of the image that is not easily affected by the zoom change. As shown in FIG. 7C, when n becomes equal to the zoom magnification, the captured image 0703 after the zoom change that has been multiplied by n and the captured image 0701 before the zoom change have the highest correlation. Here, the enlargement / reduction process is performed on the captured image after the zoom change, but the matching process may be performed after the enlargement / reduction process is performed on the captured image before the zoom change. In the enlargement / reduction process by image processing, an image with higher accuracy can be obtained by the reduction process than the enlargement process. Therefore, the reduction process may be performed with priority given to a wide-angle image. Further, the approximate zoom amount of each imaging unit may be acquired from the zoom control information of the stereo imaging control unit 0103, and the enlargement / reduction process may be performed with a value in the vicinity thereof as an initial value when n is changed.

  Thus, by calculating the correlation value by the matching process by enlarging / reducing the entire image, it is possible to acquire the focal length change amount without performing the process of evaluating and selecting a specific feature point. .

  FIG. 8 is a third diagram showing an example of processing for detecting the focal length change amount according to the second embodiment of the present invention. 8 corresponds to ST0501 and ST0502 in FIG. 5, and an image obtained by the first imaging unit 0401_1 and a second imaging unit 0401_2 in the zoom variation detection unit 0104 are obtained. This process is performed for each image.

  In FIG. 8, 0801_1 is a captured image before zoom change, 0801_2 is a captured image after zoom change, 0802_1 is a stationary feature point in the captured image 0801_1 before zoom change, and 0802_2 is a feature point 0802_1 in the captured image 0801_2 after zoom change. Is a feature point corresponding to. The relationship between the distance b1 between the feature point 0802_1 before the zoom change and the image center and the distance b2 between the corresponding feature point 0802_2 and the image center after the zoom change is the distance in the optical axis direction from the power imaging unit to the feature point. This can be uniquely expressed using Z, the focal length f1 before the zoom change, and the focal length f2 after the zoom change. Therefore, by extracting a feature point whose distance from the imaging unit is known and stationary, it is possible to acquire the amount of change in the focal length due to the zoom change by using the interval between the feature point and the image center. . When the distance Z in the space from the imaging unit to the feature point is sufficiently larger than the focal lengths f1 and f2, the interval between the feature point and the image center is substantially proportional to the focal length. By using this approximation, the processing amount may be reduced without using the distance Z in the space from the imaging unit to the feature point. Although one feature point is used in FIG. 8, the accuracy may be improved by using two or more feature points. The coordinate value of the center of the image can be used by approximating the x coordinate and the y coordinate with values obtained by dividing the image width and the image height by half. However, in actuality, the value of the image center coordinate varies slightly from image pickup unit to image pick-up unit. Therefore, an accurate image center coordinate value may be measured at the time of shipment from the factory and stored in a memory or the like. The stationary feature point detection and the feature point tracking process when the zoom is changed may be performed in the same manner as the first example of the focal length change detection process in the first embodiment. As described above, it is difficult to detect a plurality of feature points that are equidistant from the imaging unit by using an interval between one or more feature points whose distance from the imaging unit is known and the image center. The focal length change amount can be acquired even when the method shown in FIG. 7 cannot be applied, or when there are many moving objects in the screen and the method shown in FIG. 7 cannot be applied.

  The focal length change amount measurement processing shown in FIGS. 6 to 8 may be used alone or in combination depending on the shooting environment and processing capability. At this time, for example, a value selected by weighting the focal length change amount according to the reliability of the feature point tracking or the correlation value at the time of the matching process may be used.

  As described above, according to the present embodiment, in the stereo distance measurement process, even when a zoom magnification error occurs between the imaging units when the zoom of the stereo imaging unit changes, the required amount of change in the focal length for each camera. By calculating the above, correcting the zoom magnification error between the imaging units, and using the correct focal length value, accurate distance measurement can be continued. Further, in this embodiment, since the focal length change amount is acquired using the captured image of the monocular imaging unit, the appearance between the imaging units is compared with the acquisition processing of the focal length change amount using the stereo image processing. It is difficult to be affected by the difference, and it is possible to obtain a more accurate focal length change amount.

  In the second embodiment, the zoom magnification difference is corrected by calculating the zoom change amount of the first image capturing unit 0401_1 and the zoom change amount of the second image capturing unit 0401_2 in the stereo distance measurement system. In distance measurement by stereo image processing, the zoom magnification difference between the imaging units greatly contributes to the accuracy. However, in the method described in the second embodiment, the zoom change amount calculation is performed in one imaging unit or both imaging units. If there is an error in the result, the zoom magnification difference may not be accurately corrected. In addition, when zoom control is repeatedly performed, errors generated when calculating the zoom change amount may be accumulated. Therefore, in order to correct the zoom magnification difference between the imaging units with high accuracy, the following algorithm may be used.

  FIG. 9 is a diagram showing an example of a processing sequence of the stereo distance measuring system according to the third embodiment of the present invention. The difference calculation processing sequence between the imaging units having the focal length shown in FIG. 9 is executed by the zoom change amount detection unit 0104.

  In the calculation processing sequence of the difference between the imaging units at the focal length in FIG. 9, in ST0901, the first captured image acquired from the first imaging unit 0401_1 and the second captured image acquired from the second imaging unit 0401_2. Are used to calculate the difference in focal length between the first imaging unit 0401_1 and the second imaging unit 0401_2. The description of the focal length difference calculation process will be described later. In ST0902, the difference information of the focal length between the first imaging unit 0401_1 and the second imaging unit 0401_2 is output to the stereo imaging control unit 0103. The stereo imaging control unit 0103 uses the difference information of the focal length between the first imaging unit 0401_1 and the second imaging unit 0401_2 to determine the zoom change amount and the change direction necessary for correcting the zoom difference. Then, the zoom difference is corrected by controlling the zoom and focus of the first imaging unit 0401_1 or the second imaging unit 0401_2. At this time, if there is a main imaging unit used for recording or monitoring as the imaging unit that controls the zoom, the other imaging unit may be selected so as to match the zoom of the main imaging unit. Further, the two imaging units may be controlled so as to be weighted according to the current zoom magnification and matched with a specific zoom magnification between the zoom magnifications of the two imaging units. In addition, the correction of the zoom magnification shift may be corrected not only by optically correcting the zoom and focus lens positions but also by using an electronic zoom in the imaging unit. In ST0903, the stereo image processing unit 0102 is notified of the difference information of the focal length between the first imaging unit 0401_1 and the second imaging unit 0401_2. The stereo image processing unit 0102 performs magnification correction by electronic zoom as pre-processing for stereo image processing, using the difference information of the focal length between the first imaging unit 0401_1 and the second imaging unit 0401_2. In this case, even when there is an error in zoom magnification between the imaging units, it is possible to perform stereo image processing such as accurate matching processing, and the accuracy of distance information to be measured is improved. Furthermore, even when the zoom is changing, it is possible to correct the zoom magnification difference between the captured images and continue the highly accurate distance measurement. Note that the image whose magnification is to be corrected may be determined based on the same standard as the imaging unit that performs zoom control in ST0503. In addition, the stereo image processing unit 0102 calculates the focal length value necessary for distance measurement after correcting the zoom magnification between the captured images. The calculation of the value of the focal length may be performed using a feature whose distance is known from the captured image and disparity information on the feature. As this distance information, a distance measurement result previously calculated by the stereo image processing unit 0102 can be used. Also, if the error due to the zoom correction process accumulates and the reliability of the distance measurement result deteriorates, or if the field of view of the imaging unit changes and the existing distance measurement result cannot be used, distance information input (not shown) The user's input from the unit or distance information input by another distance sensor may be used.

  As described above, the difference in the focal length is calculated using the captured image acquired by the first imaging unit 0401_1 and the captured image acquired by the second imaging unit 0401_2, and the zoom magnification difference between the imaging units is detected. By doing so, it is possible to detect the zoom magnification difference with high accuracy compared to the method shown in the second embodiment because the captured images between the imaging units are directly compared. Therefore, the method shown in the present embodiment may be used as preprocessing, postprocessing, or linked processing of the processing shown in the first embodiment or the second embodiment. In this case, by combining the zoom magnification difference detection process between the imaging units shown in the present embodiment and the zoom change amount detection process shown in the first embodiment or the second embodiment, Distance measurement by high-precision stereo image processing can be performed.

  Further, in this embodiment, compared with the method shown in the second embodiment, the amount of memory used can be reduced because there are few captured images used for image processing.

  FIG. 10 is a first diagram illustrating an example of a process for detecting focal length difference information according to the third embodiment of the present invention. The focal length difference information detection processing in FIG. 10 is processing performed in the zoom change amount detection unit 0104.

  In FIG. 10, 1001 is a first captured image that can be acquired from the first imaging unit 0401_1, 1002 is a second captured image that can be acquired from the second imaging unit 0401_2 having a zoom magnification different from that of the first imaging unit 0401_1, Reference numeral 1003 denotes an image obtained by enlarging and reducing the second captured image by n times in the zoom change amount detection unit 0404 by image processing. The zoom change amount detection unit 0104 performs matching processing between the second captured image 1003 and the first captured image 1001 multiplied by n while changing the value of n, and evaluates the correlation value, thereby changing the focal length. The quantity can be obtained from the value of n. However, since there is a parallax due to the difference in imaging position between the first captured image 1001 and the second captured image 1003 multiplied by n, matching processing is performed while shifting the two images in the parallax direction. Get the maximum correlation value. In the matching process, the correlation value may be calculated for the entire image, or in order to reduce the influence of the occlusion and the difference in appearance due to the difference in the imaging position, the correlation value is calculated by dividing into blocks of any size. Also good. As shown in FIG. 10, when n is equal to the zoom magnification, the second captured image 1003 and the first captured image 1001 that have been multiplied by n obtain the highest maximum correlation value. Here, the matching process is performed after the enlargement / reduction process is performed on the second captured image. However, in the enlargement / reduction process based on the image process, an image with a higher accuracy than the enlargement process can be obtained. Reduction processing may be performed with priority given to images. Further, the approximate zoom amount of each imaging unit may be acquired from the zoom control information of the stereo imaging control unit 0103, and the enlargement / reduction process may be performed with a value in the vicinity thereof as an initial value when n is changed. In this way, by calculating the correlation value by the matching process by enlarging / reducing the entire image, the process of evaluating and selecting a specific feature point can be performed at high speed without performing the process of selecting a specific feature point. Acquisition can be done.

  FIG. 11 is a second diagram illustrating an example of a process for detecting focal length difference information according to the third embodiment of the present invention. The focal length difference information detection processing in FIG. 11 is processing performed in the zoom change amount detection unit 0104.

  In FIG. 11, 1101_1 is a first captured image that can be acquired from the first imaging unit 0401_1, 1101_2 is a second captured image that can be acquired from the second imaging unit 0401_2, and 1102_1 is captured by the first captured image 1101_1. A subject with a known distance from the imaging unit 1102_2 is a subject corresponding to the subject 1102_1 that is captured in the first captured image 1101_2. The zoom change amount detection unit 0104 detects two or more features from the subject 1101_1, and performs matching processing between the first captured image 1101_1 and the second captured image 1101_2 for each feature. For feature detection, edge enhancement filters such as Prewitt and Sobel, and corner extraction filters such as Harris operator can be used. For the matching process, various correlation calculation methods such as normalized cross-correlation and sum of absolute differences can be used. In addition, when performing matching processing, considering that there is a difference in zoom magnification between the imaging units in advance, the matching accuracy is improved by changing the block size in block matching and the search area for searching for corresponding points The search processing amount can be reduced. At this time, the approximate zoom amount of each imaging unit may be acquired from the zoom control information of the stereo imaging control unit 0103 and used as a reference for the change amount of the block size and the search area. If matching processing is performed for each feature, a shift of the epipolar line is detected for each feature. When the two imaging units are installed horizontally so that their scanning line directions match, the epipolar line matches the scanning line if there is no difference in zoom magnification between the imaging units. On the other hand, when there is a difference in zoom magnification between the two imaging units, as shown in FIG. 11, a shift in the coordinate of the corresponding feature in the vertical axis direction occurs as a shift in the epipolar line. At this time, by using the epipolar shift c1 in the first feature in the subject 1101_1, the epipolar shift c2 in the first feature, and the distance Z in the optical axis direction in the space from the imaging unit to the subject, the focal length can be reduced. The difference can be calculated. Even if the first feature and the second feature have different distances from the imaging unit, if the distance information is known, the difference in focal length can be calculated. Of course, even if the first feature and the second feature do not exist in the same subject. In addition, the accuracy may be improved by estimating an optimal solution for the difference in focal length using a plurality of features. At this time, by using the correlation evaluation value at the time of the matching process and weighting the calculated difference in focal length, it is possible to reduce the influence of the incorrect correspondence at the time of the matching process. Even when the two imaging units are not installed horizontally so that the scanning line directions coincide with each other, by performing parallelization processing on the stereo captured image using information such as the geometric relationship between the imaging units. The two imaging units can be handled in the same manner as in the case where they are installed horizontally so that the scanning line directions coincide with each other. As the information used for the parallelization processing, information used at the time of calibration may be acquired and used by the stereo image processing unit 0102. In this way, by using the epipolar line shift of a feature whose known distance from the imaging unit in the stereo image is different from each imaging unit, the appearance of the captured image varies greatly due to the influence of occlusion and the difference in the viewing direction. Even when the method shown in FIG. 10 is not applicable, it is possible to acquire the difference in focal length between the imaging units with high accuracy by using the selectively detected features.

  FIG. 12 is a third diagram illustrating an example of a process for detecting focal length difference information according to the third embodiment of the present invention. The focal length difference information detection processing in FIG. 12 is processing performed in the zoom change amount detection unit 0104.

  In FIG. 12, 1201 is a first captured image that can be acquired from the first imaging unit 0401_1, and 1202 is a feature point group on a projected image of linear components existing in the space in the first captured image 1201. As shown in FIG. 12A, the linear component existing in the space exists as a straight line on the image in the state where there is no influence of lens distortion. Therefore, distance measurement is performed by stereo image processing using the first captured image 1201 and a second captured image (not shown) that can be acquired from the second imaging unit 0401_2, and the three-dimensional position of the feature point group 1202 is acquired. Then, when there is no zoom magnification difference between the captured images, the three-dimensional position of the feature point group 1202 exists on a straight line as shown in FIG. On the other hand, when there is a zoom magnification difference between the captured images, the three-dimensional position of the feature point group 1202 exists on the curve as shown in FIG. The zoom change amount detection unit 0104 extracts a feature point group of three or more points on a straight line from the first captured image 1201. For feature detection, edge enhancement filters such as Prewitt and Sobel, and corner extraction filters such as Harris operator can be used. Whether or not the extracted feature exists on a straight line may be evaluated using a Hough transform or the like. For the feature point group on the straight line obtained in this way, matching processing is performed in consideration of the zoom magnification difference between the imaging units, and the three-dimensional position is obtained by distance measurement. When performing the matching process, the block size in the block matching and the search area for searching for the corresponding points are changed according to the zoom magnification difference, thereby improving the matching accuracy and reducing the search processing amount. . Here, correction at the time of the matching process is performed using the predicted value of the zoom magnification difference. At the time of distance measurement, parameters necessary for distance measurement are updated using the predicted value of the zoom magnification difference. An actual zoom magnification difference is obtained from the predicted zoom magnification difference by determining whether the three-dimensional positions of the three or more feature point groups obtained in this way exist on a straight line, and the focal length Calculate the difference between However, since the zoom magnification difference between the imaging units is unknown, a plurality of values are used as predicted values, and an optimum result is used. At this time, the approximate zoom amount of each imaging unit may be acquired from the zoom control information of the stereo imaging control unit 0103 as a predicted value, and may be changed and used in the vicinity thereof. In addition, during the matching process, instead of sequentially correcting the zoom magnification difference, a correction process using electronic zoom is performed so that there is no zoom magnification difference between captured images. It can be done. When there are many linear components in the shooting environment, such as indoor or outdoor scenes where there are many buildings, use a method that uses the three-dimensional position information of the feature points on the straight line to accurately The difference in focal length can be acquired.

  Thus, according to the present embodiment, in stereo distance measurement, even when there is a zoom magnification difference between the imaging units, the zoom magnification difference between the imaging units is detected with high accuracy, and accurate stereo image processing is performed. And accurate distance measurement can be performed.

  In the first to third embodiments, it is possible to detect a zoom magnification difference between the imaging units, a zoom change amount, and the like, and to perform distance measurement by high-precision stereo image processing. However, when the lens group is driven for zooming or focusing, the optical axis of the lens may be slightly shifted depending on the zoom and focus mechanisms. It is also known that the distortion characteristics of the lens change according to the zoom amount. Optical axis correction and lens distortion correction for performing stereo image processing with high accuracy are realized by calibration processing in the stereo image processing unit 0102, but parameters necessary for calibration are zoom and focus parameters. Since the influence of the optical axis and lens distortion changes when the position of the lens group changes, it is necessary to use optimal values as appropriate. In such a case, it may be solved by the following method.

  FIG. 13 is a schematic diagram showing a stereo distance measuring system according to a fourth embodiment of the present invention. In FIG. 13, reference numeral 1305 denotes a correction table storage unit.

  The correction table storage unit 1305 corrects the relationship between the control amount of the lens group that performs zooming and focusing and correction data for correcting the change in the optical axis and lens distortion accompanying the zooming and focusing at the time of shipment from the factory. Stored as a data table. In this correction data table, each control amount and correction data of the lens group may be described as a multi-dimensional table. In order to reduce the data amount, the lens group is regarded as one lens, and the lens group The integrated control amount and correction data may be described as a two-dimensional table. In addition, if only some representative relationships are stored in the correction data table and the lens is controlled by values that are not stored in the data table, the neighboring values stored in the data table are stored. It is also possible to reduce the amount of stored data by using the interpolation.

  The correction table storage unit 1305 acquires the zoom change amount of the first image capturing unit 0401_1 and the second image capturing unit 0401_2 from the zoom change amount detecting unit 0104, and controls the lens group control amount during zooming based on the information. Is read from the table and output to the stereo image processing unit 0102. The stereo image processing unit 0102 can perform highly accurate distance measurement by performing calibration processing using the acquired correction data.

  Since the zoom change amount detection unit 0104 detects the change amount of the zoom using image processing, the detection accuracy of the change amount of the zoom decreases due to the change of the optical axis and the change of the lens distortion characteristic due to the change of the zoom. there is a possibility. Therefore, the correction table storage unit 1305 acquires the zoom change amount detected by the zoom change amount detection unit 0104, outputs the corresponding correction data table to the zoom change amount detection unit 0104, and the zoom change amount detection unit 0104 acquires the same. A process of detecting the zoom change amount may be performed again after performing a stereo image correction process as a pre-process using the corrected data table. By repeating this process recursively, the zoom change amount detection unit 0104 can finally detect a highly accurate zoom change amount that is not affected by changes in the optical axis or lens distortion characteristics. It becomes. When the recursive process of detecting the zoom change amount is performed, the zoom change amount detection unit 0104, which is most affected by the change in the axis and the distortion characteristic of the lens, is the initial value of the zoom change amount. Instead of using the detected zoom change amount, general zoom control information output from the stereo imaging control unit 0103 may be used.

  Further, the lens distortion amount and the change amount of the optical axis may be detected by a known method using a calibration pattern or the like, and the correction data table may be updated. As a result, even if the value of the correction data table actually required has changed from the time of shipment from the factory due to aging, vibration, etc., it can be updated again to the optimum parameter and used. This process may be performed by the zoom change amount detection unit 0104 in parallel with the zoom change amount detection process as necessary.

  Thus, according to the present embodiment, a high-accuracy stereo image can be obtained using the correction data table even when the optical axis changes or the lens distortion characteristics change due to zooming or focusing in addition to zooming. Distance measurement by processing can be performed.

  In the present invention, an example of a method of controlling the zoom change amount in the stereo imaging control unit 0102 will be described below.

  The stereo imaging control unit 0102 performs control so that the zoom of the first imaging unit 0401_1 and the second imaging unit 0401_2 changes at a constant speed between the imaging units. For example, there is a difference in the driving amount of the lens for changing the zoom magnifications of the first imaging unit 0401_1 and the second imaging unit 0401_2 by an equal amount due to individual variations in lens characteristics and the accuracy of the structure. When the imaging unit 0401_1 needs to be controlled so that the lens is driven by n times the second imaging unit 0401_2, the stereo imaging control unit 0103 performs the lens driving of the first imaging unit 0401_1. Control is performed so as to be performed at a speed 1 / n times the lens driving of the imaging unit 0401_2. As described above, the zoom magnification of each image capturing unit moves at a constant speed, so that the captured images captured by the first image capturing unit 0401_1 and the second image capturing unit 0401_2 in the middle of the zoom change are respectively captured by the image capturing unit. Regardless of the difference in lens characteristics and structure, the zoom magnification is always equal, so that distance measurement by stereo image processing can be performed with high accuracy even during zoom change.

  In addition, when the zoom of the first imaging unit 0401_1 and the second imaging unit 0401_2 changes significantly, the stereo imaging control unit 0102 does not change the zoom at a time, but changes a plurality of zooms until the zoom reaches a desired amount. Zoom control may be performed in stages. At this time, the zoom change detection unit 0104 and the stereo image processing unit 0102 use the image captured at the timing when the zoom control is completed for image processing, thereby reducing the influence of blurring that occurs in the captured image during zooming. Accurate image processing can be performed. Further, when each of the plurality of zoom controls is finished, the zoom change amount detection unit 0104 sequentially detects a difference in zoom change amount between the image pickup units, and the stereo image pickup control unit 0103 performs zoom control of the next stage. By performing zoom control for correcting the difference in zoom change amount before, it is possible to avoid the occurrence of a large difference in zoom change amount between the imaging units when the zoom is greatly changed at once.

  Thus, according to the present embodiment, by performing zoom control suitable for stereo image processing, high-accuracy stereo image processing is performed using a stereo image that has no or small difference in zoom magnification between captured images. Distance measurement can be performed.

  FIG. 14 is a schematic diagram showing a stereo distance measuring system according to the sixth embodiment of the present invention. In FIG. 14, reference numeral 1406 denotes a control instruction interface unit.

  The stereo distance measurement system shown in FIG. 14 is a system that considers the case where a lens that can be manually zoomed, such as a varifocal lens, is used as a lens group that performs zooming and focusing of an imaging unit.

  The user operation instruction interface unit 1406 acquires information on the zoom change amount or zoom magnification difference between the first image capturing unit 0401_1 and the second image capturing unit 0401_2, which is output from the zoom change amount detecting unit 0103. Zoom control information necessary for correcting the difference in zoom magnification between the imaging units is notified to the user using video and audio. As a means for notification, various video terminals and sound sources such as a TV or PC monitor, a portable terminal, a speaker, and a buzzer can be used. Based on an instruction from the control instruction interface unit 1406, the user directly operates the lens group of the first imaging unit 0401_1 or the second imaging unit 0401_2 to perform zoom control. As described above, the zoom control information necessary for correcting the zoom magnification difference is instructed to the user by using the user operation instruction interface unit 1406, so that the image pickup unit having the lens group that performs zoom control manually is used. However, it is possible to correct the difference in zoom magnification between the imaging units with high accuracy.

  FIG. 15 is a diagram showing an example of a method for instructing a user zoom control information for zoom correction according to a sixth embodiment of the present invention using an image.

  In FIG. 15, reference numeral 1501 denotes a user operation instruction interface, which is assumed to be a PC monitor. On the monitor of the PC, information necessary for performing zoom control is superimposed and displayed on a captured image of an image capturing unit that needs to perform zoom control. As an imaging unit that needs to perform zoom control, if there is a main imaging unit used for recording or monitoring, the other imaging unit may be selected to match the zoom of the main imaging unit. In addition, the two imaging units may be controlled so as to be weighted according to the current zoom magnification and adjusted to a specific zoom magnification between the zoom magnifications of the two imaging units. In this case, the captured images of the imaging units may be displayed side by side. In FIG. 15, a method using a specific subject in a captured image and a method using characters and symbols are used in combination as a representation of information necessary for zoom control superimposed on the captured image. As a method using a specific subject in the captured image, the position where the subject in the captured image is displayed after zoom correction is indicated by transmitting the subject enlarged at the magnification after zoom correction and displaying it with a dotted line. Yes. The user manually controls the zoom so that the subject in the actual captured image overlaps the subject after zoom correction superimposed on the monitor, so that the necessary amount of zoom control can be operated intuitively. It can be carried out. In addition, the zoom control method itself may be illustrated so that the user can easily understand the zoom control method. For example, when zoom control is performed with a screwdriver, the user can easily perform zoom control operations by showing the position and direction of the screw on the image. Further, as displayed on the upper right of the monitor, the zoom direction and zoom magnification necessary for zoom correction may be described by arrows or characters. In this example, the combination of the up arrow and “× 1.1” indicates that the zoom needs to be controlled 1.1 times in the enlargement direction. In addition, the necessary zoom control direction, control amount, and degree of coincidence may be displayed as a combination of characters, symbols, and numerical values. In this case, the amount of information and the amount of display can be reduced compared to the case shown in the figure, so that it can be displayed on a portable terminal or the like.

  Although FIG. 15 shows a method for instructing the user to perform zoom control using an image, audio may be used instead of the image. Zoom control information when displaying text on the video may be notified to the user as it is, and the zoom direction, zoom magnification, and degree of coincidence necessary for zoom correction are combined with the buzzer sound interval, pitch and volume May be expressed. When sound is used, it is possible to notify the user of information necessary for zoom control even when there is no space for installing a monitor. Further, there is an advantage that the user can work while looking at the imaging unit operated by the user.

  It should be noted that the instruction of the operation method to the user as described above can be used not only at the time of zoom correction but also when correcting the deviation of the optical axis described in the fourth embodiment.

  Thus, according to the present embodiment, even in a stereo distance measurement system in which the lens group is controlled manually by notifying the user of information necessary for correcting the difference in zoom magnification between the imaging units, It is possible to correct the zoom magnification difference between the imaging units with high accuracy.

  In the present invention, an example in which the zoom change amount detection processing performed by the zoom change amount detection unit 0104 is applied to correction of secular change will be described.

  The zoom change amount detection unit 0104 acquires the zoom change amount and the zoom magnification difference between the imaging units by the method shown in the first to third embodiments. This process is always performed even when the zoom imaging control unit is not performing zoom control, so that even if the zoom changes minutely due to vibration or deterioration of the device over time, the zoom is controlled to reduce the effect. Correction can be performed, and highly accurate stereo image processing can be performed.

  In addition, the process of detecting the zoom change amount using the change in the interval between the two feature points in the captured image shown in FIG. 6 is applied, and the parallel movement of the image pickup unit is performed from the change amount of the feature point position. , And rotational movement around the optical axis. When the imaging unit moves in parallel with the imaging surface or rotates around the optical axis without causing a zoom change of the imaging unit, the interval between two feature points in the captured image does not change. Only the position of the feature point on the image changes. Therefore, by detecting a change in the position of the feature point on the image, the movement of these imaging units can be detected. In this manner, the zoom change amount detection process may be applied to detect and correct the influence of the secular change other than the zoom change on the optical axis change and the like.

  Thus, according to the present embodiment, the robustness of the stereo distance measurement system with respect to secular change is improved by applying the zoom change amount detection processing shown in the first to third embodiments. Is possible.

  In addition, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. In addition, a part of the configuration of a certain embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of a certain embodiment.

  The present invention can be used for surveillance cameras, stereoscopic cameras, and in-vehicle cameras.

It is a schematic diagram which shows the stereo distance measuring system which concerns on 1st Example of this invention. It is a 1st figure which shows an example of the detection process of the focal distance variation | change_quantity based on 1st Example of this invention. It is a 2nd figure which shows an example of the detection process of the focal distance variation | change_quantity based on 1st Example of this invention. It is a schematic diagram which shows the stereo distance measuring system which concerns on 2nd Example of this invention. It is a figure which shows an example of the process sequence of the stereo distance measuring system which concerns on 2nd Example of this invention. It is a 1st figure which shows an example of the process which detects the focal distance variation | change_quantity of 2nd Example of this invention. It is a 2nd figure which shows an example of the process which detects the focal distance variation | change_quantity of 2nd Example of this invention. It is a 3rd figure which shows an example of the process which detects the focal distance variation | change_quantity of 2nd Example of this invention. It is a figure which shows an example of the process sequence of the stereo distance measuring system which concerns on 3rd Example of this invention. It is a 1st figure which shows an example of the process which detects the difference information of the focal distance which concerns on 3rd Example of this invention. It is a 2nd figure which shows an example of the process which detects the difference information of the focal distance which concerns on 3rd Example of this invention. It is a 3rd figure which shows an example of the process which detects the difference information of the focal distance which concerns on 3rd Example of this invention. It is a schematic diagram which shows the stereo distance measuring system which concerns on 4th Example of this invention. It is a schematic diagram which shows the stereo distance measuring system which concerns on 6th Example of this invention. It is a figure which shows an example of the method of instruct | indicating the zoom control information for the zoom correction | amendment based on 6th Example of this invention to a user using an image | video.

Explanation of symbols

0101 Stereo imaging unit 0102 Stereo image processing unit 0103 Stereo imaging control unit 0104 Zoom change amount detection unit 0201_1 First captured image 0201_2 before zoom change First captured image 0202_1 after zoom change Second captured image before zoom change 0202_2 Second captured image after zoom change 0203_1 Feature point of still object before zoom change 0203_2 Feature point 0301 of still object after zoom change Still object 0302 First moving object 0303 Second moving object 0401_1 First imaging Unit 0401_2 Second imaging unit 0601_1 Captured image before zoom change 0601_2 Captured image 0602-2_1 after zoom change In the captured image before zoom change, the feature point has a constant distance in the optical axis direction from the imaging unit and is stationary Feature point 0602-3_2 Zoom Feature points 0701 corresponding to 0602 to 3_1 after the change Captured image 0702 before the zoom change Captured image 0703 after the zoom change Captured image 0801_1 after the n-fold zoom change Captured image 0801_2 before the zoom change After the zoom change Captured image 0802_1 Feature point 0802_2 stationary in the captured image before the zoom change Feature point 1001 corresponding to 0802_1 in the captured image after the zoom change First captured image 1002 Second captured image 1003 Second captured image multiplied by n Image 1101_1 First captured image 1101_2 Second captured image 1102_1-2 Subject 1201 First captured image 1202 Feature point group 1305 Correction table storage unit 1406 User operation instruction interface unit 1501 Zoom control instruction interface

Claims (17)

Stereo imaging means with variable zoom for performing stereo imaging;
Stereo image processing means for performing stereo image processing using the stereo captured image output by the stereo imaging means and calculating distance information;
Stereo imaging control means for controlling the zoom of the stereo imaging means;
Zoom change amount detection means for detecting a zoom change amount using a stereo captured image output by the stereo image pickup means;
The zoom change amount detecting means detects a stationary feature during zoom control from a stereo image by image processing, calculates a zoom change amount based on a change in parallax information before and after zoom control of the feature, and The image processing means calculates a focal length of the stereo imaging means after zoom control based on a zoom change amount output from the zoom change amount detection means and uses it for calculation of distance information;
A distance measuring device characterized by The distance measuring device according to claim 1 ,
When the zoom change amount detection means calculates the zoom change amount, it calculates an estimated value of the zoom change amount for each of the stationary features during zoom control, and detects the zoom change amount using the histogram information. To do,
A distance measuring device characterized by A first imaging means with variable zoom for imaging;
A second imaging means with variable zoom for imaging;
Stereo image processing means for performing stereo image processing using the stereo image output from the first imaging means and the second imaging means and calculating distance information;
Stereo imaging control means for controlling zooming of the first imaging means and the second imaging means;
A subject that is stationary during zoom control is detected by image processing from captured images output from the first imaging unit and the second imaging unit, and a change in zoom is based on a change in position in the image before and after zoom control. Zoom change amount detecting means for calculating the amount;
With
The stereo imaging control unit acquires information on the zoom change amount from the zoom change amount detection unit, and when there is a difference in zoom magnification, the first image pickup unit and the second image pickup control unit correct the difference. The stereo image processing means controls the focal length of the first imaging means and the second imaging means after zoom control based on the zoom change amount output from the zoom change amount detection means. Is used to calculate distance information,
A distance measuring device characterized by The distance measuring device according to claim 3 ,
When the stereo image processing means obtains zoom change amount information from the zoom change amount detection means and performs stereo image processing, the zoom magnification of the first image pickup means and the zoom of the second image pickup means If there is a difference in magnification, the magnification of the stereo image is corrected by image processing so that the difference is corrected.
A distance measuring device characterized by In the distance measuring device according to claim 3 or 4 ,
Calculating the zoom change amount of the first image pickup means and the zoom change amount during zooming of the second image pickup means when the zoom change amount detection means detects the zoom change amount;
A distance measuring device characterized by The distance measuring device according to claim 5 ,
When the zoom change amount detecting means detects the zoom change amount, a plurality of features that are stationary at the time of zoom control are detected, and the zoom change is based on the change amount before and after the zoom control of the interval between the plurality of feature points. Calculating the quantity,
A distance measuring device characterized by The distance measuring device according to claim 5 ,
When the zoom change amount detecting means detects the zoom change amount, one of the captured image acquired before the zoom change and the captured image acquired after the zoom change is enlarged / reduced by a plurality of magnifications and matched with the other captured image. Calculating the amount of zoom change using the correlation value obtained by processing,
A distance measuring device characterized by The distance measuring device according to claim 5 ,
When the zoom change amount detecting means detects the zoom change amount, a feature in the captured image that is stationary during zoom control is detected, and the change amount before and after zoom control of the interval between the feature and the image center is detected. To calculate the amount of zoom change,
A distance measuring device characterized by In the distance measuring device according to claim 3 or 4 ,
When the zoom change amount detecting means detects the zoom change amount, the zoomed image magnification difference is calculated by comparing the captured image acquired from the first image capturing means with the captured image acquired from the second image capturing means. ,
A distance measuring device characterized by The distance measuring device according to claim 9 , wherein
When the zoom change amount detecting means detects a zoom magnification difference, one of the captured image acquired from the first imaging means and the captured image acquired from the second imaging means is enlarged or reduced at a plurality of magnifications, Calculating the zoom magnification difference using the correlation value obtained by performing matching processing with the other captured image,
A distance measuring device characterized by The distance measuring device according to claim 9 , wherein
When the zoom change amount detecting unit detects a zoom magnification difference, a matching process is performed between a captured image acquired from the first imaging unit and a captured image acquired from the second imaging unit for a plurality of features. Calculating the zoom magnification difference using the information on the displacement of the matching position,
A distance measuring device characterized by The distance measuring device according to claim 9 , wherein
When the zoom change amount detection means detects a zoom magnification difference, distance measurement is performed by stereo image processing using the captured image acquired from the first imaging means and the captured image acquired from the second imaging means. Calculating the zoom magnification difference using the result,
A distance measuring device characterized by The distance measuring device according to claim 3 ,
A correction table storage means for storing parameters for correcting changes such as the optical axis and lens distortion that occur when the zoom changes,
When the stereo imaging control unit controls zooming of the first imaging unit or the second imaging unit, the correction table storage unit outputs zoom control information output from the stereo imaging control unit or the zoom change. The zoom change amount information detected by the amount detection unit is acquired, a correction parameter corresponding to the zoom change amount is output to the stereo image processing unit, and the stereo image processing unit performs a calibration process using the correction table. ,
A distance measuring device characterized by The distance measuring device according to claim 13 ,
The correction table storage unit outputs a correction table corresponding to the zoom change amount information detected by the zoom change amount detection unit to the zoom change amount detection unit, and the zoom change amount detection unit uses the correction table to perform stereo imaging. Perform the image correction process in the pre-process and detect the zoom change amount again.
A distance measuring device characterized by In the distance measuring device according to claim 1 or 2 ,
When the stereo imaging control means controls the zoom of the stereo imaging means, the zoom speed or the control amount for each imaging means so that the zoom magnification between the images is equal even in the stereo captured image acquired during the zoom control. To control the zoom,
A distance measuring device characterized by A first imaging means for manually changing a zoom for imaging;
A second imaging means for manually changing a zoom for imaging;
Stereo image processing means for performing stereo image processing using the stereo image output from the first imaging means and the second imaging means and calculating distance information;
The amount of change in zoom by detecting a change in position in the image before and after zoom control of a subject that is stationary during zoom control by image processing from captured images output by the first imaging unit and the second imaging unit Zoom change amount detecting means for detecting
User operation instruction interface means for outputting information necessary for the user to operate the zoom of the first image pickup means and the second image pickup means by video or audio;
With
When the user operation instruction interface unit obtains zoom change amount information from the zoom change amount detection unit and there is a difference in zoom magnification, the first image pickup unit and the second image pickup unit are connected to the user. To output information necessary for correcting the difference between the first image pickup unit after zoom control based on the zoom change amount output from the zoom change amount detection unit by the stereo image processing unit. Calculating the focal length of the second imaging means and calculating the distance information;
A distance measuring device characterized by The distance measuring device according to any one of claims 1 to 16 ,
Using the zoom change amount detection processing performed by the zoom change amount detecting means for dealing with secular change of the distance measuring device;
A distance measuring device characterized by

Images