JP2013127416A - Range finder, range finding system, range finding program and parallax correction method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a range finder, a range finding system, a range finding program and a parallax correction method for highly accurately correcting a parallax calculated from a first image and a second image according to a temperature when imaging the first image and the second image to exactly measure a distance.SOLUTION: The range finder includes: a parallax arithmetic section 204 for calculating the parallax on the basis of the first image and the second image; a temperature difference calculation section 205 for calculating a temperature difference to be a difference between the temperature when imaging the first image and the second image and a reference temperature; a parallax correction section 206 for calculating a first correction value for correcting an error component by expansion and contraction of a base line length and a second correction value for correcting an error component by a change of a magnification of the first image and the second image on the basis of the temperature difference and correcting the parallax calculated by the parallax arithmetic section 204 on the basis of the first correction value and the second correction value to calculate a corrected parallax; and a distance calculation section 207 for calculating a distance up to a subject on the basis of the corrected parallax, the base line length without expansion and contraction and a focal distance.

Description

  The present invention relates to a distance measuring device, a distance measuring system, a distance measuring program, and a parallax correction method.

  In recent years, ranging systems using stereo cameras have been widely used in various fields. For example, automobiles, home appliances, digital cameras, and the like equipped with a ranging system using a stereo camera have been put into practical use.

  The stereo camera includes a first imaging system including a first lens and a first imaging element, and a second imaging system including a second lens and a second imaging element. The first imaging system and the second imaging system have the same focal length and the optical axes are parallel to each other. The distance between the optical axes of the first imaging system and the second imaging system is called the baseline length.

  In the distance measuring system using a stereo camera, the distance to the subject is measured based on the first image of the subject imaged by the first imaging system and the second image of the subject imaged by the second imaging system. Specifically, a parallax that is a difference between the position of the subject image on the first image sensor and the position of the subject image on the second image sensor is obtained from the first image and the second image. Then, based on the obtained parallax, the above-described baseline length, and the focal lengths of the first imaging system and the second imaging system, the distance to the subject is measured by the principle of triangulation.

  As a camera module for a stereo camera, a stereo camera module including a lens array in which a first lens and a second lens are integrally formed on the same surface has been proposed. A stereo camera module provided with this lens array has the advantage that it can be easily miniaturized and can be mounted on various devices, while it is susceptible to temperature changes. In particular, when the lens array is made of resin, the lens array expands and contracts due to a temperature change, and an error occurs in the measured distance.

  In order to solve such a problem, a technique has been proposed in which a temperature sensor is mounted on a stereo camera module to detect an environmental temperature, and correction is performed according to the detected environmental temperature. For example, in Patent Document 1, a temperature sensor is mounted on a substrate on which an image sensor is mounted, and the temperature of the substrate is detected. Based on the detected temperature, the parallax is corrected by the change in the baseline length associated with the temperature change. Technology is disclosed.

  Japanese Patent Application Laid-Open No. 2004-228561 discloses a technique for detecting a temperature of a lens array by attaching a temperature sensor to the lens array, correcting a base line length based on the detected temperature, and calculating a distance using the corrected base line length. Is disclosed.

  However, the above-described conventional technique is a technique for correcting only the change in the baseline length according to the temperature change, and thus has a problem that the correction is insufficient. That is, when the members constituting the stereo camera module expand and contract due to temperature changes, not only the base line length but also the distance between the lens and the image sensor (back focus) and the refractive index according to the material of the lens array change, The image changes due to factors other than the change in the baseline length. However, since the conventional technique corrects only the change in the baseline length, it cannot perform highly accurate correction.

  The present invention has been made in view of the above, and corrects the parallax calculated from the first image and the second image with high accuracy according to the temperature at the time of capturing the first image and the second image. An object of the present invention is to provide a distance measuring device, a distance measuring system, a distance measuring program, and a parallax correction method capable of accurately measuring a distance.

  In order to solve the above-described problems and achieve the object, a distance measuring apparatus according to the present invention includes a first imaging system including a first lens, and a second lens including a second lens integrally formed with the first lens. Input means for inputting a first image of the subject imaged by the first imaging system and a second image of the subject imaged by the second imaging system from a stereo camera module comprising an imaging system; Disparity calculating means for calculating a disparity that is a difference between the position of the subject in the first image and the position of the subject in the second image based on the first image and the second image; and the first image And a temperature difference calculating means for calculating a temperature difference that is a difference between a temperature at the time of capturing the second image and a reference temperature, and light from the first imaging system and the second imaging system based on the temperature difference. For expansion and contraction of the baseline length, which is the distance between axes A first correction value for correcting the error component and a second correction value for correcting an error component due to a change in magnification of the first image and the second image, and calculating the parallax A parallax correction unit that corrects based on the first correction value and the second correction value to calculate a corrected parallax; the corrected parallax; the baseline length without expansion and contraction; the first imaging system and the second imaging Distance calculating means for calculating the distance to the subject based on the focal length of the system.

  The distance measuring system according to the present invention includes a first lens, includes a first imaging system that captures a first image of the subject, and a second lens integrally formed with the first lens. A parallax that is a difference between the position of the subject in the first image and the position of the subject in the second image based on the second imaging system that captures two images, and the first image and the second image Based on the temperature difference, a parallax calculating unit that calculates a temperature difference, a temperature difference calculating unit that calculates a temperature difference that is a difference between a temperature at the time of capturing the first image and the second image, and a reference temperature. A first correction value for correcting an error component due to expansion / contraction of a base line length, which is a distance between optical axes of one imaging system and the second imaging system, and a change in magnification of the first image and the second image. And a second correction value for correcting the error component A parallax correcting unit that corrects the parallax based on the first correction value and the second correction value to calculate a corrected parallax, the corrected parallax, the baseline length without expansion and contraction, and the first imaging And a distance calculating means for calculating a distance to the subject based on a focal length of the system and the second imaging system.

  A ranging program according to the present invention includes a stereo camera module including a first imaging system including a first lens and a second imaging system including a second lens integrally formed with the first lens. Based on the function of inputting the first image of the subject imaged by the first imaging system and the second image of the subject imaged by the second imaging system, and the first image and the second image, A function of calculating a parallax that is a difference between the position of the subject in the first image and the position of the subject in the second image, and a temperature and a reference temperature at the time of capturing the first image and the second image A function for calculating a temperature difference that is a difference, and an error component due to expansion and contraction of a base line length that is a distance between optical axes of the first imaging system and the second imaging system, based on the temperature difference. A first correction value and the first image; And a second correction value for correcting an error component due to a change in magnification of the second image, and correcting the parallax based on the first correction value and the second correction value. A function of calculating parallax, a function of calculating a distance to the subject based on the corrected parallax, the baseline length without expansion and contraction, and focal lengths of the first imaging system and the second imaging system; Is realized on a computer.

  The parallax correction method according to the present invention includes a stereo camera module including a first imaging system including a first lens, and a second imaging system including a second lens integrally formed with the first lens. A first image of a subject imaged by the first imaging system and a second image of the subject imaged by the second imaging system are input, and the position of the subject in the first image and the second image in the second image are input. A parallax correction method executed in an apparatus that calculates parallax that is a difference from a position of a subject, and calculates a temperature difference that is a difference between a temperature at the time of capturing the first image and the second image and a reference temperature And a first correction value for correcting an error component due to expansion / contraction of a base line length, which is a distance between the optical axes of the first imaging system and the second imaging system, is calculated based on the temperature difference. Step and said Calculating a second correction value for correcting an error component due to a change in magnification of the first image and the second image based on the degree difference; and the first correction value and the second correction value. And the step of correcting the parallax.

  According to the present invention, accurate distance measurement is performed by correcting the parallax calculated from the first image and the second image with high accuracy according to the temperature at the time of capturing the first image and the second image. There is an effect that can be.

FIG. 1 is a diagram illustrating a configuration example of a stereo camera module. FIG. 2 is a diagram in which the stereo camera module having the configuration shown in FIG. 1 is modeled. FIG. 3 is a diagram modeling the stereo camera module when the temperature rises by Δt from a certain reference temperature. FIG. 4 is a block diagram showing a functional configuration of the distance measuring apparatus. FIG. 5 is a diagram illustrating a configuration of the arithmetic circuit of the parallax correction unit. FIG. 6 is a diagram illustrating another configuration of the arithmetic circuit of the parallax correction unit. FIG. 7 is a diagram illustrating a relationship between the parallax calculated from the first image and the second image and the environmental temperature. FIG. 8 is a diagram showing the relationship between the temperature directly measured by the lens array and the temperature of the substrate measured by the temperature sensor.

  Hereinafter, embodiments of a distance measuring device, a distance measuring system, a distance measuring program, and a parallax correction method according to the present invention will be described in detail with reference to the accompanying drawings.

  The distance measuring system includes a stereo camera module and a distance measuring device. The distance measuring device is based on the images of the subject (first image and second image) captured by the two imaging systems (first imaging system and second imaging system) of the stereo camera module, from the stereo camera module to the subject. Measure the distance. The distance measuring device may be configured integrally with the stereo camera module (built in the stereo camera module), or may be configured as an external device separated from the stereo camera module, and the first image and the second image with respect to the stereo camera module. You may connect so that an image can be input. Hereinafter, an example in which the distance measuring device is configured as an external device separated from the stereo camera module will be described.

(Stereo camera module)
First, the configuration of the stereo camera module will be described with reference to FIG. FIG. 1 is a diagram illustrating a configuration example of a stereo camera module 100 according to the present embodiment. FIG. 1A is a front view of the stereo camera module 100 viewed from the front side, and FIG. XX sectional drawing of 1 (a) and FIG.1 (c) are the back views which looked at the stereo camera module 100 from the back surface side.

  The stereo camera module 100 includes a lens array 110 and a substrate 120 held by a lens frame 101.

  The lens frame 101 is, for example, a part produced by molding resin or stainless steel (SUS) into a rectangular cylindrical shape, and constitutes a side wall portion of the stereo camera module 100. A protrusion 102 is erected on the inner surface of the lens frame 101. The lens array 110 is supported on the protrusions 102. The substrate 120 is bonded to the end of the lens frame 101 and constitutes the back surface of the stereo camera module 100.

  The lens array 110 is an optical component in which the first lens 111 and the second lens 112 are integrally molded. The first lens 111 has a function of forming an image of a subject on an imaging surface of a first imaging element described later. The second lens 112 has a function of forming an image of a subject on an imaging surface of a second imaging element described later. The lens array 110 is produced by molding a transparent resin material, for example. As the transparent resin material constituting the lens array 110, for example, ZEONEX E48R (product name, manufactured by Nippon Zeon Co., Ltd., ZEONEX is a registered trademark of Nippon Zeon Co., Ltd.) is suitable.

  A first image sensor 121 and a second image sensor 122 configured as, for example, CMOS image sensors are mounted on one main surface portion of the substrate 120 facing the lens array 110. The position of the first imaging element 121 on the substrate 120 is determined so that the center of the imaging surface coincides with the optical axis center of the first lens 111. Further, the position of the second imaging element 122 on the substrate 120 is determined so that the center of the imaging surface thereof coincides with the optical axis center of the second lens 112.

  For example, the first image sensor 121 and the second image sensor 122 are formed on the same silicon wafer and cut out integrally. With this configuration, the first image sensor 121 and the second image sensor 122 can be easily aligned with the first lens 111 and the second lens configured as the lens array 110. Note that the first image sensor 121 and the second image sensor 122 can be formed independently and mounted on the substrate 120.

  The first lens 111 and the first imaging element 121 constitute a first imaging system that captures a first image of the subject. Further, the second lens 112 and the second imaging element 122 constitute a second imaging system that captures a second image of the subject. The first imaging system and the second imaging system have the same focal length and the optical axes are parallel to each other. The distance between the optical axes of the first imaging system and the second imaging system is called the baseline length.

  An aperture array 130 is disposed in front of the lens array 110 (subject side). The aperture array 130 is configured as a rectangular plate that fits on the inner periphery of the lens frame 101. The aperture array 130 includes a first aperture 131 that opens on the optical axis of the first imaging system, and a second aperture 132 that opens on the optical axis of the second imaging system. The first aperture 131 has a diaphragm function for limiting the amount of light incident on the first imaging system, and the second aperture 132 has a function of a diaphragm for limiting the amount of light incident on the second imaging system.

  Between the first imaging system and the second imaging system, the light incident through the first aperture 131 is prevented from interfering with the second imaging system, and the light incident through the second aperture 132 is A light shielding wall 103 for preventing interference with the first imaging system is provided. Depending on the distance between the first image pickup device 131 and the second image pickup device 132 and the angle of view of the first image pickup system and the second image pickup system, light interference may occur between the first image pickup system and the second image pickup system. It can also be set as the structure which does not arise. In this case, the light shielding wall 103 is not necessary.

  A thermistor 141 whose resistance value changes as the temperature of the substrate 120 changes is attached to the back side of the substrate 120 (the surface opposite to the mounting surface of the first image sensor 121 and the second image sensor 122). The thermistor 141 is connected to a temperature sensor circuit 142 that outputs a change in resistance value of the thermistor 141 as a digital value representing a temperature change of the substrate 120. In the stereo camera module 100, the temperature of the substrate 120 can be measured by the thermistor 141 and the temperature sensor circuit 142. Hereinafter, a combination of the thermistor 141 and the temperature sensor circuit 142 is referred to as a temperature sensor 140.

  A storage unit 150 is provided on the back side of the substrate 120. The storage unit 150 stores various parameters of the stereo camera module 100. Specifically, for example, a baseline length that is the distance between the optical axes between the first imaging system and the second imaging system (baseline length without expansion and contraction when the environmental temperature is the reference temperature), the first imaging system, and The focal length of the second imaging system (focal length when the ambient temperature is the reference temperature), the offset value, the camera calibration parameter for distortion correction, the output value (reference temperature) of the temperature sensor 140 at the time of calibration, etc. Stored in the storage unit 150.

  Further, on the back side of the substrate 120, an interface circuit 170 that manages signal transmission with a distance measuring device connected to the stereo camera module 100 via an FPC (flexible wiring substrate) 160 is provided. Signals transmitted from the interface circuit 170 to the distance measuring device via the FPC 160 are an image signal of the first image captured by the first imaging system and an image of the second image captured by the second imaging system. A signal, a camera control signal, an output signal of the temperature sensor 140, and various parameters described above stored in the storage unit 150.

(Principles of stereo ranging and the effects of temperature changes)
Next, the principle of stereo distance measurement using the stereo camera module 100 and the influence of temperature change will be described with reference to FIGS.

  FIG. 2 is a diagram in which the stereo camera module 100 having the configuration shown in FIG. 1 is modeled, and shows the positional relationship between the first imaging system and the second imaging system at a certain reference temperature. At the reference temperature, as shown in FIG. 2, the optical axis center (the optical axis of the first imaging system) that passes through the principal point of the first lens 111 and is perpendicular to the imaging surface of the first imaging element 121 is the first. This coincides with the center CL of the imaging surface of the image sensor 121. The optical axis center (optical axis of the second imaging system) that passes through the principal point of the second lens 112 and is perpendicular to the imaging surface of the second imaging element 122 coincides with the center CR of the imaging surface of the second imaging element 122. To do. The optical axis of the first imaging system and the optical axis of the second imaging system are parallel to each other, and the line segment connecting the principal point of the first lens 111 and the principal point of the second lens 112 is the light of the first imaging system. It is perpendicular to the axis and the optical axis of the second imaging system, and is parallel to the imaging plane of the first imaging element 121 and the imaging plane of the second imaging element 122.

  The length of the line segment connecting the principal point of the first lens 111 and the principal point of the second lens 112 is a baseline length B. A perpendicular line drawn perpendicularly to the line segment connecting the principal point of the first lens 111 and the principal point of the second lens 112 from the position of the subject is the principal point of the first lens 111 and the principal point of the second lens 112. Intersects the midpoint of the line segment connecting A distance between the intersection and the position of the subject is a distance Z to the subject to be measured. The distance between the principal point of the first lens 111 and the center CL of the imaging surface of the first image sensor 121 is the focal length f of the first imaging system, and the principal point of the second lens 112 and the second image sensor 122. The distance from the center CR of the imaging surface is the focal length f of the second imaging system, and the focal length f of the first imaging system and the focal length f of the second imaging system are equal.

Here, the distance from the center CL of the imaging surface of the first image sensor 121 to the position where the subject image is formed is dL, and from the center CR of the imaging surface of the second image sensor 122 to the position where the subject image is formed. Is a disparity ds that is a difference between the position of the subject in the first image captured by the first imaging system and the position of the subject in the second image captured by the second imaging system. It can be expressed as 1).
ds = dL + dR = B · f / Z (1)

  Under the environment of the reference temperature, the base line length B and the focal lengths f of the first imaging system and the second imaging system do not vary and are fixed values. Therefore, if the parallax ds is obtained from the first image captured by the first imaging system and the second image captured by the second imaging system, the distance Z to the subject can be calculated from the above equation (1). it can.

However, when the temperature of the environment in which the stereo camera module 100 is installed (environmental temperature) changes, the parts expand and contract according to the linear expansion coefficient of the material of each part constituting the stereo camera module 100. Then, the base line length B is expanded and contracted by expansion and contraction of the components of the stereo camera module 100. For example, the linear expansion coefficient of the transparent resin material ZEONEX E48R widely used as the material of the lens array 110 is 6 × 10 ⁻⁵ . On the other hand, since the first image sensor 121 and the second image sensor 122 on the substrate 120 are made of silicon, the linear expansion coefficient is 2.4 × 10 ⁻⁶ . Therefore, it can be seen that the expansion and contraction of the base line length B due to the temperature change is almost governed by the expansion and contraction of the lens array 110.

Further, when the environmental temperature changes, the refractive indexes of the first lens 111 and the second lens 112 change due to the expansion and contraction of the lens array 110. Since the refractive index increases as the density of the substance increases, the refractive index decreases as the temperature increases, and the refractive index increases as the temperature decreases. The variation rate of the refractive index of the first lens 111 and the second lens 112 with respect to the temperature may be regarded as a constant value in a general room temperature range excluding an extremely high temperature range or a low temperature range. For example, the temperature coefficient of the refractive index of the transparent resin material ZEONEX E48R is approximately −1 × 10 ⁻⁴ .

Further, when the environmental temperature changes, the lens frame 101 that holds the lens array 110 and the substrate 120 also expands and contracts. The lens frame 101 is often made of stainless steel (SUS). The linear expansion coefficient of SUS is 1.0 × 10 ^{−6 to} 1.5 × 10 ⁻⁶ . When the environmental temperature changes, the distance between the lens array 110 and the substrate 120, that is, the first lens 111 and the second lens 112, the first imaging element 121, and the second imaging are caused by expansion and contraction according to the linear expansion coefficient of the lens frame 101. The distance (back focus) between the elements 122 also varies.

  The change in the refractive index of the first lens 111 and the second lens 112 means that the angle of light emitted from the first lens 111 and incident on the first image sensor 121 or the second angle emitted from the second lens 112 is second. That is, the angle of light incident on the image sensor 122 changes. When the environmental temperature rises and the lens array 110 expands, the density of the first lens 111 and the second lens 112 decreases and the refractive index decreases. That is, since the light does not bend much by the first lens 111 and the second lens 112, the image projected on the first image sensor 121 and the second image sensor 122 becomes large. On the other hand, when the environmental temperature decreases, the image projected on the first image sensor 121 and the second image sensor 122 becomes smaller.

The change caused by the change in the environmental temperature of the image projected on the first image sensor 121 and the second image sensor 122 corresponds to the change in the focal length f in the pinhole camera model. Changes in the refractive indices of the first lens 111 and the second lens 112 and the distances (back focus) between the first lens 111 and the second lens 112 and the first image sensor 121 and the second image sensor 122 are in the range of room temperature. Since it is small, it may be considered that the focal length f extends in proportion to the environmental temperature. Therefore, the expansion rate τf of the focal length f can be defined as the following formula (2). Here, Δf is the amount of change in focal length f, and Δt is the amount of change in environmental temperature.
Δf = τf · f · Δt (2)

  Here, the parallax ds' observed when the environmental temperature changes is obtained. FIG. 3 is a diagram modeling the stereo camera module 100 when the temperature rises by Δt from a certain reference temperature, and shows the positional relationship between the first imaging system and the second imaging system when temperature expansion occurs in the component parts. Represents.

The distance to the subject to be measured is Z, the focal length of the stereo camera module 100 is f + Δf (f is the focal length before temperature change, Δf is the amount of change in focal length due to temperature change), and the baseline length is B + ΔB (B is Base line length before temperature change, ΔB is change amount of base line length due to temperature change), distance from the center CL of the imaging surface of the first image sensor 121 to the position where the subject image is formed, d1, second image sensor 122 The distance from the center CR of the imaging surface to the position where the subject image is accumulated is d2, and from the position on the optical axis of the first lens 111 on the imaging surface of the first imaging element 121 to the position where the subject image is formed. Is dL ′, and the distance from the position on the optical axis of the second lens 122 on the imaging surface of the second image sensor 122 to the position where the image of the subject is formed is dR ′, the parallax ds ′ is expressed by the following formula ( 3).
ds ′ = dL ′ + dR ′ + ΔB / 2 · 2 = (B + ΔB) (f + Δf) / Z + ΔB (3)

Here, if Δf · ΔB≈0, the following equation (4) is obtained.
ds ′ = ds + ΔB + (B · Δf + f · ΔB) / Z = ds (1 + Δf / f + ΔB / B) + ΔB (4)

Therefore, the equation for obtaining the parallax ds before temperature change from the observed parallax ds ′ (= dL + dR) is as shown in the following equation (5).
ds = ds′−ΔB / (1 + Δf / f + ΔB / B) (5)

Here, Δf = τf · f · Δt (the above formula (2)) and ΔB = B · τB · Δt (τB is a linear expansion coefficient of the lens array 110), and therefore, when τf + τB = τ, the above formula (5) can be modified as shown in the following formula (6).
ds = ds′−B · τB · Δt / (1 + τ · Δt) (6)

  As shown in the above formula (6), the observed parallax change amount (ds'-ds) caused by the temperature change is not only the expansion / contraction of the baseline length B but also the expansion / contraction of the baseline length B and the first lens. A change in the magnification of the image due to a change in the focal length f due to a change in the refractive index of the lens 111 or the second lens 112 is also influential. Since the amount of change in parallax due to the latter change in magnification changes according to the magnitude of temperature change, it is particularly affected when the temperature change becomes large.

  Therefore, in the distance measuring system according to the present embodiment, the observed parallax ds ′ is corrected according to the change in the environmental temperature in consideration of not only the expansion / contraction of the baseline length B but also the change in the magnification of the image. The distance to the subject can be measured accurately. Specifically, the temperature at which the stereo camera module 100 captures the first image and the second image (in this embodiment, the temperature of the substrate 120) is measured by the temperature sensor 140, and is measured by the temperature sensor 140 during calibration. A difference (temperature difference Δt) from the calculated reference temperature is obtained. Then, based on the temperature difference Δt, a first correction value for correcting an error component due to the expansion and contraction of the base line length B caused by the change in the environmental temperature, and the first image and the second image caused by the change in the environmental temperature. And a second correction value for correcting an error component due to a change in magnification. Then, the parallax ds ′ calculated based on the first image and the second image is corrected based on the first correction value and the second correction value to obtain a corrected parallax ds. Then, a distance Z to the subject is calculated from the corrected parallax ds, the base line length B without expansion and contraction, and the focal length f.

  In the present embodiment, the process of calculating the parallax ds ′ based on the first image and the second image, or correcting the parallax ds ′ based on the first correction value and the second correction value to calculate the corrected parallax ds. And a process for calculating the distance Z to the subject using the corrected parallax ds is performed by a distance measuring device which is an external device separated from the stereo camera module 100. Hereinafter, a specific configuration example of the distance measuring device will be described.

(Ranging device)
FIG. 4 is a block diagram showing a functional configuration of the distance measuring apparatus 200 according to the present embodiment. As shown in FIG. 4, the distance measuring device 200 includes a first interface unit 201, a camera control unit 202, an image signal distortion correction unit 203, a parallax calculation unit 204, a temperature difference calculation unit 205, and a parallax correction unit. 206, a distance calculation unit 207, and a second interface unit 208. For example, when the distance measuring device 200 is configured as a microcomputer including a CPU, a ROM, a RAM, an input / output circuit, and the like, these units (all or a part thereof) are programs (software) executed by the microcomputer. ). Further, each of the above-described units (all or a part thereof) included in the distance measuring device 200 is configured using dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). You can also.

  The first interface unit 201 is an interface for performing signal transmission with the stereo camera module 100 via the FPC 160. The first interface unit 201 has a function of inputting the first image and the second image from the stereo camera module 100 under the control of the camera control unit 202.

  The camera control unit 202 controls the operation of the stereo camera module 100. The camera control unit 202 controls reading of image signals from the first image sensor 121 and the second image sensor 122 and, for example, automatic exposure, white balance, image signal mode (grayscale / YUV color / raw image), and the like. Then, setting of various functions relating to image capturing by the stereo camera module 100 is executed. The setting of various functions related to image capturing by the stereo camera module 100 is determined by the value of a register in the camera control unit 202, for example.

  The camera control unit 202 reads various parameters of the stereo camera module 100 from the storage unit 150 mounted on the substrate 120 of the stereo camera module 100 when the distance measuring device 200 is turned on. Specifically, when the distance measuring apparatus 200 is turned on, the camera control unit 202 has a baseline length B without expansion and contraction when the environmental temperature is the reference temperature, a focal length f when the environmental temperature is the reference temperature, an offset value, The camera calibration parameters for distortion correction, the output value (reference temperature) of the temperature sensor 140 at the time of calibration, and the like are read from the storage unit 150 of the stereo camera module 100. These various parameters read from the storage unit 150 of the stereo camera module 100 are stored in a memory (not shown) inside the distance measuring apparatus 200.

  The image signal distortion correction unit 203 corrects the distortion of the image signals of the first image and the second image input from the first interface unit 201. The image signal distortion correction unit 203 reads the camera calibration parameters from the memory because the camera calibration parameters read from the storage unit 150 of the stereo camera module 100 when the power is turned on are stored in a memory (not shown). To correct the image signal distortion. As the image signal distortion correction method, for example, various well-known methods such as a look-up table method or a polynomial approximation method can be used. When image signal distortion correction is performed using the lookup table method, the camera calibration parameters stored in the storage unit 150 are values in the lookup table. When executing distortion of an image signal by a polynomial approximation method, polynomial coefficients are stored in the storage unit 150.

  The parallax calculation unit 204 calculates the parallax ds ′ based on the image signals of the first image and the second image after the distortion correction is executed by the image signal distortion correction unit 203, and uses the calculated parallax ds ′ as the parallax correction unit. It outputs to 206. The calculation of the parallax ds ′ by the parallax calculation unit 204 is, for example, a process of detecting the difference between the position of the subject in the first image and the position of the subject in the second image in units of pixels. For example, ZSSD (Zero-Sum of Squared Differences) can be used as a block matching algorithm for detecting a matching position in units of pixels in the parallax ds ′ calculation method by the parallax calculation unit 204. As a method for estimating a matching position in units of subpixels (subpixels), a well-known quadratic curve fitting can be used. ZSSD calculates the sum of squared differences using a value obtained by subtracting the average value of pixel values of blocks used for block matching between the first image and the second image from each pixel. The position of the matching block where the minimum value of the sum of squared differences is obtained is the matching position in pixel units, and quadratic curve fitting is performed using the value of the sum of squared differences obtained when the matching block is shifted left and right by one pixel. Go to estimate the matching position of the sub-pixel. This ZSSD method is particularly advantageous when the brightness differs between the first image and the second image.

  The temperature difference calculation unit 205 uses the first interface unit 201 to calculate the temperature of the substrate 120 (temperature during imaging) measured by the temperature sensor 140 when the first image and the second image are captured by the stereo camera module 100. To get through. The temperature difference calculation unit 205 is read from the storage unit 150 of the stereo camera module 100 by the camera control unit 202 when the distance measuring device 200 is turned on, and stored in a memory (not shown) inside the distance measuring device 200. The output value (reference temperature) of the temperature sensor 140 at the time of calibration is read from the memory. Then, the temperature difference calculation unit 205 calculates a temperature difference Δt that is a difference between the imaging temperature and the reference temperature, and outputs the calculated temperature difference Δt to the parallax correction unit 206.

  The parallax correction unit 206, based on the temperature difference Δt calculated by the temperature difference calculation unit 205, the first correction value for correcting the error component due to the expansion and contraction of the baseline length B, and the magnification of the first image and the second image A second correction value for correcting the error component due to the change is calculated, and the parallax ds ′ calculated by the parallax calculation unit 204 is corrected based on the first correction value and the second correction value, thereby correcting parallax ds. Is calculated. That is, the corrected parallax calculating unit 206 calculates the corrected parallax ds by executing the calculation represented by the above formula (6) based on the temperature difference Δt calculated by the temperature difference calculating unit 205. The parameters τ and τB used in the calculation of the above formula (6) are stored in advance in a memory (not shown) inside the distance measuring apparatus 200. The parameter B is a baseline length without expansion and contraction when the environmental temperature is the reference temperature, and is read from the storage unit 150 of the stereo camera module 100 by the camera control unit 202 when the distance measuring device 200 is turned on. It is stored in a memory (not shown) inside the apparatus 200.

  FIG. 5 shows a configuration of an arithmetic circuit when the parallax correction unit 206 executes the calculation of the above equation (6). In this case, the parallax correction unit 206 calculates the first correction value by multiplying the value obtained by multiplying the baseline length B without expansion / contraction by τB by the temperature difference Δt. In addition, the parallax correction unit 206 calculates a second correction value by adding 1 to a value obtained by multiplying τ by the temperature difference Δt. Then, the parallax correction unit 206 calculates a value obtained by dividing the second correction value by the value obtained by subtracting the first correction value from the parallax ds ′ calculated by the parallax calculation unit 204 as the corrected parallax ds.

In the case where τ · Δt is a small value, the parallax correction unit 206 can also calculate the corrected parallax ds ′ using the following equation (7) that is an approximation of the above equation (6). Here, τ · τB is sufficiently close to zero.
ds = ds ′ (1−τ · Δt) −B · τB · Δt (7)

  FIG. 6 shows a configuration of an arithmetic circuit when the parallax correction unit 206 executes the calculation of the above equation (7). In this case, the parallax correction unit 206 calculates the first correction value by multiplying the value obtained by multiplying τB by the temperature difference Δt by the baseline length B without expansion / contraction. Further, the parallax correction unit 206 calculates a second correction value by subtracting 1 from a value obtained by multiplying τ by the temperature difference Δt. Then, the parallax correction unit 206 calculates, as the corrected parallax ds, a value obtained by subtracting the first correction value from the value obtained by multiplying the parallax ds ′ calculated by the parallax calculation unit 204 by the second correction value.

The distance calculation unit 207 calculates the distance Z to the subject based on the corrected parallax ds calculated by the parallax correction unit 206, the baseline length B without expansion and contraction, and the focal length f. The distance Z to the subject can be calculated by the following equation (8), for example.
Z = B · f / (ds−d _offset ) (8)

Here, d _offset is a parallax obtained when photographing at infinity. B and f are the value of the base line length and the focal length at the time of calibration (base line length and focal length without expansion and contraction). These values are stored in advance in the storage unit 150 of the stereo camera module 100 in the same manner as the calibration parameters of the stereo camera module 100. After the distance measuring device 200 is turned on, the camera control unit 202 removes these values from the storage unit 150. It is read out and stored in a memory (not shown) inside the distance measuring device 200. The value stored in this memory is used for the calculation of the above equation (8).

  The second interface unit 208 is an interface for outputting information on the distance to the subject calculated by the distance calculation unit 207 from the distance measuring device 200 to an external information processing device via the FPC 210.

(Method for determining temperature coefficients τ, τB)
Next, a specific example of a method for determining τ and τB that are temperature coefficients used for calculating the corrected parallax ds by the calculation of the above formula (6) or the above formula (7) will be described.

  As described above, the parallax error component resulting from the change in environmental temperature is dominated by the expansion and contraction of the lens array 110. However, the temperature sensor 140 provided in the stereo camera module 100 in order to detect a change in the environmental temperature does not directly measure the temperature of the lens array 110. Further, if the material of each component constituting the stereo camera module 100 is different, each material has a specific linear expansion coefficient. Further, depending on the manufacturing method of the stereo camera module 100, each component constituting the stereo camera module 100 is also different. The way of expansion and contraction is different. Therefore, the temperature coefficients τ and τB are experimentally obtained in advance. Hereinafter, a specific method for experimentally obtaining τ and τB will be described.

  At a certain reference temperature, a subject located at a predetermined distance from the stereo camera module 100 is imaged, and the parallax is calculated from the obtained first and second images. Next, the ambient temperature is raised and lowered to change the temperature of the substrate 120, and the subject is similarly imaged and the parallax is calculated at a plurality of ambient temperatures, and the temperature measured by the temperature sensor 140 is recorded. Then, data as shown in FIG. 7 is obtained. FIG. 7 is a diagram illustrating the relationship between the calculated parallax and the environmental temperature. As shown in FIG. 7, when the temperature difference is small, the parallax changes linearly with respect to the temperature change. However, when the temperature difference becomes large, the influence of the denominator term of the above equation (6) becomes large. Therefore, the relationship between the temperature difference and the parallax deviates from the straight line. The temperature coefficients τ and τB can be determined by performing fitting using the above equation (6) on the data shown in FIG.

  Here, τB is ideally a linear expansion coefficient of the material of the lens array 110, but since the temperature sensor 140 is mounted on the substrate 120, the temperature measured by the temperature sensor 140 is the same as the temperature of the lens array 110. Is different. When actually measured, the temperature of the lens array 110 (hereinafter referred to as the lens temperature) directly measured using a thermocouple and the temperature of the substrate 120 (hereinafter referred to as the substrate temperature) measured by the temperature sensor 140 are shown in FIG. As shown in FIG. 8, it was found that there was a linear relationship. FIG. 8 is a diagram showing the relationship between the lens temperature and the substrate temperature.

  As shown in FIG. 8, the relationship between the lens temperature and the substrate temperature is linear, but the inclination is not 1. In the example of FIG. 8, the substrate temperature is steeper than the lens temperature, and the gradient is 1.1. Therefore, the relationship between the substrate temperature and the lens temperature can be expressed as substrate temperature = 1.1 × lens temperature + constant. In this case, it has been experimentally confirmed that τB is substantially equal to a value obtained by multiplying the linear expansion coefficient of the lens array 110 by 0.9. Of course, when the configuration of the stereo camera module 100 is different, the relationship between the change in the lens temperature and the change in the substrate temperature is different, so the value of τB is also different. Thus, τ and τB, which are temperature coefficients used for calculating the corrected parallax ds, can be obtained experimentally in advance.

(Effect of embodiment)
As described above in detail with specific examples, in the distance measuring system according to the present embodiment, the temperature at which the distance measuring device 200 captures the first image and the second image with the stereo camera module 100 is described. The first correction value for correcting the error component due to the expansion and contraction of the baseline length B caused by the change in the environmental temperature based on the temperature difference Δt between the first image and the reference temperature, and the first image and the first image caused by the change in the environmental temperature A second correction value for correcting an error component due to a change in magnification of the two images is calculated. Then, the parallax ds ′ calculated based on the first image and the second image is corrected based on the first correction value and the second correction value to calculate the corrected parallax ds, and the calculated correction parallax ds is calculated. Using this, the distance Z to the subject is calculated. Therefore, according to the distance measuring system according to the present embodiment, the parallax ds ′ calculated from the first image and the second image is corrected with high accuracy according to the temperature at the time of capturing the first image and the second image. Thus, the distance Z to the subject can be accurately measured.

  Further, the parallax correction unit 206 of the distance measuring device 200 performs the calculation represented by the above formula (6) to correct a value obtained by dividing the second correction value by the value obtained by subtracting the first correction value from the parallax ds ′. In the case of calculating the parallax ds, the corrected parallax ds can be calculated by correcting the parallax ds ′ very accurately by strict calculation.

  In addition, the parallax correction unit 206 of the distance measuring device 200 performs the calculation represented by the above equation (7), and the corrected parallax is obtained by subtracting the first correction value from the value obtained by multiplying the parallax ds ′ and the second correction value. When calculating as ds, the corrected parallax ds can be calculated at low cost without using a division operation.

  In the distance measuring system according to the present embodiment, the functional configuration of the distance measuring apparatus 200 shown in FIG. 4 is, for example, as a result of the CPU of the microcomputer constituting the distance measuring apparatus 200 executing the distance measuring program. Can be realized. In this case, the distance measurement program executed by the CPU of the microcomputer is provided by being incorporated in advance in a ROM of the microcomputer, for example.

  The distance measuring program executed by the CPU of the microcomputer constituting the distance measuring apparatus 200 is an installable or executable file and is read by a computer such as a CD-ROM, flexible disk, CD-R, or DVD. You may comprise so that it may record on a possible recording medium and provide. Further, the distance measuring program executed by the CPU of the microcomputer constituting the distance measuring apparatus 200 may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Good. Further, the distance measuring program executed by the CPU of the microcomputer constituting the distance measuring apparatus 200 may be provided or distributed via a network such as the Internet.

  The ranging program executed by the CPU of the microcomputer constituting the ranging device 200 includes, for example, a first interface unit 201, a camera control unit 202, an image signal distortion correction unit 203, a parallax calculation unit 204, a temperature difference calculation unit 205, The module configuration includes a parallax correction unit 206, a distance calculation unit 207, and a second interface unit 208. As actual hardware, a CPU (processor) reads out and executes a distance measurement program from the ROM, and the above-described units. Are loaded on the main storage device (for example, RAM), the first interface unit 201, the camera control unit 202, the image signal distortion correction unit 203, the parallax calculation unit 204, the temperature difference calculation unit 205, the parallax correction unit 206, and the distance calculation unit. 207 and the second interface unit 208 are generated on the main storage device Going on.

  The specific embodiments of the present invention have been described above. However, the present invention is not limited to the above-described embodiments as they are, and may be embodied by modifying constituent elements without departing from the scope of the invention in the implementation stage. can do.

DESCRIPTION OF SYMBOLS 100 Stereo camera module 110 Lens array 111 1st lens 112 2nd lens 120 Board | substrate 121 1st image sensor 122 2nd image sensor 140 Temperature sensor 200 Distance measuring device 204 Parallax calculating part 205 Temperature difference calculation part 206 Parallax correction part 207 Distance calculation Part

JP 2007-322128 A JP 2009-250785 A

Claims (6)

A stereo camera module comprising: a first imaging system including a first lens; and a second imaging system including a second lens integrally formed with the first lens, and a first image of the subject imaged by the first imaging system. Input means for inputting one image and a second image of the subject imaged by the second imaging system;
Parallax calculation means for calculating a parallax that is a difference between the position of the subject in the first image and the position of the subject in the second image based on the first image and the second image;
Temperature difference calculating means for calculating a temperature difference that is a difference between a temperature at the time of capturing the first image and the second image and a reference temperature;
Based on the temperature difference, a first correction value for correcting an error component due to expansion and contraction of a baseline length, which is a distance between the optical axes of the first imaging system and the second imaging system, and the first image and A second correction value for correcting an error component due to a change in magnification of the second image is calculated, and the parallax is corrected based on the first correction value and the second correction value, thereby correcting parallax. Parallax correction means for calculating
Distance calculating means for calculating a distance to the subject based on the corrected parallax, the base line length without expansion and contraction, and the focal lengths of the first imaging system and the second imaging system. Ranging device.   2. The distance measurement according to claim 1, wherein the parallax correction unit calculates a value obtained by dividing the second correction value with respect to a value obtained by subtracting the first correction value from the parallax as the correction parallax. apparatus.   2. The distance measuring apparatus according to claim 1, wherein the parallax correction unit calculates a value obtained by subtracting the first correction value from a value obtained by multiplying the parallax and the second correction value as the correction parallax. . A first imaging system including a first lens and capturing a first image of a subject;
A second imaging system that includes a second lens integrally molded with the first lens, and that captures a second image of the subject;
Parallax calculation means for calculating a parallax that is a difference between the position of the subject in the first image and the position of the subject in the second image based on the first image and the second image;
Temperature difference calculating means for calculating a temperature difference that is a difference between a temperature at the time of capturing the first image and the second image and a reference temperature;
Based on the temperature difference, a first correction value for correcting an error component due to expansion and contraction of a baseline length, which is a distance between the optical axes of the first imaging system and the second imaging system, and the first image and A second correction value for correcting an error component due to a change in magnification of the second image is calculated, and the parallax is corrected based on the first correction value and the second correction value, thereby correcting parallax. Parallax correction means for calculating
Distance calculating means for calculating a distance to the subject based on the corrected parallax, the base line length without expansion and contraction, and the focal lengths of the first imaging system and the second imaging system. Ranging system. A stereo camera module comprising: a first imaging system including a first lens; and a second imaging system including a second lens integrally formed with the first lens, and a first image of the subject imaged by the first imaging system. A function of inputting one image and a second image of the subject imaged by the second imaging system;
A function of calculating a parallax that is a difference between the position of the subject in the first image and the position of the subject in the second image based on the first image and the second image;
A function of calculating a temperature difference which is a difference between a temperature at the time of capturing the first image and the second image and a reference temperature;
Based on the temperature difference, a first correction value for correcting an error component due to expansion and contraction of a baseline length, which is a distance between the optical axes of the first imaging system and the second imaging system, and the first image and A second correction value for correcting an error component due to a change in magnification of the second image is calculated, and the parallax is corrected based on the first correction value and the second correction value, thereby correcting parallax. A function for calculating
Ranging that causes the computer to realize the function of calculating the distance to the subject based on the corrected parallax, the baseline length without expansion and contraction, and the focal lengths of the first imaging system and the second imaging system program. A stereo camera module comprising: a first imaging system including a first lens; and a second imaging system including a second lens integrally formed with the first lens, and a first image of the subject imaged by the first imaging system. One image and a second image of the subject captured by the second imaging system are input, and a parallax that is a difference between the position of the subject in the first image and the position of the subject in the second image is calculated. A parallax correction method executed in an apparatus that performs:
Calculating a temperature difference that is a difference between a temperature at the time of imaging the first image and the second image and a reference temperature;
Calculating a first correction value for correcting an error component due to expansion / contraction of a baseline length, which is a distance between optical axes of the first imaging system and the second imaging system, based on the temperature difference;
Calculating a second correction value for correcting an error component due to a change in magnification of the first image and the second image based on the temperature difference;
And correcting the parallax based on the first correction value and the second correction value.

Images