JP2013257244A - Distance measurement device, distance measurement method, and distance measurement program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the accuracy of correcting a camera-position parameter in the calibration.SOLUTION: A known-article detection part, storing pattern information related to the image of a predetermined article, detects a position of a known article that is the article related to the pattern information on the basis of an image represented by input image signals of plural viewpoints. A correlation detection part detects a multiple-eye infinite distance zone that is a zone in an image representing an article existing in a multiple-eye infinite distance, the article for which a significant parallax between plural viewpoints cannot be detected from individual images represented by the input image signals of plural viewpoints, and determines correlation information indicating correlation among the viewpoints of the multiple-eye infinite zone. A position parameter calculation part calculates a position parameter indicating a positional relationship between the plural viewpoints on the basis of: the position of the known article detected by the article detection part; and the correlation information determined by the correlation detection part. A distance calculation part uses the calculated position parameter to calculate a distance to the article represented by the input image of plural viewpoints.

Description

  The present invention relates to a distance measuring device, a distance measuring method, and a distance measuring program.

 Multi-lens that measures the distance from a vehicle to obstacles and objects using the principle of triangulation based on images taken by multiple cameras (stereo cameras) mounted on the vehicle A distance measuring method is known. In this method, distance is measured on the assumption that a plurality of cameras are in a predetermined positional relationship. However, since the predetermined position changes due to vibrations, impacts, fluctuations in temperature, etc. during travel of the vehicle, in order to perform accurate distance measurement, calibration (calibration) is performed to measure the distance. It is essential to maintain accuracy.

As a calibration method, a method using a dedicated marker is known. In this method, a dedicated marker having a known shape is photographed by a plurality of cameras, and the relative position between the cameras is calculated by calculating the relative position between each camera and the dedicated marker. However, in the method using the dedicated marker, since the dedicated marker is photographed every time calibration is performed, handling such as special installation of the dedicated marker becomes complicated. Therefore, a method has been proposed in which calibration is performed using an object that exists on the road and has a known shape, such as a car registration number (number plate) or a road sign (for example, traffic signal) instead of a dedicated marker. ing.
For example, Patent Literature 1 discloses a method of detecting a shift in a distance detection device that detects a distance based on stereo images obtained by a plurality of cameras including an optical system and an imaging device, and includes the optical system. A first distance value from the known object image captured by the imaging device to the known object is obtained, and then a first distance value and a second distance value to the known object detected from the stereo image, A method of detecting a shift in the distance detection device by comparing the above is described.

JP 2008-292278 A

However, in the distance detection device described in Patent Document 1, shooting may be limited depending on the characteristics of an object that is a subject, and in this case, calibration cannot be performed. For example, when a license plate is used as a known object, the distance detection device often takes a picture of the license plate in the same direction. For this reason, the angle between the direction of the optical axis of the optical system and the direction of the license plate is substantially constant except in the case of rotation associated with turning left and right of the vehicle. In other words, images having the same or similar conditions when shooting are mainly acquired. Performing calibration by using an image having the same or similar conditions multiple times and taking the average thereof approximates to perform calibration using only one image. The direction of the license plate cannot be estimated with high accuracy, and consequently calibration with high accuracy cannot be performed. Patent Document 1 does not employ a process for correcting a parameter indicating a positional relationship between cameras.
An object of the present invention is to provide a distance measuring device, a distance measuring method, and a distance measuring program that improve the correction accuracy of camera position parameters in calibration.

(1) The present invention has been made to solve the above-described problems, and one aspect of the present invention stores pattern information related to a predetermined object image, and input image signals from a plurality of viewpoints are stored. A known object detection unit that detects the position of a known object that is an object related to the pattern information based on an image to be represented, and a significant parallax cannot be detected between the plurality of viewpoints from each image represented by the input image signal of the plurality of viewpoints A correspondence detection unit that detects a multi-lens infinity region, which is a region in an image representing an object at multi-lens infinity, and determines correspondence information indicating a correspondence relationship between viewpoints of the multi-lens infinity region; and the known A position parameter calculating unit that calculates a position parameter indicating a positional relationship between the plurality of viewpoints based on the position of the known object detected by the object detecting unit and the correspondence relationship information determined by the correspondence detecting unit; Using the position parameter calculator has calculated, the distance measuring apparatus characterized by comprising: a distance calculation unit that calculates a distance to the object represented by the input image of the plurality of viewpoints.

(2) Another aspect of the present invention is the distance measuring device described above, comprising a speed monitoring unit that determines whether or not the plurality of viewpoints are stopped based on the input speed information, and the speed monitoring When the unit determines that the plurality of viewpoints are stopped, the position parameter calculation unit calculates the position parameter.

(3) Another aspect of the present invention is the above-described distance measuring device, wherein the position parameter storage unit stores the position parameter calculated by the position parameter calculation unit, and the position parameter calculated by the position parameter calculation unit. Of the first distance to the object calculated based on the second distance to the object calculated based on the position parameter stored in the position parameter storage unit, the distance to the object calculated by the distance calculation unit A position parameter determination unit that stores the position parameter calculated by the position parameter calculation unit in the position parameter storage unit when it is determined that the first distance is closer to the third distance. And

(4) Another aspect of the present invention is the distance measurement apparatus described above, wherein the known object detection unit is configured to detect the known object based on the input image signals of the plurality of viewpoints at a distance closer to the multiview infinity. When not detected, the correspondence detection unit detects the multi-lens infinity region.

(5) Another aspect of the present invention is the distance measurement device described above, wherein the known object detection unit and the correspondence detection unit determine that the plurality of viewpoints are stopped in the speed monitoring unit. The correspondence information is determined by detecting the positions of the known objects using the input image signals of a plurality of viewpoints.

(6) Another aspect of the present invention is the distance measuring device described above, including an illumination control unit that controls whether light is emitted to the illumination device, wherein the correspondence detection unit is the illumination device. The multi-lens infinity region is detected based on an input image signal at the time of lighting input when the light source emits light and an input image signal at the time of turn-off input when the lighting device does not emit light. It is characterized by that.

(7) Another aspect of the present invention is the distance measurement device described above, wherein the brightness of an object related to an image represented by at least one of the input image signal when the light is turned on and the input image signal when the light is turned off is The multi-lens infinity region is detected when the difference between the signal values is larger than a predetermined value between the input image signal when turned on and the input image signal when turned off. And

(8) Another aspect of the present invention is the above-described distance measuring device, wherein the correspondence detection unit is input the plurality of viewpoints immediately before the speed monitoring unit determines that the plurality of viewpoints are stopped. The input image signal of the plurality of viewpoints input immediately after it is determined that the plurality of viewpoints are not stopped in the speed monitoring unit, and the speed monitoring unit has the plurality of viewpoints stopped. The multi-lens infinity region is detected based on the input image signals of the plurality of viewpoints input at the time of determination.

(9) Another aspect of the present invention is the distance measuring device described above, wherein the predetermined object is an automobile registration number mark.

(10) According to another aspect of the present invention, in the distance measurement method in the distance measurement device, the distance measurement device relates to pattern information related to an image of an object determined in advance based on an image represented by an input image signal of a plurality of viewpoints. A first step of detecting a position of a known object that is an object, and the distance measuring device is capable of detecting a significant parallax between the plurality of viewpoints from each image represented by the input image signal of the plurality of viewpoints. A second step of detecting a multi-lens infinity region, which is an area in an image representing an object, and determining correspondence information indicating a correspondence relationship between viewpoints of the multi-lens infinity region; A third step of calculating a position parameter indicating a positional relationship between the plurality of viewpoints based on the detected position of the known object and the determined correspondence information; and the distance measuring device includes the calculated position parameter. Using the meter, a distance measuring method characterized by having a a fourth step of calculating the distance to the object represented by the input image of the plurality of viewpoints.

(11) According to another aspect of the present invention, in a distance measurement program in a distance measurement device, a known object that is an object related to pattern information related to an image of an object determined in advance based on an image represented by an input image signal of a plurality of viewpoints A first procedure for detecting a position, a multi-view that is an area in an image representing an object at multi-lens infinity that cannot detect a significant parallax between the multiple viewpoints from each image represented by the input image signals of the multiple viewpoints A second procedure for determining correspondence information indicating a correspondence relationship between viewpoints of the multi-lens infinity region, detecting the infinite region; Third procedure for calculating a position parameter indicating a positional relationship between viewpoints, and a fourth procedure for calculating a distance to an object represented by the input image of the plurality of viewpoints using the calculated position parameter. A distance measuring program for execution.

  According to the present invention, in the distance measuring device, the distance measuring method, and the distance measuring program, the correction accuracy in the calibration of the camera position parameter is improved.

It is a block diagram showing the structural example of the distance measuring device which concerns on embodiment of this invention. It is an overhead view which shows the attachment position of the reference camera in this embodiment, and a non-reference camera. It is a block diagram showing the example of a structure of the known shape object position detection part which concerns on this embodiment. It is a conceptual diagram which shows the perspective projection camera model at the time of calculating the distance to a license plate. It is a block diagram showing the example of a structure of the detection part corresponding to a multi-lens infinity concerning this embodiment. It is a flowchart showing the camera position parameter determination process which concerns on this embodiment which concerns on this embodiment. It is a conceptual diagram which shows the perspective projection camera model used for the multi-eye distance measuring method.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram illustrating a configuration example of a distance measuring device 1 according to the present embodiment.
The distance measuring device 1 includes an imaging unit 11, a vehicle speed detecting unit 12, and a distance measuring unit 13. The photographing unit 11 includes a reference camera 11A and a non-reference camera 11B.

The imaging unit 11 captures an object (obstacle or object) behind the vehicle, generates an image signal indicating the captured image, and outputs the generated image signal to the distance measurement unit 13.
The reference camera A and the non-reference camera B of the imaging unit 11 are each a camera module. The camera module includes a solid-state imaging device such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, and an optical system including optical components such as a lens. The camera module can be configured to include a super wide angle lens or a fisheye lens with a wide angle of view. The optical axes of the reference camera A and the non-reference camera B are parallel and perpendicular to the base line.
The imaging unit 11 can capture an image with a wide field of view using a camera module including such a lens.

The reference camera 11A is one of the two camera modules, and is a camera module that serves as a reference for positions and processing described later. The non-reference camera 11B is the other camera module. When the photographing unit 11 is a stereo camera including left and right camera modules, for example, the reference camera 11A is a left camera module, and the non-reference camera 11B is a right camera module.
The characteristics of the optical system provided in the reference camera 11A and the non-reference camera 11B are indicated by camera internal parameters. Camera internal parameters include focal length, sensor size, lens distortion coefficient, and the like. These parameters are all known parameters.
In the following description, the reference camera 11A and the non-reference camera 11B may be collectively referred to simply as “camera”. The position of the camera refers to the position of the viewpoint of the optical system provided in the camera (the imaging surface at the focal point of the lens). The viewpoint is a position where the subject is visually recognized, and is a parameter indicating the position of the camera.
In the present embodiment, a single camera (camera unit) may be provided instead of the camera module as the reference camera 11A and the non-reference camera 11B. An arrangement example of the reference camera 11A and the non-reference camera 11B will be described later.

 The vehicle speed detection unit (speed detection unit) 12 detects the speed of a vehicle (hereinafter also referred to as “own vehicle”) on which the distance measuring device 1 is mounted. The vehicle speed detection unit 12 detects an electrical speed signal indicating the speed of the host vehicle from an existing speedometer, and outputs this signal to the distance measurement unit 13.

 The distance measuring unit 13 includes a vehicle speed monitoring unit (hereinafter also referred to as “speed monitoring unit”) 132, and a known shape object position detection unit (hereinafter also referred to as “position information calculation unit”) 133. , Multi-lens infinity correspondence detection unit (hereinafter also referred to as “region correspondence detection unit”) 134, camera position parameter calculation unit (hereinafter also referred to as “position parameter calculation unit”) 135, camera. A position parameter determination unit (hereinafter also referred to as “position parameter determination unit”) 136, a camera position parameter storage unit (hereinafter also referred to as “position parameter storage unit”) 137, and a distance calculation unit 138. Consists of including. The distance measuring unit 13 may be configured as a single distance measuring device that is detachable from other components, that is, the photographing unit 11 and the vehicle speed detecting unit 12.

 The vehicle speed monitoring unit 132 determines whether the host vehicle is stopped based on the speed signal input from the vehicle speed detection unit 12. When the input speed signal indicates that the speed is 0 (zero), the vehicle speed monitoring unit 132 determines that the host vehicle is stopped and generates a stop signal. The vehicle speed monitoring unit 132 outputs the generated stop signal to the known shape object position detection unit 133 and the multi-lens infinity correspondence detection unit 134. The vehicle speed monitoring unit 132 generates a travel signal when the input speed signal indicates a certain speed. The vehicle speed monitoring unit 132 outputs the generated travel signal to the known shape object position detection unit 133 and the multi-lens infinity correspondence detection unit 134.

The known shape object position detection unit 133 calculates the relative position of the object with respect to each camera based on the image signal input from the imaging unit 11. The relative position is a three-dimensional relative coordinate value of a target object with respect to a reference object (in the above example, the reference camera 11A or the non-reference camera 11B), and may include a three-dimensional relative angle value.
When a stop signal is input from the vehicle speed monitoring unit 132, the known shape object position detection unit 133 simultaneously receives image signals from the reference camera 11A and the non-reference camera 11B.
The object whose relative position is to be calculated is a known shape object. A known shape object is an object whose shape and size are known. Hereinafter, a case where the known shape object is a “number plate” of another succeeding vehicle will be described as an example. When a stop signal is input from the vehicle speed monitoring unit 132, the known shape object position detection unit 133 calculates a relative position. The known shape object position detection unit 133 generates relative position information indicating the calculated relative position, and the generated relative position information is read from the camera position parameter calculation unit 135.

The multi-lens infinity correspondence detection unit 134 detects a multi-lens infinity region indicating an image of an object at a multi-lens infinity distance, which will be described later, from the images represented by the image signals input from the reference camera 11A and the non-reference camera 11B. Detect each.
Hereinafter, an image signal input from the reference camera 11A to the known shape object position detection unit 133 and the multi-lens infinity correspondence detection unit 134 is referred to as a reference image signal, and an image represented by the reference image signal is referred to as a reference image. An image signal input from the non-reference camera 11B is called a non-reference image signal, and an image represented by the non-reference image signal is called a non-reference image.
The multi-lens infinity correspondence detection unit 134 includes correspondence information between multi-lens infinity regions (correspondence information) indicating a correspondence relationship between the detected infinity region in the reference image and the detected infinity region in the non-reference image. Is generated. The multi-eye infinity region correspondence information is information indicating a correspondence relationship between each coordinate included in the infinity region of the reference image and a coordinate included in the infinity region of the non-reference image. In other words, the multi-lens infinity region correspondence information includes the region in which the object is represented in the infinity region of the reference image and the region in which the object is represented in the infinite edge region of the non-reference image. It is the information which shows the corresponding relationship.
The correspondence information between multi-lens infinity regions generated by the multi-lens infinity correspondence detection unit 134 is read from the camera position parameter calculation unit 135. The configuration of the multiview infinity correspondence detecting unit 134 will be described later.

Here, the multi-lens infinity is a distance corresponding to coordinates at which the difference between the coordinates in the reference image and the corresponding coordinates in the non-reference image is equal to or less than a predetermined number of pixels (for example, 1 pixel). However, it is assumed that the optical axes of the reference camera 11A and the non-reference camera B are parallel to each other and the horizontal directions of the respective image sensors (imaging devices) are on the same plane. That is, multi-lens infinity is a parallax between images taken from different viewpoints, and is a distance at which significant non-zero parallax is not detected between the viewpoints. In other words, it is a far distance in which the distance of the object represented by each image cannot be determined based on the parallax between the images. This distance depends on the positional relationship between the cameras, the characteristics of the optical system and the imaging system (for example, pixel pitch). Therefore, if the relative position of the non-reference camera 11B with respect to the reference camera 11A (for example, the parallelism of the optical axis, the interval, etc.) changes with the vibration at the start, movement, and stop of the vehicle, the passage of time, etc. In the non-reference image, a region representing an object existing at multi-lens infinity changes. Therefore, in addition to the position information of the known shape object, the camera parameter calibration (for example, the relative direction of the optical axis) using the relative position between the cameras (for example, the relative direction of the optical axis) is used as described later. Perform calibration). By using the camera parameters that have been calibrated, the calculation accuracy for the distance to the photographed object can be improved. Note that a distance at which a significant parallax is detected among parallaxes between images taken from different viewpoints, that is, a distance that is not multi-lens infinity is called multi-lens finite.
In addition, when the reference camera 11A and the non-reference camera 11B are provided with a lens having a wide viewing angle, an area where an image of a known shape object is represented in the reference image and the non-reference image becomes small. For this reason, even when camera parameters are calibrated using only an image of a known shape object, the calculation accuracy of the relative position between the cameras is lowered. Therefore, when calculating the relative position between the cameras (described later), the calculation accuracy can be improved by using the correspondence information between the multi-lens infinity regions.

  Note that the known shape object position detection unit 133 and the multi-lens infinity correspondence detection unit 134 use an image signal input immediately after it is determined that the moving host vehicle has stopped for the above-described processing. May be. Here, the known shape object position detection unit 133 and the multi-lens infinity correspondence detection unit 134 obtain an image input immediately after the signal input from the vehicle speed monitoring unit 132 transitions again from the travel signal to the stop signal. Each is used in the next processing. Accordingly, it is possible to avoid repeating the same process using an image signal obtained by photographing the same subject under the same conditions while stopped. As a result, unnecessary processing for improving accuracy is avoided, so that power consumption can be reduced.

The camera position parameter calculation unit 135 calculates the camera position parameter based on the relative position information read from the known shape object position detection unit 133 and the correspondence between the multiview infinity areas read from the multiview infinity correspondence detection unit 134. To do. The camera position parameter is a parameter indicating a relationship of relative positions between cameras. In other words, the camera position parameter is determined based on the three-dimensional relative coordinate values (X, Y, Z) of the viewpoint of the non-reference camera 11B with the viewpoint of the reference camera 11A as the origin and the direction of the optical axis of the reference camera 11A. It includes a total of six parameters consisting of three-dimensional relative angle values (θ, φ, ψ) in the direction of the optical axis of the reference camera 11B.
The camera position parameter calculation unit 135 outputs the calculated camera position parameter to the camera position parameter determination unit 136. Processing for calculating the camera position parameter will be described later.

 The camera position parameter determination unit 136 receives the camera position parameter (newly calculated) from the camera position parameter calculation unit 135, and reads the camera position parameter (before calibration) from the camera position parameter storage unit 137. The camera position parameter determination unit 136 compares the input camera position parameter with the read camera position parameter, determines a more appropriate camera position parameter, and determines the more appropriate camera position parameter as the camera position parameter storage unit 137. To remember. Camera position parameter determination processing performed by the camera position parameter determination unit 136 or other components will be described later.

 The distance calculation unit 138 reads the camera position parameters stored in the camera position parameter storage unit 137, and receives image signals from the reference camera 11A and the non-reference camera 11B, respectively. The distance calculation unit 138 uses the read camera position parameter to represent the image of each pixel of the reference image based on the image signal input from the reference camera 11A and the image signal input from the non-reference camera 11B. Calculate the distance to the moving object. The distance calculation unit 138 outputs a distance signal representing the calculated distance to the outside of the distance measurement unit 13. When calculating the distance, the distance calculation unit 138 uses, for example, a multi-eye distance measurement method based on the principle of triangulation. The multi-eye distance measuring method will be described later.

(Example of camera mounting position)
Next, examples of attachment positions of the reference camera 11A and the non-reference camera 11B will be described.
FIG. 2 is an overhead view showing the attachment positions of the reference camera 11A and the non-reference camera 11B in the present embodiment.
The center of FIG. 2 shows the host vehicle 21 on which the distance measuring device 1 is mounted. The left side of FIG. 2 shows the rear side of the host vehicle 21, and the upper and lower sides of FIG.
In FIG. 2, the reference camera 11 </ b> A and the non-reference camera 11 </ b> B are installed behind the host vehicle 21 with the optical axis directed rearward. The coordinates in the horizontal direction and the height direction of the reference camera 11A and the non-reference camera 11B are substantially equal to each other. Therefore, most of the visual field of the reference camera 11A and the visual field of the non-reference camera 11B overlap, and an object in the overlapped region is common between the reference image and the non-reference image. Thereby, the object behind the host vehicle 21 is represented in the reference image and the non-reference image. A reference image and a non-reference image captured using a camera having a lens with a large angle of view as the reference camera 11A and the non-reference camera 11B are convenient for the driver to confirm safety behind the host vehicle 21. is there.

 In the example shown in FIG. 2, the reference camera 11 </ b> A and the non-reference camera 11 </ b> B are installed in the rear part of the host vehicle 21 with the optical axis directed rearward, but the present embodiment is not limited thereto. The reference camera 11 </ b> A and the non-reference camera 11 </ b> B may be installed in the front part of the host vehicle 21 with the optical axis directed forward. The reference camera 11 </ b> A and the non-reference camera 11 </ b> B may be installed on the left side surface of the host vehicle 21 with the optical axis facing the left side, or installed on the right side surface of the host vehicle 21 with the optical axis facing the right side. It may be.

(Configuration example of known shape object position detector)
Next, a configuration example of the known shape object position detection unit 133 according to the present embodiment will be described.
FIG. 3 is a block diagram illustrating a configuration example of the known shape object position detection unit 133 according to the present embodiment. The known shape object position detection unit 133 may be referred to as a known shape object detection unit (hereinafter also referred to as “object detection unit”) 1331 and a known shape object position calculation unit (hereinafter referred to as “object position calculation unit”). 1332 and a known shape object position storage unit (hereinafter also referred to as “object position storage unit”) 1333.

 Based on the reference image signal and the non-reference image signal, the known shape object detection unit 1331 detects a region representing an image of a known shape object in each of the reference image and the non-reference image. In detecting a region representing an image of a known shape object, the known shape object detection unit 1331 uses a known method such as template matching. The known shape object detection unit 1331 performs binarization processing such as edge extraction on each of the input image signals, for example, and determines an edge portion and other portions. The edge portion is a set of pixels in which the spatial change in luminance value is larger than a predetermined spatial change value.

Hereinafter, a case where the known shape object is a license plate will be described as an example. The known shape object detection unit 1331 detects an edge portion (hereinafter referred to as a quadrilateral region) whose shape is a quadrangle or a quadrilateral shape among the determined edge portions (peripheral portions). Here, the known shape object detection unit 1331 surrounds a spatially closed region in the edge portion, and has an edge having four portions (vertices) whose curvature radius is smaller than a predetermined curvature radius. The part is determined to be a quadrilateral area. The determined quadrilateral area is a candidate area representing the license plate.
When the shape of the license plate is a rectangle or a shape that approximates a rectangle (for example, a figure in which each vertex of the rectangle is rounded), the known shape object detection unit 1331 includes a pattern storage unit. In the pattern storage unit, information indicating the shape of the license plate (for example, an aspect ratio which is a ratio between the horizontal width and the vertical width of the rectangle), and a character pattern indicating characters to be displayed on the license plate are stored in advance. Remember. That is, information stored as pattern information relating to an image representing a license plate is information indicating the shape of the license plate and a character pattern.
The known shape object detection unit 1331 performs projective transformation on the detected quadrilateral region into a rectangular region having an aspect ratio stored in advance using, for example, Equation (1).

In Expression (1), (X, Y) is the coordinate value of each vertex of the quadrilateral area before conversion. (X ′, Y ′) are the coordinate values of the corresponding vertices of the converted rectangular area. The coefficients a, b,..., I are conversion coefficients. The coefficient s is a temporary value calculated for each coordinate (X, Y). The known shape object detection unit 1331 uses (sX ′, sY ′, s) on the left side of Equation (1), the element values in the first column and the second column, and the coefficient s that is the element value in the third column, respectively. (X ′, Y ′) is calculated by dividing by.
The known shape object detection unit 1331 collates the character image arranged in the converted rectangular area with the character pattern stored in advance. The known shape object detection unit 1331 determines that the detected quadrilateral region is a region that represents a license plate when all of the character images are collated with the character pattern or when the number is greater than a predetermined number. License plate area). In the collation, for example, when the index value indicating the degree of similarity of the comparison target data indicates that it is more similar than the predetermined index value, it is determined that the collation is successful. The known shape object detection unit 1331 uses, for example, SAD (Sum of Absolute Differences). In that case, the known shape object detection unit 1331 determines that the collation is successful when the SAD is smaller than a predetermined threshold.

The known shape object detection unit 1331 is a known shape object that represents a license plate area when the stop signal input from the vehicle speed monitoring unit 132 indicates the stop of the host vehicle and the detection target license plate is stopped. Generate region information. The known shape object detection unit 1331 outputs the generated known shape object region information and the conversion coefficient used for the above-described projective transformation to the known shape object position calculation unit 1332.
Here, the known shape object detection unit 1331 determines whether or not the license plate is stopped based on the reference image signal. Here, the known shape object detection unit 1331 calculates a total value of differences between adjacent frames of gradation values included in the license plate region, and when the calculated total value is smaller than a predetermined threshold, the license plate It is determined that the vehicle is stopped (the vehicle on which the license plate is installed is stopped). Thereby, it is detected that the change in the gradation value in the license plate region is small. The known shape object detection unit 1331 may be a difference between frames separated by a predetermined number of frames (for example, two frames) instead of the difference between adjacent frames.

 The reason why the license plate is stopped is that if the relative speed of the object to be imaged with respect to the camera is not zero, a blurred region (blurring) is generated around the object in the imaged image. When the object is a license plate, the coordinates in the image of the four sides of the license plate cannot be determined accurately. Therefore, when the license plate is stopped, the position where the license plate is displayed can be accurately detected by detecting the license plate region. Therefore, it is possible to calculate the license plate area with high accuracy by determining that the host vehicle and the license plate are stopped and using an image signal representing a clear image acquired when the vehicle is stopped.

The known shape object position calculation unit 1332 calculates the relative position of the license plate corresponding to the corresponding camera based on the known shape object region information and the conversion coefficient input from the known shape object detection unit 1331. The known shape object position calculation unit 1332 stores in advance camera internal parameters of the reference camera 11A and the non-reference camera 11B and a size parameter indicating the size of the license plate. The size parameter is, for example, the length of four sides of the license plate.
The known shape object position calculation unit 1332 calculates the direction of the license plate with respect to each camera using the input known shape object region information, the conversion coefficient, and the stored camera internal parameters and size parameters.

  Here, the known shape object position calculation unit 1332 calculates the relative three-dimensional angular direction of the license plate with respect to each camera based on the conversion coefficients a, b,..., I input from the known shape object detection unit 1331. The conversion matrix including the conversion coefficient as an element indicates the direction of the license plate with respect to each camera. Based on this transformation matrix, the known shape object position calculation unit 1332 calculates the three-dimensional relative angle direction (θ, φ, ψ) so as to satisfy Expression (2).

  The known shape object position calculation unit 1332 calculates the distance from each camera to the license plate based on the calculated relative three-dimensional angular direction. Here, the known shape object position calculation unit 1332 determines the temporary license plate area based on the size parameter stored in advance, the camera internal parameter, the calculated relative three-dimensional angular direction, and the temporary relative distance from each camera to the license plate. The size of is calculated. Then, the known shape object position calculation unit 1332 calculates a temporary relative distance such that the calculated size of the temporary license plate region is the same as the size of the known shape object region indicated by the input known shape object region information. calculate. The known shape object position calculation unit 1332 determines the calculated temporary relative distance as the distance from each camera to the license plate. The known shape object position calculation unit 1332 generates information indicating the relative three-dimensional angle direction calculated for each camera and the distance from each camera to the license plate as the known shape object position information, and the generated known shape object position information is the shape. This is stored in the known object position storage unit 1333.

  The known shape object position storage unit 1333 stores the known shape object position information stored in the known shape object position calculation unit 1332 by a predetermined number (for example, 10) in the past. The known shape object position information stored in the known shape object position storage unit 1333 is read from the camera position parameter calculation unit 135 as the above-described relative position information.

Here, the principle by which the known shape object position calculation unit 1332 calculates the distance from each camera to the license plate will be described with reference to FIG.
FIG. 4 is a conceptual diagram showing a perspective projection camera model for calculating the distance to the license plate.
In FIG. 4, reference numeral 71 denotes a focal point of the camera, 72 denotes an image surface which is a surface of an image sensor provided in the camera, and 73 denotes an area (a shape known object area) where a license plate is displayed. Reference numerals 741, 742, and 743 denote license plates placed at different temporary relative distances. The temporary relative distance decreases in the order of 741, 742, and 743.
Here, since the relative direction of the license plate is known, the known shape object position calculation unit 1332 changes the temporary relative distance so that one point of the license plate region is always projected on the same coordinates in the image. (Search). One point of the license plate region is the left end of the license plate region in the example shown in FIG. As shown in 741, when the temporary relative distance is longer than the actual relative distance, the image of the license plate projected on the image plane 72 becomes narrower than the image of the known shape object. As indicated by 743, when the provisional relative distance is shorter than the actual relative distance, the image of the license plate projected on the image plane 72 is wider than the image of the known shape object. As shown in 742, when the temporary relative distance is the same as the actual relative distance, the image of the license plate projected on the image plane 72 overlaps the image of the known shape object. Therefore, the known shape object position calculation unit 1332 can calculate the distance to the license plate by performing the above-described processing.

(Configuration example of multi-lens infinity detection unit)
Next, a configuration example of the multiview infinity correspondence detecting unit 134 in the present embodiment will be described.
FIG. 5 is a block diagram illustrating a configuration example of the multiview infinity correspondence detecting unit 134 in the present embodiment. The multi-lens infinity correspondence detection unit 134 includes a multi-lens infinity region detection unit 1341, a brightness detection unit 1342, an illumination control unit 1343, an inter-region correspondence detection unit 1344, and an inter-region correspondence storage unit 1345. .

The multi-eye infinity region detection unit 1341 detects the multi-eye infinity region from the reference image signal input from the reference camera 11A and the non-reference image signal input from the non-reference camera B, respectively. The multi-lens infinity region detection unit 1341 generates multi-lens infinity region information indicating the detected multi-lens infinity region, and outputs the generated multi-lens infinity region information to the inter-region correspondence detection unit 1344.
Here, when the illumination control signal input from the illumination control unit 1343 indicates lighting, the multi-lens infinity region detection unit 1341 holds the input reference image signal (or non-reference image signal) as the lighting image signal. To do. When the illumination control signal input from the illumination control unit 1343 indicates extinction, the multi-lens infinity region detection unit 1341 holds the input reference image signal (or non-reference image signal) as the extinguishing image signal. Here, the illumination control signal is a signal indicating that a lighting device (not shown) is turned on (turned on) or turned off (turned off).
The multi-eye infinity region detection unit 1341 is a region in which the difference in gradation value for each pixel between the held on-image signal and the unlit image signal is smaller than a predetermined difference value threshold, that is, a gradation value. An area where there is little or no change is defined as a multi-view infinity area. This is because the closer the object is to the camera, the higher the brightness because it reflects the light emitted by the illuminating device, and the higher the luminance because almost no light emitted by the illuminating device reaches an object at multi-lens infinity. Based on what must not be.
It should be noted that the multi-lens infinity region detection unit 1341 has the lighting-time image signal and the unlit image signal regardless of the time when the illumination control signal indicating lighting is input before or after the time when the illumination control signal indicating lighting is input. Based on the signal, it can be determined as a multi-view infinity region.

 The multi-eye infinity region detection unit 1341 determines the threshold value of the above-described difference value based on the illuminance input from the brightness detection unit 1342. Here, the multi-lens infinity region detection unit 1341 determines the threshold value of the difference value so that it increases as the illuminance increases and decreases as the illuminance decreases. The multi-eye infinity region detection unit 1341 includes a storage unit that stores the illuminance and the threshold value of the difference value in association with each other, and reads the threshold value of the difference value corresponding to the input illuminance. When the surrounding brightness is too bright, the gradation value does not change significantly even when the lighting device is turned on. On the other hand, if it is too dark, an image representing an object that is infinitely far from the reference image or non-reference image is not recognized. For this reason, the multi-lens infinity region can be appropriately detected by using the threshold value of the difference value corresponding to the illuminance input from the brightness detection unit 1342.

 The multi-lens infinity region detection unit 1341 performs processing for detecting the multi-lens infinity region described above when the stop signal input from the vehicle speed monitoring unit 132 indicates that the host vehicle has stopped. As a result, the multi-view infinity region can be accurately detected. On the other hand, when the own vehicle moves (for example, rotates) between when the light is turned on and when the light is turned off, the multi-eye infinity region detection unit 1341 moves an image of an object existing at multi-eye infinity on the coordinates in the image. End up. For this reason, the object represented in the image differs between the image signal at the time of lighting and the image signal at the time of turning off, and the multi-lens infinity region cannot be detected correctly.

 Based on the known shape object position information input from the known shape object position calculation unit 1332, the multiview infinity area detection unit 1341 determines whether or not a license plate exists at a distance of the multiview finite distance. When it is determined that there is no license plate, the multi-lens infinity region detection unit 1341 starts detecting the multi-lens infinity region described above. When determining whether or not a license plate is present at a multi-lens finite distance, the multi-lens infinity region detection unit 1341 determines that the distance from the camera related to the position of the known shape object indicated by the known shape object position information. The determination is made based on the fact that it is smaller than the threshold value of the distance at which the multi-lens finite distance is obtained. The multi-eye infinity region detection unit 1341 is set in advance with a threshold for the distance at which multi-eye infinity is set. This distance threshold is a value determined by the distance (baseline length) between the reference camera 11A and the non-reference camera 11B, the angle of view of each camera, and the resolution. Note that a distance larger than the distance threshold is a multi-eye infinity. As a result, the multi-lens infinity region detection unit 1341 detects the multi-lens infinity region when it is determined that there is no license plate of another vehicle (for example, a preceding vehicle) at a multi-lens finite distance. When it is determined that the license plate of the other vehicle does not exist at a multi-lens finite distance, other objects are often not at the multi-lens finite distance, and thus the multi-lens infinity region tends to be wider. For this reason, when it is determined that there is no license plate of another vehicle at a multi-lens finite distance, the inter-region correspondence detection unit 1344 detects a wider multi-lens infinity region, so The probability that the corresponding information is erroneously detected can be reduced.

The brightness detection unit 1342 is the brightness of the object (ambient brightness) represented by the reference image based on the average value of the gradation value calculated based on the reference image signal input from the reference camera 11A and the exposure amount. Illuminance indicating is detected. The exposure amount is a parameter determined by the amount of light (aperture value, F value) that can be received by the image sensor included in the reference camera 11A and the exposure time per frame. In the brightness detection unit 142, an exposure amount is set in advance, and brightness detection information indicating the correspondence between the average value of the gradation values and the exposure amount and the illuminance is stored in advance in the storage unit included in the own unit. Remember. The brightness detection unit 142 reads out the illuminance corresponding to the calculated average value of the gradation within the image and the set exposure amount from the storage unit, and outputs the read illuminance to the multi-lens infinity region detection unit 1341. Thereby, the brightness of the object represented by the reference image can be detected.
In the present embodiment, whether or not the region that is originally multi-lens infinity is detected as not multi-lens infinity,
If the multi-lens infinity region is detected to some extent, and the non-multi-lens infinity region is not detected as multi-lens infinity, the effect on the generation result of the multi-lens infinity region correspondence information is limited. . Therefore, the brightness detection unit 142 detects illuminance indicating brightness that is not detected as multi-lens infinity in a region that is not originally multi-lens infinity. Thereby, the correspondence information between multi-view infinity areas is appropriately calculated.

  The illumination control unit 1343 controls whether or not to turn on an illumination device (not shown) installed in the host vehicle. The illumination control unit 1343 generates an illumination control signal indicating whether the illumination device is turned on or off based on a predetermined time interval or an operation input by the user. The illumination control unit 1343 outputs the generated illumination control signal to the multiview infinity region detection unit 1341 and a switch unit (not shown). A switch part switches the presence or absence of the supply of the electric power to an illuminating device. That is, when the illumination control signal input from the illumination control unit 1343 indicates lighting, the switch unit supplies power to the lighting device, and when the lighting control signal indicates extinguishing, the switch unit supplies power to the lighting device. Stop supplying. Therefore, when the illumination control signal indicates lighting, the lighting device emits light, and when the lighting control signal indicates extinction, the lighting device does not emit light.

  The lighting device is a lighting device that is installed in the rear portion of the host vehicle and emits light toward the rear. The lighting device may be a lighting device (for example, a backlight) installed in advance at the time of manufacturing the host vehicle, or may be a lighting device installed after shipment. However, since the illumination device is used for multi-eye infinity detection, a sufficient amount of light is required to reach the same distance as the threshold between multi-eye infinity and multi-eye finite. Therefore, it is desirable to use a flashlight with a large amount of light as the lighting device. Since the light emitted from the normal flashlight may hinder the visual recognition by the driver driving the succeeding vehicle, an invisible light flashlight that emits invisible light such as infrared rays may be used as the illumination device. When an invisible light flashlight is used as the illumination device, the reference camera 11A and the non-reference camera 11B include an image sensor that detects invisible light emitted by the illumination device as the above-described image sensor.

  The inter-region correspondence detection unit 1344 generates multi-eye infinity region correspondence information based on the multi-view infinity region information for each camera input from the multi-view infinity region detection unit 1341. The correspondence between the multi-view infinity regions can be considered as a correspondence relationship indicating a perspective transformation between the reference image and the non-reference image. The perspective transformation is a transformation from a two-dimensional image representing a three-dimensional space observed from a certain viewpoint to a two-dimensional image representing the three-dimensional space observed from another viewpoint. Therefore, the inter-region correspondence detection unit 1344 detects feature points for each of the multi-view infinity regions of the reference image and the non-reference image using the existing feature point detection method, for example, and the reference of each detected feature point Correspondence between images and non-reference images is detected for four points. For example, the inter-region correspondence detection unit 1344 detects a feature point by using a SURF (Speeded Up Robust Features) feature amount for each image. In the present embodiment, the feature points may be detected using not only the SURF feature value but also other feature values. The feature amount to be used may be a feature amount that is robust to linear transformation such as rotation of an image and has few false detections of feature points. After that, the inter-region correspondence detection unit 1344 searches for the correspondence between the feature points so that the distance between the four-dimensional feature vectors representing the set of feature points in each image becomes the smallest. The inter-region correspondence detection unit 1344 calculates a two-dimensional perspective transformation matrix based on the detected correspondence between the four feature points. The inter-region correspondence detection unit 1344 stores the calculated perspective transformation matrix in the inter-region correspondence storage unit 1345 as multi-eye infinity region correspondence information.

  The inter-region correspondence storage unit 1345 stores a predetermined number (for example, 10) of past correspondence information between multi-view infinity regions stored by the inter-region correspondence detection unit 1344. In addition, the correspondence information between multi-lens infinity regions stored in the inter-region correspondence storage unit 1345 is read from the camera position parameter calculation unit 135.

In the above description, the case where the multi-lens infinity region is detected using image signals input when the lighting device is turned on and off is described as an example. However, the present embodiment is not limited to this. For example, when the host vehicle moves so that the direction of each camera does not rotate, the multi-lens infinity region detection unit 1341 has a predetermined change amount of the gradation value between the image signals input before and after the movement. An area smaller than the threshold value may be determined as a multi-view infinity area. Here, the vehicle speed monitoring unit 132 is based on a speed signal input from the vehicle speed detection unit 12 and immediately before the own vehicle decelerates and stops, that is, a predetermined time from the time when the own vehicle stops (for example, 5 Time that precedes (second) is detected (preceding time). The multi-eye infinity region detection unit 1341 may determine the multi-eye infinity region based on the image signal input at the detected preceding time and the image signal input at the time when the host vehicle is stopped.
Similarly, the vehicle speed monitoring unit 132, based on the speed signal input from the vehicle speed detection unit 12, immediately after the departure of the own vehicle, that is, a predetermined time from the time when the own vehicle starts to move (for example, 5 seconds). ) (Delayed time) may be detected. In this case, the multi-lens infinity region detection unit 1341 determines the multi-lens infinity region based on the image signal input at the detected subsequent time and the image signal input at the time when the host vehicle is stopped.

(Process to calculate camera position parameters)
Next, a camera position parameter calculation method performed by the camera position parameter calculation unit 135 according to the present embodiment will be described.
The camera position parameter calculation unit 135 reads the shape-known object position information from the shape-known object position storage unit 1333 and the multi-eye infinity region correspondence information from the inter-region correspondence storage unit 1345, respectively, for the past predetermined number of times. The camera position parameter calculation unit 135 calculates the camera position parameter based on the read shape known object position information and the correspondence information between multi-lens infinity regions. The camera position parameter calculation unit 135 may calculate a camera position parameter obtained by averaging the calculated camera position parameters by the number of times.

  The camera position parameter calculation unit 135 calculates, for example, a relative position parameter of the non-reference camera 11B with respect to the reference camera 11A as a camera position parameter using nonlinear optimization calculation. The nonlinear optimization calculation is an operation for calculating a variable vector x that minimizes the function value y for the predetermined nonlinear function f shown in Expression (3).

In equation (3), the function f is called an objective function.
Next, an example of processing for calculating camera position parameters will be described.
The camera position parameter calculation unit 135 extracts the three-dimensional relative coordinate values (x _B , y _B , z _B ) of the license plate with respect to the non-reference camera 11B from the read shape known object position information. The camera position parameter calculation unit 135 uses the expression (2) from the extracted three-dimensional relative coordinate values (x _B , y _B , z _B ) to determine the three-dimensional relative coordinate values (x _A , y of the license plate for the reference camera 11A). _A , z _A ) is calculated.

In equation (4), by substituting the coordinate value of each vertex of the license plate detected as the three-dimensional relative coordinate value (x _B , y _B , z _B ), the coordinate value (x _A, y _A, z _A) is calculated.
The camera position parameter calculation unit 135 represents the vertexes on the reference image based on the camera internal parameters set in advance from the calculated coordinate values (x _A , y _A , z _A ) of each vertex with respect to the reference camera 11A. Coordinate values (X _B , Y _B ) are calculated. The calculated coordinate values (X _B , Y _B ) are coordinate values projected from the three-dimensional space onto the imaging surface of the reference camera 11A, which is a two-dimensional space, and the camera position parameters (x, y, z, It can be seen as a function of θ, φ, ψ).
On the other hand, the camera position parameter calculation unit 135 extracts the coordinate values (X _A , Y _A ) of each vertex of the license plate represented in the reference image from the read shape known object position information.
The camera position parameter calculation unit 135 performs the calculation of the coordinate values (X _B , Y _B ) and the extraction of the coordinate values (X _A , Y _A ) for each of the past predetermined times described above. Here, in order to distinguish the four vertices k _np and the number of times n _np , these coordinate values are (X _{A, nnp, knp} , Y _{A, nnp, knp} ), (X _{B, nnp, knp} , Y _{B). , Nnp, knp} ).

The camera position parameter calculation unit 135 minimizes an error of the calculated coordinate value (X _B , Y _B ) of each vertex of the license plate image with respect to the extracted coordinate value (X _A , Y _A ). Calculate (x, y, z, θ, φ, ψ). An example of the objective function f (x, y, z, θ, φ, ψ) representing this error is shown in Equation (5).

  When calculating the position parameters (x, y, z, θ, φ, ψ) that minimize the objective function f (x, y, z, θ, φ, ψ), the camera position parameter calculation unit 135, for example, A nonlinear optimization calculation method such as Newton's method is used. Thereby, the camera position parameter calculation unit 135 can calculate the camera position parameter using the image on which the license plate is displayed.

In addition, when the camera position parameter calculation unit 135 calculates the objective function f (x, y, z, θ, φ, ψ) position parameter (x, y, z, θ, φ, ψ), the expression (5) It is also possible to use an objective function obtained by adding a constraint based on the correspondence between multi-view infinity regions to the objective function. An example of such an objective function is shown in equation (7). Thereby, the position parameters (x, y, z, θ, φ, ψ) can be calculated with higher accuracy.
The coordinate values (X _A , Y _A ) and (X _B , Y _B ) of each vertex of the license plate represented in the reference image and the non-reference image are extracted from the read shape known object position information. Where regions license plate is detected from the input image which is included in the multi-view infinity region, the extracted coordinate values of the vertices respectively _{(X A∞, Y A∞),} (X B∞, _YB∞ ). In the case of relative angles (θ, φ, ψ) between cameras, the camera position parameter calculation unit 135 calculates the coordinate values (X _BA , Y _BA ) in the reference image corresponding to the coordinate values (X _B∞ , Y _B∞ ). For example, the calculation is performed using Expression (6).

In Expression (6), P _A and P _B are camera internal parameter matrices of the reference camera 11A and the non-reference camera 11B, respectively. P _A and P _B are 3-column and 3-column matrices each including camera internal parameters in each element.
That is, equation (6) indicates that the coordinate values (X _BA , Y _BA ) are calculated by transparent transformation of the coordinate values (X _B∞ , Y _B∞ ).
The camera position parameter calculation unit 135 performs the calculation of the coordinate values (X _BA , Y _BA ) and the extraction of the coordinate values (X _A∞ , Y _A∞ ) for each of the past predetermined times described above. Here, four apexes _{k N∞,} in order to distinguish the number _{n N∞,} these coordinate values _{(X BA, nn∞, kn∞,} Y BA, nn∞, kn∞), (X A∞ _{, Nn∞, kn∞, YA∞, nn∞} , _kn∞ ).
The camera position parameter calculation unit 135 adds the magnitude of the error between the calculated coordinate values (X _BA , Y _BA ) and the extracted coordinate values (X _A∞ , Y _A∞ ) to the objective function of Expression (6). A position parameter (x, y, z, θ, φ, ψ) that minimizes the obtained objective function f (x, y, z, θ, φ, ψ) is calculated. The objective function f (x, y, z, θ, φ, ψ) to which the magnitude of this error is added is expressed by Expression (7).

In Expression (7), the second term on the right side is a term representing the magnitude of the added error. This term is a term that gives a constraint based on the correspondence between multi-view infinity regions.
The camera position parameter calculation unit 135 is erroneously calculated from the license plate position and multi-lens infinity region correspondence information calculated for the past predetermined number of times by executing the following processing. Can be excluded. Here, the camera position parameter calculation unit 135 calculates the coordinate value of the area representing the license plate in the reference image based on the relative position of each license plate with respect to the non-reference camera 11B using the calculated camera position parameter. The camera position parameter calculation unit 135 calculates the difference between the calculated coordinate value and the coordinate value of the area representing the license plate represented by the input reference image.
Similarly, the camera position parameter calculation unit 135 inputs the correspondence information indicating the correspondence between the cameras in the coordinates included in the multiview infinity region with respect to the calculated relative angle value between the cameras, and the input multiview infinity region. Find the difference with the correspondence information. The camera position parameter calculation unit 135 is a number that has the largest difference among these calculated differences for each of the predetermined number of times in the past, or that the difference exceeds a predetermined threshold. The coordinate value and correspondence information of the area representing the plate are excluded. Then, the camera position parameter calculation unit 135 calculates the camera position parameter based on the coordinate value of the area representing the license plate that remains without being excluded and the correspondence information. Thereby, the calculation error of the camera position parameter due to the error of the coordinate value of the area representing the license plate and the correspondence information can be reduced.

(Camera position parameter judgment processing)
Next, camera position parameter determination processing according to the present embodiment will be described.
FIG. 6 is a flowchart showing the camera position parameter determination process according to the present embodiment.
(Step S51) The known shape object position detection unit 133 calculates the relative position of the known shape object, that is, the license plate, with respect to each camera based on the input image signal. Here, the known shape object position detection unit 133 detects a region where the license plate is represented in each image. Thereafter, the process proceeds to step S52.
(Step S52) The known shape object position detection unit 133 determines whether or not the license plate relating to the detected region is a license plate different from the license plate used at the time of calibration. The license plate used at the time of calibration is a license plate represented in the image used for calculating the camera position parameter stored in the camera position parameter storage unit 137. Whether the license plate is different from the license plate used at the time of calibration is determined based on whether the numbers displayed on the license plate are the same. The known shape object position detection unit 133 specifies the number displayed on the license plate using the characters recognized by the known shape object detection unit 1331. By using a license plate that is different from the license plate used at the time of calibration, individual differences and deformations of the license plate are taken into account, thereby avoiding a reduction in camera position parameter calculation accuracy. Thereafter, the process proceeds to step S53.

(Step S53) The distance calculation unit 138 calculates the distance to the license plate represented in the reference image and the non-reference image using the camera position parameter before calibration. The camera position parameter before calibration is a camera position parameter stored in the camera position parameter storage unit 137. In calculating the distance, a multi-eye distance measurement method based on the principle of triangulation is used. The distance calculation unit 138 uses an average value obtained by averaging the distances calculated for each pixel included in the area where the license plate is displayed, or a distance calculated for the pixels related to the barycentric point in the area as a representative value of the distance. Adopt as. The representative value of the adopted distance is called distance 1. Thereafter, the process proceeds to step S54.
(Step S54) The distance calculation unit 138 uses the distance to the license plate represented in the reference image and the non-reference image using the camera position parameter newly calculated by the camera position parameter calculation unit 135 in step S53. Calculated using the same method as, and adopts the representative value of distance. The representative value of the adopted distance is called distance 2. Thereafter, the process proceeds to step S55.
(Step S55) The known shape object position detection unit 133 detects the area where the license plate detected from the reference image and the non-reference image is displayed, and the relative position to the license plate related to the area detected from each camera. The distance is calculated based on The known shape object position detection unit 133 calculates an average value obtained by averaging the calculated distances between the cameras. This calculated average value is called distance 3. Thereafter, the process proceeds to step S56.

(Step S56) The camera position parameter determination unit 136 determines which of the distance 1 and the distance 2 is closer to the distance 3. If it is determined that the distance 1 is closer to the distance 3, the process proceeds to step S57. If it is determined that the distance 2 is closer to the distance 3, the process proceeds to step S58.
(Step S57) The camera position parameter determination unit 136 does not adopt the camera position parameter newly calculated by the camera position parameter calculation unit 135, and does not change the content stored in the camera position parameter storage unit 137 as it is. Thereafter, the process ends.
(Step S58) The camera position parameter determination unit 136 stores the camera position parameter newly calculated by the camera position parameter calculation unit 135 in the camera position parameter storage unit 137. Thereafter, the process ends.

  In the present embodiment, the above-described processing may be performed every time the license plate is detected, but is not limited thereto. Steps S51 to S56 may be executed each time a predetermined number of license plates are detected. In this case, in step S56, the camera position parameter determination unit 136 counts the number of times that the distance closer to the distance 3 is the distance 1 and the distance 2, respectively. The camera position parameter determination unit 136 selects the distance with the larger number of times counted. Then, depending on whether the selected distance is distance 1 or distance 2, a step to be executed is selected from step S57 and step S58. Thereby, stability of determination of Step S56 can be improved.

 As a result, the newly calculated camera position parameter is compared with the existing camera position parameter stored in the camera position parameter storage unit 137. The camera position parameter selected by the comparison is stored in the camera position parameter storage unit 137 as a calibration result. Thereby, it is possible to avoid a decrease in the accuracy of calculation of the camera position parameter due to a detection error of the license plate, a calculation error of the correspondence information between the multi-lens infinity regions, a shift of the camera position due to vibration, and the like. That is, the camera position parameter as an appropriate calibration result is stored in the camera position parameter storage unit 137. Using the stored camera position parameter, the distance calculation unit 138 can accurately calculate the distance to the subject represented by the reference image and the non-reference image.

(Multi-eye distance measurement method)
Next, a multi-eye distance measuring method using the principle of triangulation will be described. According to the principle of triangulation, it is possible to measure the direction from each point on both ends of a baseline to the measurement object and determine the position of the measurement object based on the length of the baseline and the measured direction .
FIG. 7 is a conceptual diagram showing a perspective projection camera model used in the multi-eye distance measuring method.
In FIG. 7, the reference camera 11 </ b> A and the non-reference camera 11 </ b> B are indicated by rectangles drawn with broken lines.
Points 605 and 608 in the upper center of FIG. 7 indicate the three-dimensional coordinates 605 and 608 of the respective points on the surface of the measurement object. Points 601 and 602 included in the reference camera 11A and the non-reference camera 11B indicate the focal points (viewpoints) of the reference camera 11A and the non-reference camera 11B, respectively. The line segments going to the lower right and the line going to the lower left included in the reference camera 11A and the non-reference camera 11B are image planes 602 and 603, which are the surfaces of the image sensor, respectively. The relative positional relationship between the focal point 601 and the image plane 602 and the relative positional relationship between the focal point 604 and the image plane 603 are determined by camera position parameters and internal parameters of each camera.

Intersections between the line segment connecting the focal point 601 and the three-dimensional coordinate 605 and the image plane 602, and intersections connecting the focal point 604 and the three-dimensional coordinate 605 are in-image coordinates 606 and 607 which are the coordinates of the image representing the three-dimensional coordinate 605, respectively. Indicates.
The intersection of the line segment 610 connecting the focal point 601 and the three-dimensional coordinate 608 and the image plane 602, and the intersection point of the line segment 611 connecting the focal point 604 and the three-dimensional coordinate 608 and the image plane 603 are images representing the three-dimensional coordinate 608, respectively. In-image coordinates 606 and 609, which are the coordinates of. However, in the example shown in FIG. 7, the line segment connecting the focal point 601 and the three-dimensional coordinate 605 overlaps with the line segment connecting the focal point 601 and the three-dimensional coordinate 608. Therefore, the in-image coordinates 606 that are the intersections of these line segments and the image plane 602 are common between the three-dimensional coordinates 605 and 608.

When the three-dimensional coordinates 605 are on the surface of an object that uniformly diffuses the incoming light in each direction, the gradation values are almost equal between the in-image coordinates 606 and 607. Therefore, the distance calculation unit 138 assumes that the surface of the object exists on a line segment 610 that passes through the in-image coordinates 606, and the gradation value in the in-image coordinates 606 and the corresponding gradation value in the in-image coordinates 607. Search for a three-dimensional coordinate that minimizes the difference. Thereby, it is recognized that the three-dimensional coordinate having the smallest difference is one point on the object surface. The distance calculation unit 138 can repeat the process for each coordinate of the reference image to calculate the distance to the subject surface.
As a method based on this principle, the distance calculation unit 138 may perform block matching between the reference image signal and the non-reference image signal to calculate a parallax value for each pixel. However, the distance calculation unit 138 parallelizes each of the reference image signal and the non-reference image signal based on each camera internal parameter before executing block matching. Parallelization means that the epipolar line of each image is leveled and the distortion of the image caused by the lens is corrected. This makes it possible to limit the area (parallax value candidate) for searching for a block in the block matching in the horizontal direction. The distance calculation unit 138 calculates a distance for each pixel based on the calculated parallax value based on the pixel pitch as a camera internal parameter and the baseline length as a camera position parameter.

In the above description, the case where the license plate is used as the known shape object has been described as an example. However, in the present embodiment, if the object has a fixed size and shape, other objects such as traffic lights, traffic signs, etc. It may be used.
In the above description, the reference camera 11A is the left camera module, and the non-reference camera 11B is the right camera module. In the present embodiment, the reference camera 11A may be a right camera module, and the non-reference camera 11B may be a left camera module.

As described above, according to the present embodiment, the camera to be calibrated by using the calculation of the camera position parameter based on the known shape object and the calculation of the camera position parameter based on the correspondence between the multi-lens infinity regions together. The calculation accuracy of the position parameter can be improved. By using the calculated camera position parameter, it is possible to accurately calculate the distance from the multi-viewpoint image.
Further, in the present embodiment, by acquiring an image used for calibration every time the host vehicle stops, an image captured under more conditions in which the positional relationship between the subject and the camera is fixed. It can be used for calibration. Thereby, highly accurate calibration becomes possible.
Further, in the present embodiment, when it is determined that an object having a known shape does not exist at a multi-lens finite distance, detection corresponding to a multi-lens infinity region is performed. Accordingly, it is possible to reduce the possibility of erroneously calculating the correspondence relationship between the multiview infinity regions, and to prevent erroneous calculation of the camera position parameter.
Furthermore, in this embodiment, the camera position parameter calculation result is verified using a known shape object different from the known shape object used for calibration. As a result, camera position parameters with higher calculation accuracy can be stored, and a decrease in distance calculation accuracy due to erroneous calculation of parameters can be prevented. As described above, the calculation accuracy of the camera position parameter is improved, and a distance measuring device capable of more accurate distance measurement can be realized.

In addition, a part of the distance measurement unit 13 in the above-described embodiment, for example, the vehicle speed monitoring unit 132, the shape known object position detection unit 133, the multi-lens infinity correspondence detection unit 134, the camera position parameter calculation unit 135, the camera position parameter The determination unit 136 and the distance calculation unit 138 may be realized by a computer. In that case, the program for realizing the control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by a computer system and executed. Here, the “computer system” is a computer system built in the distance measuring unit 13 and includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” is a medium that dynamically holds a program for a short time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, In such a case, a volatile memory inside a computer system serving as a server or a client may be included and a program that holds a program for a certain period of time. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.
Further, a part or all of the distance measuring unit 13 in the above-described embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the distance measuring unit 13 may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Further, in the case where an integrated circuit technology that replaces LSI appears due to progress in semiconductor technology, an integrated circuit based on the technology may be used.

  As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the above, and various design changes and the like can be made without departing from the scope of the present invention. It is possible to

DESCRIPTION OF SYMBOLS 1 Distance measuring device 11 Image pick-up part 11A Reference camera 11B Non-reference camera 12 Vehicle speed detection part 13 Distance measurement part 132 Vehicle speed monitoring part 133 Shape known object position detection part 134 Multi-lens infinity correspondence detection part 135 Camera position parameter calculation part 136 Camera position parameter determination unit 137 Camera position parameter storage unit 138 Distance calculation unit 1331 Shape known object detection unit 1332 Shape known object position calculation unit 1333 Shape known object position storage unit 1341 Multi-lens infinity region detection unit 1342 Brightness detection unit 1343 Illumination Control unit 1344 Inter-region correspondence detection unit 1345 Inter-region correspondence storage unit

Claims (11)

Pattern information related to an image of a predetermined object is stored, a known object detection unit that detects a position of a known object that is an object related to the pattern information based on an image represented by an input image signal of a plurality of viewpoints;
Detecting a multi-view infinity region, which is a region in an image representing an object at multi-view infinity where no significant parallax can be detected between the multiple views from each image represented by the input image signals of the plurality of views; A correspondence detection unit for defining correspondence information indicating a correspondence relationship between viewpoints in the eye infinity region;
A position parameter calculation unit that calculates a position parameter indicating a positional relationship between the plurality of viewpoints based on the position of the known object detected by the known object detection unit and the correspondence relationship information determined by the correspondence detection unit;
A distance calculation unit that calculates a distance to an object represented by the input image of the plurality of viewpoints using the position parameter calculated by the position parameter calculation unit;
A distance measuring device comprising: A speed monitoring unit for determining whether or not the plurality of viewpoints are stopped based on the input speed information;
The distance measuring apparatus according to claim 1, wherein when the speed monitoring unit determines that the plurality of viewpoints are stopped, the position parameter calculating unit calculates the position parameter. A position parameter storage unit that stores the position parameter calculated by the position parameter calculation unit;
Of the first distance to the object calculated based on the position parameter calculated by the position parameter calculation unit and the second distance to the object calculated based on the position parameter stored in the position parameter storage unit, When it is determined that the first distance is closer to the third distance to the object calculated by the distance calculation unit, the position parameter calculated by the position parameter calculation unit is stored in the position parameter storage unit A position parameter determination unit;
The distance measuring device according to claim 1, further comprising:   When the known object detection unit does not detect the known object based on the input image signals of the plurality of viewpoints at a distance closer than the multiview infinity, the correspondence detection unit detects the multiview infinity region. The distance measuring device according to any one of claims 1 to 3, wherein   The known object detection unit and the correspondence detection unit respectively determine the positions of the known objects using input image signals of a plurality of viewpoints that are input when the speed monitoring unit determines that the plurality of viewpoints are stopped. 5. The distance measuring device according to claim 2 or claim 3 dependent on claim 2, wherein the correspondence relationship information is determined by detection. An illumination control unit for controlling whether to emit light to the illumination device;
The correspondence detection unit is based on an input image signal at the time of lighting input when the lighting device emits light and an input image signal at the time of lighting input when the lighting device does not emit light. The distance measuring apparatus according to claim 1, wherein the multi-lens infinity region is detected.   The brightness of an object related to an image represented by at least one of the input image signal at the time of lighting and the input image signal at the time of turning off is a signal between the input image signal at the time of lighting and the input image signal at the time of turning off. The distance measuring device according to claim 6, wherein the multi-eye infinity region is detected when the brightness is such that a difference between the values is larger than a predetermined value.   The correspondence detection unit determines that the multiple viewpoints are not stopped in the input image signal of the multiple viewpoints input immediately before the speed monitoring unit determines that the multiple viewpoints are stopped or the speed monitoring unit. Based on the input image signal of the plurality of viewpoints input immediately after the input, and the input image signal of the plurality of viewpoints input when the speed monitoring unit determines that the plurality of viewpoints are stopped, The distance measuring device according to any one of claims 2 and 3 or 4 depending on claim 2, wherein the multi-lens infinity region is detected.   The distance measuring device according to claim 1, wherein the predetermined object is an automobile registration number mark. In the distance measuring method in the distance measuring device,
The distance measuring device detects a position of a known object that is an object related to pattern information related to an image of a predetermined object based on an image represented by an input image signal of a plurality of viewpoints;
The distance measuring device is a multi-view infinity region that is a region in an image representing an object at multi-view infinity where no significant parallax can be detected between the multiple views from each image represented by the input image signals of the multiple views A second process of determining correspondence information indicating the correspondence between the viewpoints of the multi-view infinity region;
The distance measuring device calculates a position parameter indicating a positional relationship between the plurality of viewpoints based on the detected position of the known object and the determined correspondence information;
The distance measuring device calculates a distance to an object represented by the input image of the plurality of viewpoints using the calculated position parameter;
A distance measuring method characterized by comprising: In the distance measurement program in the distance measurement device,
A first procedure for detecting a position of a known object that is an object related to pattern information relating to an image of a predetermined object based on an image represented by an input image signal of a plurality of viewpoints;
Detecting a multi-view infinity region, which is a region in an image representing an object at multi-view infinity where no significant parallax can be detected between the multiple views from each image represented by the input image signals of the plurality of views; A second procedure for defining correspondence information indicating a correspondence between viewpoints in an infinite eye region;
A third procedure for calculating a position parameter indicating a positional relationship between the plurality of viewpoints based on the position of the detected known object and the determined correspondence information;
A fourth procedure for calculating a distance to an object represented by the input image of the plurality of viewpoints using the calculated position parameter;
Distance measurement program to execute.

Images