JP2010218226A - Measurement map generation device and traveling environment confirmation device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain the coordinate values of a world coordinate system in real time from a wide angle image with distortion. <P>SOLUTION: A measurement map generation device includes: distortion flattening processing 30 for converting the image coordinate value [Ud, Vd] of a wide angle image 10 into a distorted plane coordinate value [Xcd, Ycd] by using a coordinate relation expression 20 and each parameter; ideal flattening processing 32 for converting the distorted plane coordinate value [Xcd, Ycd] into an ideal plane coordinate value [Xcm, Ycm]; a world coordinate creation processing 34 for converting the ideal plane coordinate value [Xcm, Ycm] into a world coordinate value [Xwm, Ywm, Zwm]; and a map generation processing 36 for storing the world coordinate value [Xwn, Ywm, Zwm] of the image coordinate value [Ud, Vd] into a storage part 14 as a measurement map 38. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a technical field for confirming a traveling environment by image processing, and more particularly to a technique for extracting an object of a traveling environment that is mounted on a moving body.

The moving body is, for example, a four-wheeled vehicle, a two-wheeled vehicle, an electric vehicle, or the like. The driver usually looks forward in front of the vehicle, except during an operation for parking. When changing lanes or turning left and right, check the back with various mirrors. Further, the moving body has a blind spot from the driver. The front, side, and rear of the moving body are referred to as a traveling environment.
As a system for assisting the driver in confirming the driving environment, there is a system that monitors the front, rear, or side of the host vehicle and confirms the behavior of the object using image processing. For example, the rear and side confirmation devices capture objects (obstacles, other vehicles, approaching vehicles) approaching from the rear, and issue an alarm or the like according to the driving operation of the driver. There is also a method of providing a display unit for presenting information to the driver and displaying a rear image on the display unit.

In Patent Document 1, for the purpose of detecting an obstacle behind the vehicle, a bright spot pattern projected on a flat road surface is imaged and used as a reference image, and the same bright spot pattern projected on a monitoring target is used as a reference image. A method is disclosed in which an obstacle is detected by comparing and performing three-dimensional measurement based on a difference in images (paragraphs 0014 to 0016, FIGS. 3 and 4).
And in this patent document 1, in correspondence with the distortion (distortion) of a wide-angle image when using a wide-angle lens, reference is made to the information of adjacent bright spots projected on the reference flat road surface as the bright spot search direction. A data generation method is disclosed (paragraphs 0023 to 0027, FIGS. 5 and 12).
In Patent Document 2, for the purpose of extracting an optical flow from successive rear images and associating it with the risk level, the flow size corresponding to the total amount of the optical flow and the relative speed with the own vehicle are set as the risk level. An associating technique is disclosed.
In Patent Document 3, for the purpose of detecting the approach of another vehicle to the blind spot area of the own vehicle, a method for estimating the relative positional relationship between the other vehicle and the own vehicle after the disappearance of the optical flow, A method of detecting a flow using (FOE, Focus of Expansion) is disclosed.
In Patent Document 4, for the purpose of performing image measurement with high accuracy, a calibration plate is imaged, an image height obtained by actual measurement and an image height on an imaging screen are obtained, a distortion aberration rate is calculated, and correction is performed. A technique for obtaining a curve is disclosed.
In Patent Document 5, a spatial model is assumed (paragraphs 0108 to 0115 and FIGS. 15 to 18) for the purpose of accurately detecting an approaching object even while traveling on a curve, and the optical flow of the camera image is three-dimensional. A method of converting to a coordinate system (paragraph 0116 to paragraph 0131, FIG. 19) is disclosed. In addition, when converting the two-dimensional coordinates of the camera coordinate system to the three-dimensional coordinates of the world coordinate system, it is possible to determine a unique conversion by adding a condition that the spatial model is at a position higher than the road surface. Is disclosed (paragraph 0120, equation 9).
In Patent Document 6, for the purpose of accurately detecting and warning the approach of the adjacent lane, the distortion of the wide-angle image using the wide-angle lens is corrected, the white line is extracted, and the approaching vehicle is detected by the optical flow. A technique is disclosed.

In this way, the traveling environment confirmation device uses a camera attached to a moving moving object to automatically confirm the surrounding situation of the host vehicle, and performs processing such as optical flow from the captured image. The relative position, speed, and direction of movement of the object are measured.
Further, the world coordinate system of the three-dimensional space of the object and the two-dimensional image coordinate system of the photographed image are made to correspond by calibration work such as camera calibration. In order to make this world coordinate system correspond to the image coordinate system, a linear positional relationship is assumed, and therefore a camera with a small lens aberration is used. In principle, a camera having a small lens aberration has a small area that can be photographed.
By the way, recently, a camera for confirming the back is popular for the purpose of parking assistance. In this rear confirmation camera, a wide-angle lens having a large lens aberration is used so that the rear of the vehicle can be displayed to the driver in a wide range. The wide-angle image obtained by this wide-angle lens is not linear and has a stronger distortion than that handled in Patent Document 1 and Patent Document 6.
A wide-angle image using a wide-angle lens has a non-linear correspondence between the two-dimensional camera coordinate system and the three-dimensional world coordinate system, and is not suitable for automatic measurement as it is.

JP-A-6-325298 JP 11-259634 JP2000-127849 JP 2001-133223 JP2004-56763 JP 2007-26321

As described above, the conventional example has a disadvantage that the method described in each of the cited references cannot be applied to the wide-angle image due to the nonlinearity of the wide-angle image.
In addition, even if a rear-viewing camera that captures a wide-angle image for the purpose of parking assistance or the like is mounted, the rear-confirming device that performs the rear safety confirmation using a technique such as optical flow or white line recognition is used for lens aberration. There was an inconvenience that a small dedicated camera had to be introduced separately.

  For example, in Patent Document 2 described above, the movement of the flow in the three-dimensional world coordinates is predicted by specifying the temporal transition of the luminance value on the two-dimensional image plane. When this method is applied to a wide-angle image, it becomes difficult to establish correspondence between two-dimensional and three-dimensional coordinates as the distance between the vehicle and the host vehicle decreases.

In Patent Document 3, the flow is detected using FOE. However, when this method is applied to a wide-angle image, the flow vector extending from the FOE becomes a curve (non-vector), and the distance between the vehicle and the vehicle decreases. The FOE estimation is not suitable and it becomes impossible to take a response.
In addition, for the processing based on the FOE estimation residual method (processing in which a set of arbitrary flows is defined as one group and FOE is set and determined in the group), etc., the same obstacle due to nonlinear components due to distortion is also used. Will occur.
Since it takes time to select where an arbitrary group having a common FOE is first divided, the number of combinations increases as the number of obstacles behind increases, and the calculation load may increase significantly. Yes, it is difficult to predict the processing time.
Furthermore, in order to execute grouping quickly, first of all, in advance of the grouping process, remove inappropriate flows as much as possible according to the nature (direction / direction) of the flow, and perform processing to reduce the number of combinations in advance. For this purpose, a technique for determining valid / invalid flow at the stage of a single flow is required.
In addition, regarding the effective search range, it is assumed that the search in the direction directly behind the vehicle is disabled and the obstacle detection is limited to the approaching obstacle behind the adjacent diagonal line. In order to confirm the safety of the entire camera field of view, additional introduction of other cameras and supplement / collaboration work are required.

In the above-mentioned patent document 5, the spatial model and its height are set to associate the coordinates between two-dimensional and three-dimensional coordinates. I can't.
In Patent Document 6, since object extraction processing is performed after distortion of a wide-angle image is corrected, firstly, processing time is required to generate a corrected image, and work requiring shorter processing and real-time properties is required. Not suitable for. Second, since the corrected image adds unnaturalness to the shape of the object in the image due to the image processing at the time of distortion removal, the accuracy of the optical flow and the matching operation is lowered.

It is desirable to create reference data for a wide-angle image in advance by projecting the bright spot pattern of Patent Document 1 above, because a corrected image is not generated during actual processing. However, in the projection of a bright spot pattern, the resolution is rough and the rear It becomes difficult to accurately capture a vehicle approaching from the vehicle.
In the distortion aberration correction method of Patent Document 4 described above, distortion aberration can be corrected by a correction equation that is a polynomial. Although the distortion of the wide-angle image is large, the distortion cannot be corrected satisfactorily for the outer portion that is important for the confirmation of the surrounding environment. In addition, since a correction curve is obtained and distortion correction is calculated from the correction curve for each image, it is not suitable for work requiring real-time characteristics, as in Patent Document 6 and the like.

  In particular, in the optical flow process for confirming the driving environment, a white line is extracted to identify the road and the vanishing point of the road is determined. Also when using, like the method of patent document 6, you have to correct | amend to a linear image. When a distorted image is corrected, matching accuracy required for optical flow extraction processing or the like decreases.

[Problem 1] As described above, the conventional example has a disadvantage that a wide-angle image with distortion cannot be used for real-time traveling environment confirmation processing.
[Problem 2] Further, in the above conventional example, there is a disadvantage that the traveling environment cannot be confirmed in real time using the wide-angle image.

  [Object of the Invention] An object of the present invention is to obtain coordinate values of the world coordinate system in real time from a distorted wide-angle image. Furthermore, the present invention is to check a traveling environment in real time using a camera having a wide-angle lens.

  [Focus Point] The inventor of the present invention does not correct the entire wide-angle image, but performs the optical flow matching process or the like with the wide-angle image, and obtains the world coordinate value for the extracted flow. Pay attention. And they came up with the idea that the above problem could be solved by devising the definition of the coordinate system.

[Problem Solving Means 1] A first group of the present invention corresponding to the first embodiment is a storage unit storing external parameters and internal parameters of a camera having a wide-angle lens, and image processing of a wide-angle image input from the camera. And a generation processing unit that generates a predetermined measurement map for use. And the said memory | storage part has memorize | stored the coordinate system definition which defines a some coordinate system, the some coordinate relational expression showing the relationship of each coordinate system, and the relational expression parameter used as the coefficient of each coordinate relational expression. .
Further, the coordinate system defined by the coordinate system definition includes a two-dimensional image coordinate system in units of pixels of the wide-angle image, and a three-dimensional camera coordinate system of the camera as an ideal two-dimensional ideal plane without distortion. A projected ideal plane coordinate system, a distorted plane coordinate system in which the distortion of the wide-angle image by the wide-angle lens is projected onto the ideal plane as a distorted plane, and a three-dimensional world coordinate system of the world to be imaged by the camera It is.
Then, the generation processing unit converts the image coordinate value of the wide-angle image into a distortion plane coordinate value by using the coordinate relational expression and each parameter for each coordinate value of the coordinate system. Processing, an ideal planarization process that converts the distorted plane coordinate value into an ideal plane coordinate value, and a world coordinate conversion process that converts the ideal plane coordinate value into the world coordinate value. A configuration is adopted in which a map generation process for storing the world coordinate value in the storage unit as the measurement map is provided.
Thereby, the said subject 1 was solved.

[Problem Solving Means 2] A second group of the present invention corresponding to the second embodiment includes a camera having a wide-angle lens and photographing a traveling environment of the own vehicle, a storage unit storing a predetermined measurement map, and the camera. A confirmation processing unit configured to confirm the traveling environment of the host vehicle based on the input wide-angle image and the measurement map.
And the said measurement map is a map which memorize | stored beforehand the correspondence of the image coordinate value of an image coordinate system, and the world coordinate value of the said world coordinate system about the range of the measurement frame in the world coordinate system of the said driving environment.
Further, the confirmation processing unit uses the continuous wide-angle image to track the feature point in the wide-angle image within the measurement frame, and the image coordinates of the flow that is the tracked feature point A position reference process for referring to the measurement map from the value and specifying the position of the flow in the world coordinate system.
Thereby, the said subject 2 was solved.

  The present invention interprets the meaning of the terms described in each claim in consideration of the description of the present specification and the drawings, and certifies the invention according to each claim. There are the following advantageous effects in relation to

[Functional Effect 1 of Invention] In the measurement map generating apparatus of the problem solving means 1, the generation processing unit converts the image coordinate value of the wide-angle image into the distortion plane coordinate value as the distortion planarization process, and performs the ideal planarization process. The distorted plane coordinate value is converted into an ideal plane coordinate value. Therefore, it is possible to obtain a distortion plane coordinate value while leaving the distortion of the wide-angle image, and further correct this distortion and convert it into an ideal plane coordinate value. For this reason, the ideal plane coordinate value can be converted into the world coordinate value as the world coordinate processing. Thus, by using the ideal plane and the distorted plane, the image coordinate value of the wide-angle image can be associated with the world coordinate value. And a map production | generation process stores the said world coordinate value of each said image coordinate value in the said memory | storage part as said measurement map.
Therefore, by specifying the pixel position (image coordinate value) of the wide-angle image and using this measurement map, the world coordinate value in the world coordinate system can be specified easily and at high speed. In particular, if this measurement map is generated, three-dimensional measurement can be performed while taking advantage of the wide field of view of the wide-angle image. Furthermore, on the premise of using a measurement map, a wide-angle image can be subjected to image processing without correction, and matching accuracy required for optical flow extraction processing or the like can be improved.

  [Operation Effect 2 of the Invention] In the traveling environment confirmation device of the problem solving means 2, the storage unit has the image coordinate value of the image coordinate system and the world coordinate system within the range of the measurement frame in the world coordinate system of the traveling environment. A measurement map in which the correspondence with the world coordinate value is stored in advance, and the confirmation processing unit uses the continuous wide-angle image as a tracking process, and distorts the image without correcting the range. The feature point in the wide-angle image is tracked (tracking process), and the position of the flow in the world coordinate system is specified by referring to the measurement map from the image coordinate value of the flow that is the tracked feature point. (Location reference processing). For this reason, it is possible to extract the flow in real time without correcting the unstable wide-angle image itself, which is difficult to predict the calculation amount, and by referring to the measurement map, the world in the world coordinate system of the flow A coordinate value (position) can be obtained.

It is a block diagram which shows the structural example of one Embodiment of this invention. (Examples 1 and 2) It is explanatory drawing which shows the example of a definition of a coordinate system. (Examples 1 and 2) It is explanatory drawing which shows the example of a definition of a distortion plane and an ideal plane. (Examples 1 and 2) It is explanatory drawing which shows the example of a relationship between a wide angle lens and distortion. (Examples 1 and 2) 5A shows an example of image coordinate values, FIG. 5B shows an example of a distortion plane, FIG. 5C shows an example of an ideal plane, and FIG. 5D shows the world coordinate system. It is explanatory drawing which shows an example (Example 1-2). 3 is a flowchart illustrating a configuration example of Embodiment 1. (Example 1) 7A is a diagram showing an example of a wide-angle image, FIG. 7B is a diagram showing an example of a corrected image by a radial distortion correction formula, and FIG. 5C is a diagram by a radial distortion correction formula and a logarithmic correction formula. It is explanatory drawing which shows an example of a correction image. (Examples 1 and 2) It is a flowchart which shows an example of the information processing which produces | generates a measurement map. (Example 1) It is a flowchart which shows an example of the information processing which extracts a measurement map with a measurement frame. (Example 1) FIGS. 10A and 10B are explanatory diagrams illustrating an example of a measurement map of the usage range. (Example 1) FIGS. 11A and 11B are explanatory diagrams showing an example of a measurement map with reduced accuracy for explanation. (Example 1) FIGS. 12A to 12C are explanatory diagrams illustrating an example of a measurement frame measurement map when the assumed ground height Yconst is changed. (Example 1) 10 is a block diagram illustrating a configuration example of Example 2. FIG. (Example 2) FIG. 10 is a block diagram illustrating a configuration example of hardware resources according to the second embodiment. (Example 2) It is explanatory drawing which shows an example of the relationship which measures an optical flow. (Example 2) 16A and 16B are explanatory diagrams illustrating an example of a wide-angle image. FIGS. 17A and 17B are explanatory diagrams illustrating an example in which a template for optical flow extraction is registered in a wide-angle image. 18A and 18B are explanatory diagrams showing an example of the extracted flow. 19A and 19B are explanatory diagrams showing an example of a wide-angle image masked with a measurement map in the X direction. 20A and 20B are explanatory diagrams illustrating an example of a wide-angle image masked with a measurement map in the Z direction. 21A to 21E are explanatory diagrams showing an example of the relationship between the extracted flow and the apparent speed. It is a flowchart which shows an example of a back confirmation process. It is explanatory drawing which shows an example of the flow extracted using the speed range. It is explanatory drawing which shows an example which uses several assumption ground height. It is explanatory drawing which shows an example which extracts a flow in multiple layers. It is explanatory drawing which shows an example of the flow extracted by the multi-hierarchy. 10 is a block diagram illustrating a configuration example of Example 3. FIG. (Example 3) It is explanatory drawing which shows the example of a relationship between a wide angle image and an expansion correction image. It is a flowchart which shows an example of a distortion correction image generation process. FIG. 30A shows an example of a simple map with an assumed ground height Yconst = 345, FIG. 30A shows an overlapping portion, FIG. 30B shows an error of the simple Xmap, and FIG. 30C shows a simple Zmap. It is a figure which shows an error. FIG. 31A shows an example of a simple map in which the assumed ground height Yconst = 0, FIG. 31A shows an overlapping portion, FIG. 31B shows an error of the simple Xmap, and FIG. 31C shows the simple Zmap. It is a figure which shows an error. FIGS. 32A to 32E are explanatory diagrams showing an example in which the overlapping portions shown in FIG. 30A are individually displayed according to the overlapping number.

Three embodiments will be disclosed as the best mode for carrying out the invention. Example 1 is a measurement map generation device, Example 2 is a travel environment confirmation device, and Example 3 is a distortion image correction device. Embodiments including Examples 1 to 3 are referred to as embodiments.
The first embodiment relates to a method of generating a measurement map 38 (a two-dimensional table having the same size as the image) that can be used even when the correspondence between the two-dimensional wide-angle image 10 and the world coordinates in the three-dimensional space is nonlinear. In particular, this technique is suitable when the image becomes nonlinear due to image distortion caused by lens aberration.
The second embodiment relates to a technique for using the measurement map 38 and the optical flow extraction process in combination.

<1 Creating a measurement map for wide-angle images>
<1.1 Distorted and ideal planes>
First, Example 1 of this embodiment is disclosed. In the first embodiment, the image coordinate values [Ud, Vd] of the wide-angle image 10 and the world coordinate values [Xw, Yw, Zw] of the world coordinate system XYZ are used in order to make the wide-angle image 10 easy to use in the traveling environment confirmation device. A corresponding measurement map 38 is generated.
The measurement map generation apparatus according to the first embodiment includes a storage unit 14 and a generation processing unit 16 as main elements.
In the example illustrated in FIG. 1, the camera 12 includes a wide-angle lens 28 and captures a wide-angle image 10. The wide-angle image 10 is stored in the image memory 15 of the storage unit 14.
The storage unit 14 includes a coordinate system definition 18, a coordinate relational expression 20, and a parameter group 22. The coordinate system definition 18 defines an image coordinate system UV, an ideal plane coordinate system XcmYcm, a distorted plane coordinate system XcdYcd, and a world coordinate system XYZ. The coordinate relational expression 20 is an arithmetic expression (formula (1), formula (2), formula (3) and formula (4) which will be described later) representing the relationship between coordinate values of each coordinate system.
Further, the storage unit 14 stores, as the parameter group 22, external parameters r and t of the camera 12, internal parameters a and c, and relational expression parameters k and p that are coefficients of the coordinate relational expression 20.
Each parameter is not single, but has a value corresponding to the number of dimensions and the number of terms. When specifying individual parameters, they are indicated with a subscript, and the subscript n is sometimes used when subscripts are bundled.

As shown in FIGS. 2 and 3, the image coordinate system UV is a two-dimensional coordinate system in which the pixel of the wide-angle image 10 is a unit, and specifies the image coordinate value [Ud, Vd]. This image coordinate system UV is determined depending on the resolution (number of pixels) of the camera 12.
The ideal plane coordinate system XcmYcm is a coordinate system obtained by projecting the three-dimensional camera coordinate system xyz of the camera 12 onto an ideal two-dimensional ideal plane 24 (FIG. 5C) without distortion. Specify the value [Xcm, Ycm]. The ideal plane 24 is a projection plane having no lens aberration, and maintains a linear relationship with the coordinate values of the world coordinate system XYZ.
The distorted plane coordinate system XcdYcd is a coordinate system in which the distortion of the wide-angle image 10 by the wide-angle lens 28 is projected onto the ideal plane 24 to form a distorted plane 26 (FIG. 5B). , Ycd]. This distorted plane coordinate system XcdYcd is non-linear.
The world coordinate system XYZ is linear and three-dimensional, and specifies world coordinate values [Xwm, Ywm, Zw] of the world to be imaged by the camera 12.
Here, the subscript w indicates a value in the world coordinate system, the subscript c indicates a value on the image plane, the subscript m indicates a linear image, and the subscript d Indicates an image including distortion caused by a wide-angle lens. That is, the ideal plane coordinate value [Xcm, Ycm] is the ideal state and XwmYwmZw is linearly projected onto the image plane, and the distortion plane coordinate value [Xcd, Ycd] is the theoretical projection of m on the image plane due to lens aberration. The actual projection coordinate values and image coordinate values [Ud, Vd] in which the points are distorted are an image coordinate system in a display device such as a monitor. Therefore, UdVd and XcdYcd are image coordinate values including distortion. UdVd is equivalent to input data coordinates by an input device such as a camera.
The usage of this subscript is the same as in the second and third embodiments. In the third embodiment, a coordinate system whose size is larger than that of the wide-angle image 10 is used, but a subscript “a” is used as a coordinate value of the coordinate system whose size is enlarged.

The generation processing unit 16 generates a measurement map 38 used for image processing of the wide-angle image 10 input from the camera 12, a distortion planarization process 30, an ideal planarization process 32, a world coordinate conversion process 34, And a map generation process 36.
The distortion flattening process 30 uses the coordinate relational expression 20 and the respective parameters for each coordinate value of the coordinate system to convert the image coordinate values [Ud, Vd] of the wide-angle image 10 into distortion plane coordinate values. Convert to [Xcd, Ycd].
Similarly, the ideal planarization process 32 converts the distorted plane coordinate value [Xcd, Ycd] into an ideal plane coordinate value [Xcm, Ycm] using the coordinate relational expression 20 and the parameters.
Similarly, the world coordinate processing 34 converts the ideal plane coordinate value [Xcm, Ycm] into the world coordinate value [Xwm, Ywm, Zwm] using the coordinate relational expression 20 and parameters.
Then, the map generation process 36 stores the world coordinate values [Xwm, Ywm, Zwm] of the image coordinate values [Ud, Vd] as the measurement map 38 in the storage unit 14.

  The measurement map 38 preferably includes an X measurement map 38A (FIG. 10 (A)) and a Z measurement map 38B (FIG. 10 (B)) while assuming the value of the Y axis in the world coordinate system XYZ. It is good to prepare. The X measurement map 38A shows the correspondence between the U axis of the image coordinate system UV and the X axis of the world coordinate system XYZ for each value of the V axis. The Z measurement map 38B indicates the correspondence between the V axis of the image coordinate system UV and the Z axis of the world coordinate system XYZ for each value of the U axis.

Referring to FIG. 2, the world coordinate system XYZ and the camera coordinate system xyz can be associated with each other using a pinhole camera model using a pinhole PN. The parameters used for the correspondence between the two coordinate systems are an external parameter r representing rotation and an external parameter t representing translation.
The image coordinate system UV (pixel coordinate system) is a coordinate system projected two-dimensionally with the camera coordinate system xyz as the center c, and is associated with the ideal plane 24 (image plane). The ideal plane 24 is a plane of the ideal plane coordinate system XcmYcm obtained by linearly projecting the world coordinate system XYZ in two dimensions. The ideal plane coordinate system XcmYcm and the image coordinate system UV are associated by the internal parameters a and c of the camera 12. The internal parameter a is determined by linear lens distortion, and the internal parameter c is determined by the definition of the center position.
Each parameter r, t, a is obtained by calibration of the camera 12. The calibration of the camera 12 includes a known steepest descent method, a gradient method, and the like for the relationship between the known three-dimensional coordinates and the coordinates of the displayed image plane, and parameters can be obtained by these methods.

In the example shown in FIG. 2, the point Mr of the object OJ having the coordinate value [Xwm2, Ywm2, Zwm2] in the world coordinate system XYZ is photographed as a point m on the wide-angle image 10 in the image coordinate system UV. The object OJ is an obstacle, a moving body such as a vehicle, a luminance change of the road surface GR, and a background.
Then, as shown in FIG. 2, since the three-dimensional world coordinate system XYZ is projected onto the two-dimensional image coordinate system UV, the point m in the image coordinate system is not Mr [Xwm2, Ywm2, Zwm2] Mv [Xwm1, Ywm1, Zwm1] is also possible, and it is not possible to determine whether it is Mv or Mr only by the information of the image coordinate system UV.

  Referring to FIG. 3, in the first embodiment, the camera 12 is installed behind the host vehicle MT and the traveling environment behind the host vehicle MT is photographed. In order to install the camera 12 tilted downward, a plane having the normal axis of the optical axis of the camera 12 is defined as an ideal plane 24. An object OJ in the range of the world coordinate system XYZ from the road surface GR to the height of the vehicle MT of the host vehicle MT (upper ground height upper limit 48) is projected onto the ideal plane 24.

  Referring to FIG. 4, the characteristic portion CA of the object OJ in the world coordinate system XYZ becomes a characteristic point CP in the image coordinate system UV. When the lens aberration is small, the angle AN1 formed by the optical axis and the position Mr [Xwm2, Ywm2, Zwm2] of the characteristic portion CA is the same angle on the camera 12 side, and corresponds to the ideal plane coordinate value Xcm of the ideal plane 24. To do. However, when the wide-angle lens 28 having a large lens aberration is used, the correspondence between the target three-dimensional spatial coordinates Mr [Xwm2, Ywm2, Zwm2], the lens focus, and the image coordinate system UV is linearly related due to distortion. In other words, the angle AN2 formed with the optical axis is smaller than the angle AN1 formed on the object side. That is, compared with the ideal plane coordinate value Xcm in the ideal plane 24, it corresponds to the distortion plane coordinate value Xcd of the distortion plane 26 which is distorted inside the image coordinate system UV and the ideal plane 24 is distorted. This distortion includes a pincushion distortion and a barrel distortion.

  Thus, in the camera 12 having the wide-angle lens 28, an image is projected at a position refracted toward the center side due to the light condensing property of the wide-angle lens 28. In this case, since the similarity relationship of the triangle is not established, the distance to the feature portion CA cannot be estimated on the assumption of the similarity relationship. However, since the image can be projected at the intersection of the straight line refracted inward and the surface of the image coordinate system UV using the light condensing property of the wide-angle lens 28, a wider range of images can be taken with the same size imaging device. can do. Therefore, compared with the camera 12 with a small lens aberration, the same imaging range can be imaged with a smaller imaging device.

  Referring to FIG. 5, the feature points CP1 and CP2 of the wide-angle image 10 of the image coordinate system UV shown in FIG. 5 (A) are shown in the distorted plane coordinate system XcdYcd shown in FIG. 5 (B) and in FIG. 5 (C). Via the ideal plane coordinate system XcmYcm, the feature portions CA1 and CA2 of the world coordinate system XYZ shown in FIG.

  Referring to FIG. 6, the generation processing unit 16 first reads the parameter group 22 (step S1). The parameter group 22 includes external parameters r and t determined in advance by calibration of the camera 12, internal parameters a and c, a radial distortion correction parameter k in Expression (2) that is a relational expression parameter, and Expression (3). Logarithmic distortion correction parameter p). Next, an assumed ground height Yconst is set as necessary (step S2).

The generation processing unit 16 further uses the following equation (1) to convert the image coordinate values [Ud, Vd] of the wide-angle image 10 shown in FIG. 5A into the distorted plane coordinates shown in FIG. Conversion into values [Xcd, Ycd] (distortion planarization process 30, step S3). The distortion flattening process 30 is a process of projecting the distorted image coordinate values [Ud, Vd] onto the distortion plane 26 without correcting the distortion caused by the wide-angle lens 28. For example, the feature point CP1 [Ud1, Vd1] shown in FIG. 5 is converted into the distorted plane coordinate value [Xcd1, Ycd1], and the feature point CP2 [Ud2, Vd2] is converted into the distorted plane coordinate value [Xcd2, Ycd2].
The internal parameter a in the following equation (1) is a value determined in advance according to a slight linear distortion of the wide-angle lens 28, and the internal parameter c is a value according to the center position [Cx, Cy] of the image coordinate system UV. It is.

-1: Indicates an inverse matrix
[Ud, Vd]: Image coordinate value in the wide-angle image 10, pixel position (two-dimensional plane) in the image coordinate system UV that captures the object OJ
[Xcd, Ycd]: Distorted plane coordinate value (two-dimensional plane) in the distorted plane coordinate system XcdYcd (image plane with distortion)
a _nn : Linear distortion with internal parameters of camera 12
Cxd, Cyd: Image center of wide-angle image 10

Subsequent to step S3, the generation processing unit 16 uses the following equation (2) or the following equation (3) to obtain the distortion plane coordinate values [Xcd, Ycd] shown in FIG. ) To the ideal plane coordinate value [Xmc, Ymc] (ideal plane processing 32, step S4). For example, the distortion plane coordinate value [Xcd1, Ycd1] is converted to the ideal plane coordinate value [Xcm1, Ycm1] using the equation (3), and the distortion plane coordinate value [Xcd2, Ycd2] is ideal using the equation (2). Convert to plane coordinate value [Xcm2, Ycm2].
In the following equations (2) and (3), Rcm is the radius of the coordinate value [Xcd, Ycd] of the distorted plane coordinate system XcdYcd. The relational expression parameter is the radial distortion correction parameter k in the expression (2), and the logarithmic distortion correction parameter p in the expression (3).

[Xcm, Ycm]: Ideal plane coordinate value (two-dimensional plane) in the ideal plane coordinate system XcmYcm (undistorted image plane)
Rcm: radius of the coordinate value [Xcd, Ycd] of the distorted plane coordinate system XcdYcd
Rck: calibration circle Rck (radius) in the distorted plane coordinate system XcdYcd
k _n : Radial distortion correction parameter
p _n : Logarithmic distortion correction parameter

  When an expression such as “a”, “b”, “c”, or the like is added to a number indicating an expression number, the expression indicating the expression number includes all expressions identified by the alphabet. For example, the expression (2) includes expressions (2a) and (2b).

The storage unit 14 may store the relational expression parameters k and p, or indirectly store the relational expression parameter by storing the value of the denominator on the right side of the expressions (2) and (3). You may make it memorize.
The denominators on the right side of Equation (2) and Equation (3) are prepared as Rcm characteristic equations (Equation (14) and Equation (19) described later) respectively, and solved for Rcm using Laguerre's method, etc. The denominator value of each equation can be specified.

  Rck is a value serving as a threshold value for switching between Expression (2) and Expression (3), and is the radius of the calibration circle Rck predetermined in the distorted plane coordinate system XcdYcd. As shown in FIG. 5, Equation (2) is used for the coordinate values [Xd2, Yd2] inside the calibration circle Rck, and Equation (3) is used for the coordinate values [Xd1, Yd1] outside the calibration circle Rck. Is preferably used (switching process 42).

  Further, instead of the equations (2) and (3), another equation for correcting the distortion may be used, and the coordinates of the ideal plane 24 and the distortion can be obtained by photographing and comparing calibration plates. The coordinates of the plane 26 may be associated with each other. Preferably, the wide-angle image 10 can be widely used by using the calibration circle Rck and the equation (3).

  Subsequent to step S4, the generation processing unit 16 uses the following equation (4) to convert the ideal plane coordinate values [Xcm, Ycm] shown in FIG. 5C into the world coordinates shown in FIG. Conversion into values [Xwm, Ywm, Zwm] (world coordinate processing 34, step S5). For example, ideal plane coordinate values [Xcm1, Ycm1] are converted into world coordinate values [Xwm1, Ywm1, Zwm1]. When the assumed ground height Yconst is determined, the world coordinate values [Xwm1, Yconst, Zwm1] are used.

The external parameter t in the following equation (4) is a value determined in advance according to the translation between the camera coordinate system xyz and the world coordinate system XYZ, and the external parameter r is a value determined in advance according to the rotation.
The parameter s is a proportional constant that associates the length of the ideal plane 24 corresponding to the similarity between the ideal plane 24 and the world coordinate system XYZ with the length of the world coordinate system XYZ.
The assumed ground height Yconst is the height of the object OJ assumed in advance in order to obtain the three-dimensional world coordinate values [Xwm, Ywm, Zwm] from the two-dimensional image coordinate values [Ud, Vd]. That is, in the example shown in the following equation (4), Xwm and Zwm of the world coordinate values are calculated from the ideal plane coordinate values [Xcm, Ycm].

-1: Indicates an inverse matrix
r _nn , t _n : External parameters of camera 12, r is rotation component, t is translation component
[Xwm, Yconst, Zwm]: World coordinate value of the object OJ in the world coordinate system XYZ (three-dimensional space)
Yconst: Assumed ground clearance
s: Parameter

  Subsequent to step S5, the generation processing unit 16 stores the world coordinate values [Xwm, Ywm, Zwm] of the image coordinate values [Ud, Vd] in the storage unit 14 as the measurement map 38 (map generation). Process 36, step S6). This measurement map 38 is an overall measurement map 50 (not shown) in which the corresponding world coordinate values [Xwm, Zwm] are associated with all the pixels of the wide-angle image 10 in the image coordinate system. In the example of defining the measurement frame 44 in the world coordinate system XYZ, it is determined for each image coordinate value [Ud, Vd] whether or not the corresponding world coordinate value [Xwm, Zwm] is within the measurement frame 44 ( Step S7), a measurement frame measurement map 52 is generated.

1.1 Effects of the distortion plane and the ideal plane As described above, by using the ideal plane 24 and the distortion plane 26, the image coordinate values [Ud, Vd] of the wide-angle image 10 are converted into the world coordinate values of the world coordinate system XYZ [ Xwm, Ywm, Zwm]. When the pixel position [Ud, Vd] is specified for the processing result obtained by performing the image processing with the wide-angle image 10, the world coordinate value [Xwm, Ywm, Zwm] in the world coordinate system XYZ is referred to by referring to the measurement map 38. Can be easily identified.
In particular, if the measurement map 38 is generated, the world coordinate values [Xwm, Ywm, Zwm] can be specified for each pixel of the wide-angle image 10, and the wide-angle image 10 can be utilized while taking advantage of the wide field of view. The image 10 can be used for three-dimensional measurement. Furthermore, on the premise that the measurement map 38 is used, the wide-angle image 10 can be subjected to image processing without correction, and the matching accuracy required for the optical flow extraction processing or the like can be improved.

<1.2 Measurement map for each assumed ground height>
The compound eye camera 12 such as a stereo camera can reproduce information in a three-dimensional space from two two-dimensional images of the same object. However, the monocular camera 12 cannot reproduce the information of the three-dimensional space from one two-dimensional image, and cannot specify information about the depth direction, for example.
For this reason, in order to check the driving environment with the monocular camera 12, the object OJ that is the target is grounded on the road surface GR, or the feature part CA at the bumper or tire height of the following vehicle. It may be defined that the focus is narrowed and the height Y from the road surface GR is known.
In this example, the world coordinate processing 34 shown in FIG. 1 presets an assumed ground height Yconst of the characteristic portion CA of the object OJ in the world coordinate system XYZ, and the ideal plane coordinate values [Xcm, Ycm] Is assumed to be provided with a hypothetical ground height processing 40 for converting the value to the world coordinate value [Xwm, Yconst, Zwm].

  Referring to FIG. 2 again, at the point m [Ud, Vd] of the wide-angle image 10 taken by the monocular camera 12, the difference between Mv [Xwm1, Ywm1, Zwm1] and Mr [Xwm2, Ywm2, Zwm2] is determined. Can not. That is, the three values [Xwm, Ywm, Zwm] cannot be obtained from the two values [Ud, Vd]. In this regard, if the Mr height Y is known (eg, Y = ground 0, Y = tire height, etc.), the parameter s in equation (4) can be obtained. Then, the assumed ground height processing 40 can obtain the remaining two values of [Xwm, Zwm] from [Ud, Vd, Yconst].

  Referring to FIG. 3 again, the assumed ground height Yconst can be determined according to the measurement purpose, and may be, for example, the height of Yconst1 or the height of Yconst2.

・ 1.2 Effect of measurement map for each assumed ground height As described above, the assumed ground height Yconst is set, and the value assumed to obtain three values from two values is the height in the world coordinate system XYZ. The characteristic feature CA of the object OJ can be specified in advance by practically reproducing 3D information from 2D information and devising the height of the assumed ground height Yconst. For example, in the process of extracting an optical flow, the efficiency of extracting effective information is increased.

<1.3 Calibration circle and correction formula>
In the example shown in FIG. 5B, the correction formula used between the inside and outside of the calibration circle Rck is switched. This switching process 42 will be described.
In this example, the storage unit 14 corrects a radial distortion correction equation (2) for correcting a radial distortion as the coordinate relational expression 20 between the distortion plane coordinate system XcdYcd and the ideal plane coordinate system XcmYcm, and a logarithmic distortion. Logarithmic distortion correction equation (3). The radial distortion correction equation (2) is the above-described equations (2a) and (2b), and the logarithmic distortion correction equation (3) is the above-described equations (3a) and (3b).
The ideal planarization process 32 shown in FIG. 1 includes a switching process 42. As shown in FIG. 5B, the switching process 42 uses the radial distortion correction formula 2 for the inside of a predetermined calibration circle Rck in the ideal plane coordinate system XcmYcm or the distorted plane coordinate system XcdYcd. To do. On the other hand, the switching process 42 uses the logarithmic distortion correction formula 3 outside the calibration circle Rck. The distorted plane coordinate value [Xcd, Ycd] is converted into the ideal plane coordinate value [Xcm, Ycm] while switching these two formulas between the inside and outside of the calibration circle Rck.

  The calibration circle Rck may be defined by the ideal plane coordinate system XcmYcm or by the distorted plane coordinate system XcdYcd. In the calculation of the switching process 42, it is easy to define it in the distorted plane coordinate system XcdYcd. Further, the calibration circle Rck may not be a strict circle, but may be a shape with rounded polygon corners. This is determined depending on what combination of correction equations is used. As the calculation amount of the ideal planarization process 32, it is preferable that the calibration circle Rck is a simple circle because the calculation amount can be reduced.

FIG. 7A shows a wide-angle image 10 of the image coordinate system UV, FIG. 7B shows an example of a radial distortion corrected image 11A corrected by the equation (2), and FIG. An example of the logarithmic distortion corrected image 11B corrected using 2) and Equation (3) is shown.
For the calibration circle Rck, a position where the error becomes small with respect to the calibration point at the time of calibration of the camera 12 is obtained and a value is set. The radial distortion correction image 11A shown in FIG. 7B is substantially corrected within the range indicated by the dotted line and can be superimposed on the ideal plane 24. However, in the range shown in FIG. 7B, the advantage of wide field of view of the wide-angle image 10 cannot be utilized. On the other hand, in the logarithmic correction image shown in FIG. 7C, the range that can be overlapped with the ideal plane 24 is greatly expanded.

Hereinafter, on the premise that the equations (2) and (3) are switched with the calibration circle Rck, the equation (4) is derived from the above-described equation (1), and the right side of the equations (2) and (3) is further derived. Describe the denominator solution.
First, the correspondence between the two-dimensional image coordinate system UV and the three-dimensional world coordinate system XYZ when the image is taken with the camera 12 having a small lens aberration is expressed by the following equation (5a) via an ideal plane coordinate system XcmYcm without distortion. ) And (6). The parameters r and a are obtained by calibration of the camera 12 as described above.

[Xwm, Ywm, Zwm]: World coordinate value of object OJ in world coordinate system XYZ (three-dimensional space)
[Xcm, Ycm]: Ideal plane coordinate value in the ideal plane coordinate system XcmYcm (two-dimensional plane)
[Ud, Vd]: Image coordinate value (two-dimensional plane) in the image coordinate system UV that captures the object OJ
r _nn , t _n : External parameters of camera 12, r is rotation component, t is translation component
a _nn : Linear distortion with internal parameters of camera 12
Cxd, Cyd: Image center with camera 12 internal parameters
s: Parameter

Referring to FIG. 4 again, when the image is projected at a position refracted toward the center due to the light condensing property of the wide-angle lens 28, the triangular similarity is not established, and the distance to the characteristic portion CA on the assumption of the similarity Cannot be estimated. Therefore, in the normal method, the parameter s in Expression (5) cannot be obtained, and the position (distance) in the world coordinate system XYZ cannot be estimated.
Thus, in the camera 12 having a large lens aberration, distortion (distortion) occurs in the captured image. When R is defined by the following equation (7), the relationship between the generated distortion and the image is generally expressed by the following approximate equation (8) by a technique such as Tsai, and is a nonlinear image having a linear image and distortion. Correlation of images can be obtained. Incidentally, the approximate expression (8) were censored in third order terms of R ^2.
By equation (8), the ideal plane coordinate system XcmYcm and the distorted plane coordinate system XcdYcd can be associated. Further, the distorted plane coordinate system XcdYcd and the image coordinate system UV are expressed by the following equation (9).

[Xcd, Ycd]: Distorted plane coordinate value (two-dimensional plane) in the distorted plane coordinate system XcdYcd (image plane with distortion)
Rcm: Radius of coordinate value [Xcm, Ycm] of ideal plane coordinate system XcmYcm
Rck: calibration circle Rck (radius) in the distorted plane coordinate system XcdYcd
k _n : Radial distortion correction parameter The subscript T indicates transposition of the matrix.

From equation (8), equation (2) can be derived.
In this equation (8), when the field of view of the camera 12 is enlarged and the lens aberration becomes larger than the tolerance, the non-linear element is strong with respect to the approximate correction, and the preceding approximate equation at the pixel position away from the image center [cx, cy]. (8) does not hold.

  An image including distortion when the lens aberration exceeds the tolerance of the approximate expression of Tsai is a radial distortion corrected image 11A shown in FIG. As shown in FIG. 7B, the distortion near the center of the image is corrected, but it becomes difficult to correct the image to the ideal plane 24 as it moves away from the center of the screen and out of the frame. In such a state, it is possible to detect an object OJ (following vehicle) that approaches the rear view only in a limited range close to the center of the screen. Therefore, even with the wide-angle camera 12 introduced for the purpose of confirming a wide range of safety, it is difficult to perform safety confirmation with a single camera in all fields of view.

The applicant deals with Rcm ² <Rck ^{2 with} respect to the distortion approximate expression of Expression (8) in order to correct the nonlinear part regarding the position away from the center of the image with respect to the handling of the image by the camera 12 having a large lens aberration. The following equation (10) is proposed in which the arithmetic expression is divided into a region of Rcm ² ≧ Rck ² and correction is performed. The approximate expression (10) for the outside divided into these areas is based on a logarithmic expression based on actual measurement data for calibration obtained at a position away from the center of the screen.

p _n : Logarithmic distortion correction parameter

From this equation (10), equation (3) can be derived.
Then, the switching process 42 cuts the formula according to the value of Rcm as follows.
Rcm ² <Rck ² formula (2)
Rcm ² ≧ Rck ² formula (3)
As a result of introducing this correction formula and dividing the approximate formula, the wide-angle image 10 having distortion is corrected and converted into a logarithmic distortion corrected image 11B as shown in FIG. 7C. Several other correction methods for images containing distortion have been proposed, but the correction screen obtained in this way approximates a linear image, so it should be handled almost the same as a general linear image space. The two-dimensional image plane and the three-dimensional space can be handled in the entire field of view of the camera 12.

  When Y in Expression (5a) is Yconst, Expression (5b) is obtained. When this is converted, the following equation (4) can be derived. This equation (4) is an equation used by the world coordinate processing 34.

  The subscript -1 indicates an inverse matrix.

Next, for the approximation with Rcm ² <Rck ² , the world coordinate values [Xwm, Ywm, Zwm] of the three-dimensional space recorded in the measurement map 38 are obtained from the image coordinate values [Ud, Vd].
From the equations (7) and (8), the following equations (11) and (12) are established.

Further, the image coordinate value [Ud, Vd] can be obtained based on the distortion plane coordinate value [Xcd, Ycd] from the relationship of the equation (9). Therefore, Xcd and Ycd can be replaced with [Ud, Vd] from the relationship between Expression (7) and Expression (9). That a _nn, c _n by the following equation (13) holds because it is known values already obtained from the calibration.

f ⁻¹ (a _nn , c _n ) is an inverse matrix of f (a _nn , c _n ).

From this equation (13), equation (1) can be derived.
In the equation (13), the same Z component on the right side and the left side can be ignored as a common term. Therefore, a characteristic equation relating to Rcm like the following equation (14) is established with a two-dimensional vector by paying attention to the X and Y components. Since f (a _nn , c _n ) indicates affine transformation, its inverse matrix is given by the following equation (15), and the Z component of equation (13) is 1 on the right side of the left side and can be ignored as a common term. Therefore, the equation (14) focusing on the X and Y components can be used as the characteristic equation for Rcm.

The characteristic equation shown in Equation (14) is usually an equation of five dimensions or more. This equation is solved for Rcm by a method such as Laguerre's solution (or eigenvalue solution of the companion matrix).
The characteristic equation is usually an equation of five dimensions or more. In equation (8), radial distortion (radiation) distortion is modeled by Tsai's approximate equation of distortion. In order for this model to accurately approximate the distortion of the wide-angle lens 28, it is considered desirable to have an order greater than or equal to the square of the radial direction Rcm ² , that is, the fourth or greater of Rcm. Even if the approximate expression is cut off by the square of Rcm, if the calculation according to equation (8) is performed, the degree of the equation for obtaining Rcm is Rcm ² x (1 + k ₁ Rcm ² ) ² and Rcm is 6 To the power.

  In general, it is known that an equation having five or more dimensions can be obtained only by a calculation method called an iterative method. However, in the iterative method, the calculation process is repeated until the solution converges or is terminated a predetermined number of times. Therefore, in order to ensure the accuracy of the solution, it is necessary to ensure a large upper limit of the number of convergence terminations. In image processing that handles moving images, it is almost impossible to execute these processes in real time in units of all pixels.

Then, it is possible to determine the value of With Rcm obtained ^{_{{1 + k 1 Rcm 2 +}} k 2 Rcm 4 + k 3 Rcm 6}, the ideal coordinate values from equation (8) [Xcm, Ycm] and distortion coordinate values The relationship of [Xcd, Ycd] can be expressed as a mathematical expression, and the denominator of the right side of Expression (2) can be specified.
Moreover, the relationship between [Xcm, Ycm] and [Ud, Vd] can be mathematically expressed by replacing [Xcd, Ycd] with [Ud, Vd] from equation (13).
Hereinafter, this mathematical relationship is expressed as Xcm = gx [Ud, Vd] and Ycm = gy [Ud, Vd].

When Ywm = Yconst (arbitrary constant), the position of the remaining three-dimensional space coordinates [Xwm, Zwm] can be specified from the obtained [Xcm, Ycm] by the following formula.
Arbitrary height Yconst is given to Y, and the matrix of the formula (5b) is transformed into the matrix of the following formula (5c). Since r _nn and t _n are known values acquired from the calibration, the actual value of the parameter s can be acquired for the coordinates [Ud, Vd] on the screen.
Therefore, the position of [Xwm, Zwm] relative to [Ud, Vd, Yconst] can be specified using the obtained actual value of s.

  The subscript -1 indicates the inverse matrix

Next, [Xwm, Ywm, Zwm] is obtained from [Ud, Vd] with respect to the approximate expression for Rcm ² ≧ Rck ² .
In this formula, the logarithmic partial term of p ₁ ln Rcm ² + p ₂ is further approximated and replaced by a higher-order characteristic equation of Rcm ² of five dimensions or more as shown in the following formula (17) by a method such as steepest descent method.

n _n is an approximation coefficient when the equation for Rcm in equation (10) is approximated to an equation of five dimensions or more.

  Again, paying attention to the X and Y components, a characteristic equation for Rcm as shown in the following equation (19) is set up by a two-dimensional vector.

Since the value of {1 + p ₁ ln Rcm ² + p ₂ } can be specified by using the obtained Rcm, the relationship between [Xcm, Ycm] and [Xcd, Ycd] can be formulated from the equation (10).
Further, the relationship between [Xcm, Ycm] and [Ud, Vd] can be expressed by replacing [Xcd, Ycd] with [Ud, Vd] from the equation (13).
Therefore, by setting Yconst, the [Xwm, Zwm] coordinates for each pixel [Ud, Vd] on the two-dimensional image plane can be obtained similarly.

As described above, after setting Yconst, [Ud, Vd] is changed according to the screen size, the process is switched with Rcm ² <Rck ² , Rcm ² ≧ Rck ² , and all [Ud, Vd, Yconst] are set to [Xwm, Zwm] can be obtained.
Further, by arranging the acquired [Xwm, Zwm] value on the two-dimensional plane in accordance with the [Ud, Vd] position on the corresponding image, the measurement map 38 is obtained for the entire field of view of the wide-angle camera 12 related to Xwm and Zwm. Can be created.

  Next, a method for generating the measurement map 38 by switching the above-described equation (2) and equation (3) with the calibration circle Rck will be described. In this measurement map generation method, a measurement map 38 is generated using a measurement map generation device. The measurement map generation apparatus includes a storage unit 14 and a generation processing unit 16 illustrated in FIG.

Referring to FIG. 8, first, an assumed ground height Yconst and a calibration circle Rck are input (step S21). Next, using the internal parameters a and c of the camera 12 and the equation (1), the image coordinate value [Ud, Vd] of the wide-angle image 10 is converted into the distorted plane coordinate value [Xcd, Ycd] (distortion). Planarization step, step S22). Then, it is checked whether or not the distortion plane coordinate value [Xcd, Ycd] is within the range of the calibration circle Rck (step S23). Then, in the case of the inside of the predetermined calibration circle Rck in the distortion plane coordinate system XcdYcd, the distortion plane coordinate value [Xcd, Ycd] is calculated as the ideal plane coordinate using the radial distortion correction formula 2. Conversion into values [Xcm, Ycm] (ideal planarization step, step S24). On the other hand, in the case of outside the calibration circle Rck, the distortion plane coordinate value [Xcd, Ycd] is converted into the ideal plane coordinate value [Xcm, Ycm] using the logarithmic distortion correction equation 3 (ideal Planarization step, step S25).
Further, using the external parameters r and t of the camera 12 and the equation (4), the ideal plane coordinate value [Xcm, Ycm] is converted into the world coordinate value [Xwm, Yconst, Zwm] (world coordinate conversion). Process, step S26). Then, the world coordinate value [Xwm, Yconst, Zwm] converted in step S26 is recorded as the world coordinate value corresponding to the image coordinate value [Udi, Vdj] specified in step S22 (step S27).

  Then, the measurement map 38 is generated by repeating steps S22 to S27 until the world coordinate values [Xwm, Ywm, Zwm] of the image coordinate values [Ud, Vd] are generated (map generation step, Steps S28, S29, S30, S30). In the map generation process, the maximum value of the image coordinate system is set to imax and jmax, Vdj is fixed and i is incremented (steps S28 and S29), and j is incremented when imax is reached for this Vdj (step S30). , S31), the conversion to the world coordinate values [Xwm, Yconst, Zwm] is repeated until all the image coordinate values [Udi, Vdj] are obtained until both Udi and Vdj reach the maximum values.

-1.3 Effect of calibration circle and correction formula As described above, by using the logarithmic distortion correction formula (3), it is possible to correct well outside the calibratable range by actual measurement.
In addition, by generating the measurement map 38 in advance, it is not necessary to perform a correction with a high calculation load at the time of image processing for confirming the traveling environment, and the wide-angle image 10 can be used for image processing as it is.
For example, in the conventional example, it is possible to take a two-dimensional image plane and a three-dimensional space for an object OJ at a position aimed at a certain height from an image linearly approximated by distortion removal. Uses a device with a high calculation load such as the steepest descent method. Therefore, for a device that requires processing in a short time and real-time performance, a CPU with very high specifications is required to deal with all pixels on the image plane. It cannot be requested or processed in real time.
When the wide-angle image 10 is corrected, the change between the two consecutive images becomes discontinuous due to the shape correction conversion performed when the linear approximate image is created, and image processing such as matching processing and optical flow processing is performed. There is a risk of hindrance.
Therefore, by introducing a measurement map 38 in which three-dimensional coordinates corresponding to coordinates on the image plane (however, the height is a constant value) is recorded in a two-dimensional table having the same size as the image, the processing speed is increased, Even with a CPU with practical specifications, the position of the object OJ can be specified.

<1.4 Combining with measurement frame>
Referring to FIG. 1 again, in a preferred example, the storage unit 14 stores in advance a measurement frame 44 that divides a measurement range to be subjected to image processing by the maximum value and the minimum value of the world coordinate values XYZ. Yes.
The generation processing unit 16 may include a measurement frame process 46 that extracts a value of an image coordinate value in which the world coordinate value is included in the measurement frame 44 in the measurement map 38.
The measurement frame 44 is a frame for excluding a region that does not need to be measured when confirming the traveling environment from the measurement target. The measurement frame 44 can be defined in the world coordinate system XYZ. For example, the range of the distance to the host vehicle MT may be defined by the Z range of the world coordinate system XYZ, and the horizontal range may be defined by the X value. Further, it is possible to define a region higher than the ground height upper limit 48 shown in FIG.
As shown in FIG. 6, the measurement frame processing 46 may generate the measurement frame measurement map 52 for the inside of the measurement frame 44 after generating the entire measurement map 50 (step S <b> 6).

Referring to FIG. 9, in order to generate the measurement frame measurement map 52 from the overall measurement map 50, first, the measurement frame 44 is input (step S41). The measurement frame 44 is close to the own vehicle MT and has a lower Z minimum value Zmin in the image coordinate system UV, a Z maximum value Zmax far from the own vehicle MT and above the image coordinate system UV, a world coordinate system XYZ, and an image coordinate system. The minimum value Xmin and the maximum value Xmax in the left-right direction of UV.
Subsequently, the coordinate value of the image coordinate system UV is specified (step S42), and the world coordinate value [Xwm, Zwm] is read (step S43). If the world coordinate value [Xwm, Zwm] of the specified image coordinate value [Udi, Vdj] is outside the range of each maximum value and minimum value of the measurement frame 44 (step S44), the world coordinate value [Xwm, Zwm] Is overwritten with the background value (step S45). If it is within the range of the measurement frame 44, it is not overwritten.

Next, it is confirmed whether or not the pixel position i of Udi is the maximum value (step S46). If it is not the maximum value, i is incremented and steps S42 to S45 are repeated for the pixel position of i + 1 (step S47). ). When the pixel position of Udi reaches the maximum value, j is incremented and steps S42 to S46 are repeated until j reaches the maximum value (step S48).
As a result, the world coordinate values [Xwm, Yconst, Zwm] outside the measurement frame 44 are overwritten with the background value, and the measurement frame measurement map 52 is generated.

FIG. 10 is an example of the generated measurement frame measurement map 52, and the assumed ground height Yconst is set to 350 [mm]. FIG. 10A shows an example of an X measurement map 38A with a rear field of view from −5,100 to +5,100 [mm], and FIG. 10B shows an example of a Z measurement map 38B with a rear field of view of 250 to 17,000 [mm].
In the X measurement map 38A, dark indicates far left and bright indicates right far. In the Z measurement map 38B, dark indicates the distance between the vehicles, and light indicates the distance between the vehicles.

For the convenience of display, in order to make the measurement frame measurement map 52 shown in FIG. Indicated.
Moreover, it is preferable to set the setting field of view as the measurement frame 44 in both wing directions (Xmin and Xmax) in three lanes (about ± 5 to 6 m) including both adjacent lanes. For the front and back (Zmin and Zmax), the measurement frame 44 is appropriately set according to the specifications of the camera 12 to be used. For example, an approaching vehicle that is far away becomes too small on the image and it is difficult to determine what it is. Also, if it is too close, even if an alarm is issued, it will not be in time, so it may be excluded from measurement.
By setting the measurement frame 44, useless search during image processing can be avoided. That is, the optical flow process is not performed from the beginning at the invalidation (white = background color) position where X and Z coordinate data is not set in the measurement frame measurement map 52. In other words, efficiency can be improved by not calculating as a position unnecessary for determination.

With respect to the measurement frame measurement map 52 downsized to, for example, 8 bits shown in FIGS. 11A and 11B, one area indicates the following information depending on the resolution. (White = background color position is invalid)
Fig. 11 (A) X measurement map 38A: (5,100x2) / 255 ≒ 39 [mm]
Fig. 11 (B) Z measurement map 38B: (17,000) / 255 ≒ 66 [mm]
In the case of the 8-bit measurement map 38, if the coordinates of the start point of the optical flow to be confirmed are [Ud1, Vd1] and the coordinates of the end point are [Ud2, Vd2], X1 [Ud1, Vd1] obtained from the measurement frame measurement map 52, From Z1 [Ud1, Vd1] and X2 [Ud2, Vd2], Z2 [Ud2, Vd2], the target three-dimensional spatial coordinates W1 [Xwm1, Ywm1, Zwm1], W2 [Xwm2, Ywm2, Zwm2] are as follows: .
W1 [Xwm1, Ywm1, Zwm1] = [39 x X1 [Ud1, Vd1], Yconst, 66 x Z1 [Ud1, Vd1] The unit is mm
W2 [Xwm2, Ywm2, Zwm2] = [39 x X2 [Ud2, Vd2], Yconst, 66 x Z2 [Ud2, Vd2] The unit is mm

If the resolution of the measurement map 38 is increased to 16 bits and 32 bits, the information of the three-dimensional positions X and Z at each pixel position can be acquired in more detail.
16-bit X measurement map 38A: (5100x2) / 65536 ≒ 0.16mm
16-bit Z measurement map 38B: (17000) / 65536 ≒ 0.25mm

  FIG. 12 is a diagram illustrating an example of the measurement frame measurement map 52 that is the Z measurement map 38B. FIGS. 12A to 12C are Z measurement maps 38B when the assumed ground height Yconst is changed. The assumed ground height Yconst1 in FIG. 12 (A) is the highest, and the assumed ground height Yconst3 in FIG. 12 (C) is the lowest. This assumed ground height corresponds to the assumed ground height Yconst shown in FIG.

1.4 Effect of Combining with Measurement Frame As described above, the measurement frame 44 can be defined in the world coordinate system XYZ and reflected in the measurement map 38. Then, when there is the measurement frame 44 for the purpose of checking the driving environment, it is not necessary to access the reference of the measurement frame 44 and the reference of the world coordinate system XYZ twice. Information can be obtained.

<2 Driving environment confirmation device>
<2.1 Optical flow of wide-angle images>
Next, Example 2 is disclosed.
The travel environment confirmation device includes a confirmation processing unit 60, a tracking process 62, and a position reference process 64 as main elements.
Referring to FIG. 13, the traveling environment confirmation device is input from the camera 12 having a wide-angle lens 28 and photographing the traveling environment of the host vehicle MT, the storage unit 14 storing a predetermined measurement map 38, and the camera 12. And a confirmation processing unit 60 for confirming the traveling environment of the host vehicle MT based on the wide-angle image 10 and the measurement map 38.
Then, the measurement map 38 includes the image coordinate value [Ud, Vd] of the image coordinate system UV and the world coordinate value of the world coordinate system XYZ [within the range of the measurement frame 44 in the world coordinate system XYZ of the traveling environment [ Xwm, Ywm, Zwm] is a map storing the correspondence in advance. This measurement map 38 is a measurement frame measurement map 52 to which the measurement frame 44 shown in FIGS. 10 to 12 is attached, and may be referred to as an X measurement map 38A and a Z measurement map 38B, respectively.

  The confirmation processing unit 60 includes a tracking process 62 and a position reference process 64. The tracking process 62 tracks feature points CP in the wide-angle image 10 within the measurement frame 44 using the continuous wide-angle image 10. The position reference process 64 refers to the measurement map 38 from the image coordinate value of the flow FL that is the tracked feature point CP, and specifies the position of the flow FL in the world coordinate system XYZ.

  In the example illustrated in FIG. 13, the storage unit 14 stores an image memory 15 that temporarily stores the wide-angle image 10, a speed range 68 that is used to extract the flow FL, a template 70, and a measurement frame 44. Yes. The confirmation processing unit 60 also extracts an effective flow extraction process 66 for extracting an effective flow FL according to the purpose of confirmation from the extracted flows FL, a grouping process 72 for grouping the flows FL, and an extracted flow FL. And a determination process 74 for determining the presence or absence of an approach or alarm output based on the above.

Referring to FIG. 14, the traveling environment confirmation device uses, as hardware resources, a vehicle speed sensor 82 that detects the speed of the host vehicle MT and a steering angle that detects the steering angle of the host vehicle MT according to the purpose and function of the confirmation. A sensor 84, a winker control unit 86 for detecting presence / absence of a winker operation, and a navigation device 88 for guiding a travel route based on map information and the position of the host vehicle MT are provided. The traveling environment confirmation device may further include a display unit 90 that displays the wide-angle image 10 from the camera 12 and a display control unit 92 that controls display of the wide-angle image 10 displayed on the display unit 90. The display control unit 92 may perform image processing according to switching of the viewpoint of the image to be displayed.
The traveling environment confirmation device includes a controller 80 that functions as the confirmation processing unit 60. The controller 80 has a CPU. In the second embodiment, since a high-load process for sequentially correcting the wide-angle image 10 is not performed, a relatively inexpensive CPU can be used.
Further, the traveling environment confirmation device may include an alarm output unit 94 that outputs an alarm according to the determination result of the determination process 74 by the controller 80. The alarm output unit 94 includes a sound output control unit and a speaker if it is a sound alarm. If it is an alarm by display, the alarm is displayed and controlled in conjunction with the display control unit 92. Further, the alarm output unit 94 operates in conjunction with the winker control unit 86 to issue an alarm when the winker is operated to change the lane and there is a rear vehicle extracted from the flow FL in the lane. You may make it output.

Hereinafter, a flow FL is detected from an image obtained by continuously photographing the rear with the camera 12 having the wide-angle lens 28, and a flow FL that may pose a danger to the traveling of the vehicle MT and the operation of the driver is extracted from the plurality of flows FL. A driving environment confirmation device that outputs a warning to the driver will be described as an example.
In the second embodiment, the traveling environment confirmation device uses the measurement map 38 described in the first embodiment.
For example, the traveling environment confirmation device extracts the flow FL from the wide-angle image 10 to monitor the movement of the object OJ that becomes an obstacle, and the object OJ that is determined to require safety confirmation from the measurement map 38. Referring to the world coordinate values [Xwm, Ywm, Zwm] of the three-dimensional space of OJ, the relative speed and relative position with the vehicle MT are automatically measured. To emit.

As shown in FIG. 15, the assumed ground height Yconst is preferably a height at which a license plate of a general vehicle is provided in the figure.
When the measurement map 38 is created with the assumed ground height Yconst = 0, only the flow FL that can be detected at the boundary between the object OJ and the road surface GR is extracted. At this boundary portion, the feature point CP is likely to be unmatched due to a change in the shape of the shadow (disturbance such as weather, sunshine direction, road surface GR state, etc.), and the detection efficiency of the flow FL is lowered. For this reason, it is desirable to avoid the assumed ground height Yconst = 0. The assumed ground height Yconst is preferably a height (100 to 500 [mm]) that includes elements such as bumpers, license plates, lights, tire wheels, etc. that have high contrast and large feature amounts. If there is a margin in the storage capacity of the storage unit 14, a plurality of assumed ground heights Yconst may be set as shown in FIG. 12, and the X and Z measurement maps 38 may be multilayered.

Further, as shown in FIG. 15, the height higher than the ground height upper limit 48 is preferably excluded from the measurement target.
In the measurement map 38, the straight line connecting the lens focus and the object does not intersect at a position where the straight line connecting the lens focus and the object is at an angle of 0 degrees (parallel) or more with the plane parallel to the assumed ground height Yconst. Therefore, the X and Z coordinates cannot be obtained, and the detection efficiency of the flow FL is low at this height. Therefore, in the second embodiment, the installation height of the camera 12 is set to the ground height upper limit 48, and a portion higher than this is considered to have been taken from the rear aerial direction and is excluded from the measurement frame 44. Thereby, an efficient flow FL can be executed.

  In the example shown in FIG. 15, when the rear image of the vehicle MT is captured by the camera 12 and the vehicle that is the rear object OJ moves from the dotted line position to the solid line position during the time Δt, the flow in the world coordinate system XYZ. The FL becomes the flow FL on the ideal plane 24 and the distortion plane 26 in the pinhole camera model.

  FIG. 16A shows an example of the wide-angle image 10 before movement, and FIG. 16B shows an example of the wide-angle image 10 after movement after Δt. 17 shows a state in which the template 70 is overlaid on the wide-angle image 10 shown in FIG. 16A, FIG. 17A shows an example in which the whole template 70A is overlaid, and FIG. 17B shows the template 70 in the measurement frame. An example is shown in which measurement frame templates 70B having a range of 44 are overlapped. FIG. 18 shows an example in which the extracted flow FL is superimposed on the wide-angle image 10 shown in FIG. FIG. 18A shows the flow FL extracted by the whole template 70A, and FIG. 18B shows the flow FL extracted by the measurement frame template 70B.

  The optical flow process by the tracking process 62 will be described using the entire template 70A as an example. In the optical flow method, first, with respect to the entire screen shot before moving, the wide area image shown in FIG. 16A is divided into blocks by the template 70 using the whole template 70A shown in FIG. Register the 70A block. Next, with respect to the image after movement shown in FIG. 16 (B), a neighboring peripheral image is searched in units of blocks of each template 70 registered before movement, and a matching point / corresponding point is investigated. The feature portion CA of the object OJ in the image is regarded as a feature point CP on the image, and the behavior of the object OJ is estimated.

The optical flow is calculated as follows.
(1) The image before movement to be compared is divided into template 70 areas and registered.
(2) The registration point of the moved image template 70 and the corresponding points of the surrounding image are searched for in the registered template 70 unit. The degree of coincidence at the time of searching is determined by examining a change in luminance value for each pixel by the size of the template 70 at the search position and the luminance value of the template 70. In general, evaluation is performed by sum of absolute differences (SAD) or normalized cross-correlation (NCC).
Further, noise removal processing such as a median filter may be performed in order to improve processing efficiency. Then, a point such as the road surface GR (road) where the luminance distribution clearly does not change or undulate may be passed as a point having a small feature amount.
(3) If the corresponding point is a registered point of the template 70, the object OJ moving into the template 70 is stationary, and if a corresponding point is found around the registered point of the template 70, the object OJ in the template 70 moves. It is thought that.
(4) Furthermore, regarding (3), if the camera 12 (own vehicle MT) is moving, it can be determined that the stationary object OJ follows and the object that moves backward is the background.
In the example shown in FIG. 17, the blocks of the template 70 have a linear lattice shape. However, the blocks may be distorted according to the distorted plane 26, and more finely extracted particularly in the Z direction. .

When the flow FL is detected in the wide-angle image 10, the measurement map 38 is referred to, and the world coordinate values [Xwm, Zwm] before and after the movement are determined from the image coordinate values [Ud, Vd] of the detected start point and end point of the flow FL. Ask for. In this case, since the correspondence between the two-dimensional image plane and the three-dimensional space has already been calculated when the measurement map 38 is created, there is no need to repeat the calculation for each flow FL detected for each process, and there is a large lens aberration, for example. Even for an image including distortion captured by the camera 12 of the wide-angle lens 28, world coordinate values [Xwm, Zwm (and Yconst)] recorded from the image coordinate values [Ud, Vd] to the respective coordinates of the measurement map 38. Can be referred to. As a result, the world coordinate values [Xwm, Yconst, Zwm] corresponding to the image coordinate values [Ud, Vd] of the start point and the end point can be instantly specified.
Further, by recording the shooting time (frame rate) Δt at the time of shooting with the images before and after the movement to be compared, the moving direction, moving speed, and current inter-vehicle distance of the object OJ can be determined from the X and Z coordinates of the detection flow FL. Can be acquired.

  19 and 20 show images in which the wide-angle image 10 from the camera 12 and the measurement frame measurement map 52 are superimposed. If the registration of the template 70 is passed in the background portion of the measurement map 38 shown in FIGS. 19 and 20, the processing can be made efficient without performing a search at a useless position. That is, when the measurement frame 44 of the measurement map 38 is extracted and set as the measurement frame measurement map 52, unnecessary template 70 registration can be passed by invalid data.

  Referring to FIG. 17B again, as shown in FIG. 17B, by registering the measurement frame template 70B only inside the measurement frame 44, as shown in FIG. ), It is possible to extract only the flow FL useful for extracting the rear vehicle from the beginning.

  As described above, the vehicle behind the vehicle can be extracted using the camera 12 having the wide-angle lens 28 used for parking assistance by using the measurement map 38 and the optical flow process together. This is a process in which the tracking process 62 acquires the motion of the object OJ behind the vehicle by continuously processing the input image from the camera 12 and tracking the flow FL. By tracking this change in continuous images (moving images) as a flow FL, the movement state of the object OJ behind the host vehicle MT can be determined based on the relative relationship with the host vehicle MT.

Then, the position reference process 64 obtains the world coordinate values [Xwm, Zwm] of the feature portion CA before and after the movement from the start point and end point [Ud, Vd] of the flow FL of the detected feature point CP. In this case, since the correspondence between the two-dimensional image plane and the three-dimensional space has already been calculated at the time of creating the measurement map 38, it is not necessary to repeat the calculation for each flow FL detected for each process, and according to the large lens aberration. Even for a wide-angle image 10 having distortion, world coordinate values [Xwm, Zwm (and Yconst)] recorded in correspondence with the coordinates [Ud, Vd] of the measurement map 38 from the image coordinate values [Ud, Vd]. You can refer to the value.
Further, by recording the shooting time at the time of shooting with the images before and after the movement to be compared, from the world coordinate values [Xwm, Zwm] of the detection flow FL, the moving direction, moving speed, and own vehicle of the feature O The inter-vehicle distance from the MT can be acquired. For example, by referring to the coordinate value based on the Z measurement map 38B at the start point and end point of the flow FL, the distance between the host vehicle MT and the object OJ can be calculated by (Z end point−Z start point) / Δt The relative speed between the OJ direction and the vehicle MT can be grasped.

As described above, the flow FL generated in the direction in which the inter-vehicle distance is reduced is determined to approach, and the flow FL in which the relative speed is generated in the direction in which the inter-vehicle distance increases (including the background) is different from other invalid flows FL. Similarly, it can be excluded from the determination of approach or the like.
The approach accompanying the lane change of the object OJ using the X measurement map 38A can also be determined by the same method.

2.1 Effect of optical flow of wide-angle image In Example 2, the image coordinate value of the image coordinate system and the world coordinate value of the world coordinate system XYZ in the range of the measurement frame 44 in the world coordinate system XYZ of the driving environment By using the measurement map 38 in which the correspondence is previously stored, the wide-angle image 10 can be directly subjected to image processing without being corrected. In particular, the confirmation processing unit 60 uses the continuous wide-angle image 10 as the tracking process 62, and the feature point CP in the wide-angle image 10 within the measurement frame 44 remains distorted without correction. Is tracked (tracking process 62), and the position of the flow in the world coordinate system XYZ is specified by referring to the measurement map 38 from the image coordinate value of the flow as the tracked feature point CP (position reference process 64). ). For this reason, the flow is extracted in real time without referring to the correction processing of the wide-angle image 10 itself, which is difficult to predict the amount of calculation, and has a high load and is unstable. The world coordinate value (position) in the world coordinate system XYZ can be obtained.

  In this way, the flow FL is extracted, and the world coordinate value [Xwm, Yconst, Zwm] of the flow FL is specified by the measurement map 38, and the two-dimensional image coordinates [Ud, Vd 3D world coordinate values [Xwm, Yconst, Zwm] can be read, and the wide-angle image 10 input from the camera 12 can be directly processed at high speed. Therefore, the traveling environment confirmation device can be configured using the camera 12 having a larger lens aberration, a wider field of view and angle of view, and a relatively low-spec CPU.

Further, since the feature point CP can be extracted from the raw image of the wide-angle image 10 without correcting the distortion by the wide-angle lens 28, the shape correction of the image that occurs during the distortion removal process, etc. Since processing is not required, high matching accuracy can be maintained for optical flow processing using other distortion correction screens.
Furthermore, it is possible to check the driving environment without including a process in which the calculation amount cannot be predicted in advance, such as a process of solving a polynomial having five or more dimensions in order to correct a large distortion. Then, by preparing the measurement map 38 in advance, it is possible to extract a flow in real time without performing a heavy processing such as correction of the wide-angle image 10.
In addition, since the camera 12 having a large lens aberration can be used, a back-eye camera that is generally used for the purpose of a parking guide or the like can be diverted. Therefore, it is not necessary to add a camera 12 dedicated to rearward monitoring and measurement with a small lens aberration and a narrow field of view.

In the conventional method of performing image processing after correcting the wide-angle image 10, the distortion-corrected image becomes larger than the original input image. Become. Also, when returning the image size to the original size, processing time is required for image processing for image size conversion. Then, due to the reduction, the display size of the object OJ reflected near the center of the screen becomes small and recognition becomes difficult. Further, in the creation of a distortion-corrected image, a pixel gap is generated as much as the image size is increased, and thus pixel interpolation processing such as bilinear interpolation is required. Therefore, the processing time for the complementary processing is required. The more time is required for processing, the more the moving amount of the approaching object OJ increases, and the corresponding search range must be increased accordingly. Therefore, further processing time is required.
In this regard, in the method using the measurement map 38 of the second embodiment, the distortion correction image is not created, and the optical flow process can be performed using the input wide-angle image 10 as it is, so that the image creation time is not required. be able to.

<2.2 Flow speed range>
Next, an example in which the effective flow extraction process 66 is performed using the speed range 68 will be described. The flow FL detected by the optical flow is generated in the vicinity of the following vehicle approaching, another vehicle traveling on the opposite oblique line, the rear object OJ accompanying the movement of the host vehicle MT, and the like. The speed range 68 is used to distinguish the flow FL from the flow FL that may be related to the traveling of the host vehicle MT and the operation of the driver, and the unrelated flow FL. That is, for the flow FL generated at a height that deviates from the set assumed ground height Yconst, the calculation results such as speed are correct (within the threshold range) because the calculation is different from the preconditions for the assumed ground height Yconst. (Not moving direction / speed) is used.
That is, in this example, the measurement map 38 presupposes the ground height Yconst of the characteristic portion CA of the object OJ in the world coordinate system XYZ, and calculates the image coordinate value [Ud, Vd] of the image coordinate system UV. It is the map projected on the world coordinate value [Xwm, Yconst, Zwm] of the world coordinate system XYZ.
And the said memory | storage part 14 has memorize | stored the speed range 68 predetermined by the said world coordinate system XYZ about the said assumed ground height Yconst of the said driving environment. The speed range 68 is preferably determined for each of the Z direction (traveling direction) and the X direction (left-right direction).
Further, the confirmation processing unit 60 extracts the flow FL within the speed range 68 from the flow FL whose position is specified as the object OJ in which the characteristic portion CA having the assumed ground height moves. An extraction process 66 is provided.

In this example, by using the correspondence between the flow FL and the measurement map 38 and determining the speed of the flow FL, the flow detected at a height other than the assumed ground height Yconst can be removed.
Referring to FIG. 21, the Z direction will be examined by taking three flows FL shown in FIGS. 21A and 21B as an example. 21A to 21E indicate the actual flow FL speed in the world coordinate system XYZ, and the black flow indicates the apparent speed FLY projected on the assumed ground height Yconst.

  First, for the flow FL to be extracted, the Z coordinate of the world coordinate system XYZ corresponding to the image coordinate values [Ud, Vd] of the flow FL start point / end point is acquired from the Z measurement map 38A. Then, based on the photographing time interval Δt before and after the movement and the photographing time difference, the moving speed (Z end point−Z starting point) / Δt photographing time difference is calculated. Then, the speed of each flow FL can be obtained. A flow FL having a backward movement direction is removed as a background.

This speed is a relative speed with the host vehicle MT. The setting of the upper and lower and lower limits of the speed range 68 of the flow FL is determined by the resolution of the camera 12 to be used, the application for confirming the traveling environment, and the assumed ground height Yconst. A vehicle whose speed in the Z direction is negative is excluded because it is a following vehicle that is moving away or a flow FL generated by the background.
The Z direction is preferably within the length of the speed range 68 shown in FIG.

As shown in FIG. 21C, the Z value of the measurement map 38 relating to the start point and end point of the flow FL acquired at the position corresponding to the position of the flow FLb, that is, the height of the assumed ground height Yconst, becomes the apparent speed FLYb. It is within the speed range 68 and is an appropriate value. Therefore, the threshold at which the flow FLb can be taken out can be treated as the speed of the approaching vehicle.
On the other hand, as shown in FIG. 21D, the flow FL acquired at the position of the flow FLa, that is, the position higher than the height of the assumed ground height Yconst intersects with the Yconst plane at the rear of the actual position. The upper speed FLYa increases and becomes faster than the speed range 68 and is excluded.
Further, as shown in FIG. 21E, the flow FLc acquired at the position of the flow FLc, that is, the position lower than the Yconst height intersects the Yconst plane in front of the actual position, so the measured apparent speed FLYc. Decrease.
Then, by setting a threshold value of the extraction speed at which the correct speed can be extracted, only the flow FL at the position related to Yconst can be extracted.

  Similarly, the same speed calculation is performed in the X direction. When a plurality of assumed ground heights Yconst are set to have a multi-level measurement map 38, the approach target can be three-dimensionally restored by superimposing the extracted flows FL.

  The speed range 68 in the X direction (the left-right direction of the vehicle MT and the direction crossing the lane) is at least 2 lanes per second (3.0 × 2 [m / s] ≒ 22 [km / h]) It is possible to remove the flow FL indicating the object OJ with the erroneous response flow FL. That is, the upper limit of the speed range 68 in the X direction is the speed (6,200 [mm / sec]) that the following vehicle crosses two lanes (3,100 [mm] x 2) in one second when the host vehicle MT moves straight ahead. = 22.32 [km / h] can be set, and the width of the lane is an average value, so if you look at the moving speed of 2 lanes per second, Even if the wobbling movement occurs, the confirmation processing unit 60 does not excessively react excessively.

Regarding the lower limit of the speed range 68 in the Z direction, when the lower limit is 0, when the speed in the Z direction is 0, the length of the flow FL is 0. The reason why the length of the flow FL becomes 0 while the host vehicle MT is running is that the relative speed is 0 and there is a high probability that a parallel running vehicle exists. This can be used when searching for a parallel running vehicle at that position.
Moreover, in the specification that excludes vehicles that run in parallel, if the lower limit of the speed range 68 in the Z direction is set to +5 [km / h], ambiguous information below +5 [km / h] can be removed. That is, the short flow FL can be removed. Increasing the lower limit speed to be set is more resistant to noise and erroneous detection, but approaching vehicles that move at a low speed are also excluded from the target. For this reason, it is advisable to adjust the setting lower limit according to the usage, frequency, and purpose of checking the traveling environment.

The upper limit of the speed range 68 in the Z direction can also be calculated in relation to the effectiveness of the alarm. An example is a travel environment confirmation device that sets a grace period of 1 second from detection of an approaching vehicle and warning time until a collision can occur while left unattended. Furthermore, with this device, the camera 12 whose size on the image of the approaching vehicle is too small for the block size of the template 70 to be used at a position 11 [m] away from the host vehicle MT and cannot be recognized. When a vehicle is used, an approaching vehicle with a relative speed of 40 [km / h] = 11,111 [mm / sec] is detected after a video rate of 1/30 seconds and outputs an alarm (after running over 11.1m) after 1 second. Catch up with the car MT. For this reason, even if an upper limit of 40 [km / h] or more is set, the Z upper limit of the speed range 68 in the Z direction is 40 [km / h] in relative speed.
When the camera 12 is changed to a camera 12 where the image of the approaching vehicle becomes too small at a position 14 m away from the host vehicle MT, an upper limit can be set up to 14 [m / sec] × 3.6 = 50.4 [km / h]. Even if the grace time or the block size of the template 70 is changed, the upper limit of the speed Z changes.

  FIG. 22 is a flowchart illustrating an example of processing for extracting an effective flow using the wide-angle image 10. First, the camera 12 is calibrated, and external parameters and internal parameters of the camera 12 including distortion are measured. Then, a measurement map 38 is prepared.

In actual processing, the wide-angle image 10 is acquired (step S61), and then pre-processing of optical flow processing such as registration of the template 70 and image quality improvement by a median filter is performed (step S62). Then, the feature point CP is tracked and the flow FL is detected (tracking process 62, step S63). For the image coordinate values [Ud, Vd] at the start and end points of the flow, the world coordinate values [Xwm, Yconst, Zwm] are specified with reference to the measurement map 38 (position reference process 64, step S64).
Subsequently, the velocity of the flow FL is calculated, the invalid flow is removed using the velocity range 68, and the valid flow is extracted (valid flow extraction process 66, step S65). Subsequently, erroneous determination processing such as correction of flow error is performed (determination processing 74, step S66). This series of processing is repeated in real time.

The advantages of the second embodiment will be described with reference to FIG.
In the conventional example, after the wide-angle image 10 is acquired, a corrected image in which distortion is corrected is created, a process for complementing the gap in the distortion-corrected image is performed, and in some cases, the image size must be corrected. On the other hand, in Example 2, optical flow preprocessing can be performed without creating a corrected image.

The optical flow process is premised on the assumption that, when comparing images taken in a very short time, the shape change accompanying the movement of the object OJ on the screen is small and can be treated as equivalent to parallel movement. ing. If time is required for the distortion correction conversion process, the shooting of the next image to be compared is delayed, and the time interval with the previous image as a reference is extended.
Therefore, the amount of movement of the object OJ increases, the shape change of the image itself increases, and the matching level between images deteriorates. For example, as the appearance changes depending on the position, objects that are far from the approaching vehicle appear as dots, so it is possible to detect the amount of movement and direction between objects that are close to each other, but the gap is large between objects that are far and those that are close. Thus, it is not possible to take correspondence, and the matching accuracy of the feature point CP is lowered. As a result, the flow FL detection efficiency decreases. In addition, as the amount of movement increases, the range for searching for matching points also expands (eg, if you move from one end of the screen to the other, the entire direction of the vertical (or horizontal) line 1 is the target of search investigation), which further increases the processing time.

Therefore, Example 2 is superior to the conventional method in the following points.
(1) Since all elements related to operations such as distortion removal and two-dimensional / three-dimensional correspondence are recorded in the measurement map 38 in advance, it is not necessary to perform conversion calculation every time, so the processing can be speeded up. A traveling environment confirmation device can be configured with a camera 12 having a wide field of view (large lens aberration) and an affordable CPU (inexpensive and not high in calculation ability).
(2) For comparison with a raw wide-angle image 10 including distortion, it is not necessary to perform shape processing / correction of an artificial image such as distortion removal, and an input image can be used as it is. It is possible to keep the change smaller and maintain high accuracy such as matching.
As described above, the flow FL detection efficiency is superior to the conventional method using the corrected removal image.

  Note that in an apparatus that performs image processing such as a conversion operation from a two-dimensional image plane to a three-dimensional world coordinate, the processing is usually started with the occurrence of a flow FL as a trigger. That is, if the flow FL does not occur, it is determined that the rear view is not changed / abnormal, and no processing is performed. However, in the conventional example, the wide-angle image 10 has to be corrected regardless of whether or not the flow FL has occurred, and the CPU must always be operated. On the other hand, in the second embodiment, when the flow FL is not detected, the measurement map 38 is not referred to and no processing is performed.

  Further, according to the correction approximation calculation like the conventional Tsai approximation, the image created by the distortion correction processing includes a calculation error. In general, when an image corrected for distortion is created from the wide-angle image 10 from the camera 12, the correction-converted value is a discrete value. Therefore, pixels that cannot be linked to a raw image that includes distortion are generated in the distortion-corrected image. As a result, a worm-eaten image with missing pixels at pixel positions that cannot be linked is obtained. In such a case, it is necessary to assign a virtual luminance value to the position of the generated worm-eaten pixel by a complementary method such as a bilinear method to fill in a missing pixel. That is, in the distortion correction of the wide-angle image 10, the amount of light collected by the wide-angle lens 28 needs to be stretched outside during correction, so that “raw image size” <“distortion-corrected image size”. At that time, pixel omission occurs as much as the image is stretched.

For this reason, compared with the comparison between raw image data without correcting distortion, the comparison between image data created by distortion correction processing deteriorates the image quality due to the addition of a calculation error, and the original shape of the feature portion CA is changed. It will be damaged. Therefore, the coincidence level between the comparison images is deteriorated and the detection efficiency of the flow FL is lowered. In addition, the missing pixel position and the complemented virtual luminance value similarly affect the flow FL detection efficiency.
In this respect, a calculation error also occurs in the measurement map 38 in the same manner as the distortion correction processing. However, for the flow FL detection, the calculation error at the image quality level does not occur as much as the raw image is used. There is no effect.

After detecting the flow FL, it is possible to correct the calculation error included in the measurement map 38 by a separate means. However, if a distortion-corrected image that includes a calculation error is used, the flow itself cannot be extracted due to image degradation due to the calculation error. If the flow FL cannot be extracted, it cannot be corrected afterwards. In this way, it is desirable to return to the basics of image processing and deal with a direction in which the image quality is not deteriorated so that the detection efficiency of the flow FL is not lowered as much as possible. In this regard, in the second embodiment, the wide-angle image 10 can be used for the flow FL extraction process as it is.
In the creation of the measurement map 38, the world coordinate values [Xwm, Yconst, Zwm] are assigned to the pixels of the wide-angle image 10 on a one-to-one basis, so the values [Xwm, Zwm] of each measurement map 38 are assigned. ] Is continuous data on each map and is not discretely arranged. Therefore, no omission or worm-feeding occurs, and no supplementary processing is required. That is, since “image size of the raw image” = “size of the measurement map 38”, no omission occurs.

Next, the optical flow process (step S63) is compared.
In the conventional example, there is a possibility that the image quality may be deteriorated due to the influence of a calculation error related to the conversion at the time of creating the distortion-corrected image and an error such as gap compensation when the image is stretched. As a result, the difference between the images before and after the movement to be compared increases, and the flow FL may not be detected.
If the flow FL cannot be detected, the opportunity to search for distance and speed is lost, so that the processing cannot be shifted to post-processing. That is, since the flow FL itself to be corrected is not detected, the flow FL error correction for the position / velocity calculation cannot be performed.

  On the other hand, in the second embodiment, the flow FL in the wide-angle image 10 does not deteriorate the image quality, and the detection efficiency of the flow FL is high because the entire processing can proceed at a higher speed. Even if a calculation error such as a quantization error is included in the value of the measurement map 38 corresponding to the detected flow FL, the error can be corrected by post-processing.

  After extracting the flow FL, in the conventional example, the position of the vanishing point FOE is obtained, and the coordinate values of the world coordinate system XYZ are calculated individually in relation to the vanishing point FOE. Therefore, when the FOE cannot be obtained, the coordinate value of the flow FL of the object OJ cannot be obtained. On the other hand, in the second embodiment, by using the measurement map 38, the world coordinate values [Xwm, Yconst, Zwm] can be obtained directly from the image coordinate values. Therefore, even if the FOE cannot be obtained, such as when the white line of the road surface GR is faint or when the driving lane is curved, in the second embodiment, the flow world coordinate values [Xwm, Yconst, Zwm] Can be calculated.

  Next, in Example 2, the position reference process 64 is performed using the measurement map 38 (step S64). Even if an extremely high-speed CPU is introduced and the position / velocity is obtained by real-time calculation without using the measurement map 38, it is difficult to ensure the same accuracy as the measurement map 38. In other words, it is difficult to obtain the calculated value of the characteristic equation of Expression (14) or Expression (19) by an iterative method in real time, and it is only possible to use an expression that can predict the amount of calculation, and the approximation accuracy deteriorates. .

In the effective flow extraction by the effective flow extraction process 66 (step S65), in the second embodiment, the effective flow and the invalid flow can be identified based on the characteristics of the flow FL itself. There is no need to analyze. That is, a speed range 68 that becomes an effective flow FL is determined according to the assumed ground height Yconst and the like, and in order to determine whether it is valid or invalid by comparing with the speed range 68, invalid flows are excluded by simple processing. be able to.
In fact, as shown in FIG. 23, by extracting the effective flow based on the speed range 68, the invalid flow due to the oncoming vehicle and the shadow can be removed from the flow shown in FIG.
Further, in the analysis by the determination process 74 (step S66), the flow is extracted well, the effective flow is further extracted, and the world coordinate values [Xwm, Yconst, Zwm] of the start point and end point of the flow are acquired. It is possible to determine from a state where the amount of information is large, and it is possible to satisfactorily determine the identification of the vehicle type of the object OJ, the determination of the traveling lane, the degree of approach, and the like.

2.2 Effect of Flow FL Speed Range As described above, by adjusting the assumed ground height Yconst and the speed range 68, the number, bumper, etc. can be set for the feature CA located near a predetermined height. It can be picked up selectively. Then, by applying these characteristics, it is possible to efficiently extract the flow FL indicating the approach of the rear or side object OJ related to the traveling of the host vehicle MT.

<2.3 Flow for each assumed height>
In the second embodiment, the object OJ can be detected three-dimensionally using a plurality of measurement maps 38. In this example, the storage unit 14 stores a plurality of measurement maps 38 for each of a plurality of assumed ground heights, and the confirmation processing unit 60 extracts a flow 52 for each of a plurality of assumed ground heights Yconst.
Referring to FIG. 24, in this example, three assumed ground heights Yconst, Yconst1, Yconst2, and Yconst3, are defined as the assumed ground height Yconst. The measurement map 38 becomes a measurement map 38 for each assumed ground height as shown in FIG. By using these three types of measurement maps 38, three flows FLa, FLb, and FLc can be extracted.

The grouping processing 72 shown in FIG. 12 combines the flows FLa, FLb, and FLc extracted by the confirmation processing unit 60 for each of the plurality of assumed ground heights Yconst into one group GP as shown in FIGS. . The grouping process 72 plots the flow in the three-dimensional world coordinate system XYZ as follows, paying attention to the start point and the end point of the flow FL detected and detected.
[Ud1i, Vd1i] ⇒ [Xwmi, Yconst1, Zwmi]
[Ud2j, Vd2j] ⇒ [Xwmj, Yconst2, Zwmj]
[Ud3k, Vd3k] ⇒ [Xwmk, Yconst3, Zwmk]
i is the total number of flows FL found in the measurement map 38 based on Yconst1
j is the total number of flows FL found in the measurement map 38 based on Yconst2.
k is the total number of flows FL found in the measurement map 38 based on Yconst3

  The determination process 74 determines whether the vehicle is a rear vehicle, the type of the vehicle, and the like using information on the speed, width, and height of the entire group GP.

2.3 Effect of Flow for Each Assumed Ground Height By extracting the flow for each assumed ground height, the type and position of the object OJ can be determined by capturing a plurality of characteristic portions CA of the same object OJ. Specifically, it is possible to make a more accurate determination as to whether or not the vehicle is a moving body, whether it is a four-wheeled or two-wheeled vehicle, and a traveling lane.

・ 2.4 Application example Forward monitoring Although the position information of the approaching vehicle detected by the camera 12 alone can only be obtained from the relative position and relative speed from the own vehicle MT, if various sensors such as the vehicle speed sensor 82 and the steering angle sensor 84 are added, From the relationship of “the amount of movement of the background obtained from the measurement map 38” = “the amount of movement of the host vehicle MT obtained from the sensor”, the difference determination between the “movement speed of the host vehicle MT” and the “flow + measurement map 38” is performed. You can also check the absolute speed of the obstacle.
When these are applied, it is possible to distinguish the background flowing from front to back in the forward direction and the forward vehicle running at a speed slower than the own vehicle MT. That is, in the second embodiment, the determination system can be configured by using the monocular wide-angle camera 12 as the front monitoring camera 12 for applications such as automatic tracking and collision determination.

  In the camera 12 with distortion, since the movement “the movement of the object OJ in the image moves on the straight line in the direction of divergence / contraction from the FOE (infinite point)” does not become a movement, it cannot be handled by the conventional method. Also, the amount of movement in the screen at that time cannot be measured due to the influence of distortion (= cannot be compatible with a vehicle speedometer). In this regard, if the measurement map 38 is used, the relative distance and the relative speed of the object OJ in the distorted wide-angle image 10 can be obtained. Then, if the behavior of the own vehicle MT obtained from the speedometer is subtracted, the background is approached. The vehicle can be distinguished and can be used for forward monitoring.

Application to vehicle periphery monitoring in cornering When the vehicle speed sensor 82, the steering angle sensor 84, and the navigation device 88 are used in combination, the position / movement direction in the absolute system with respect to the reference point for the approaching vehicle and the own vehicle MT in addition to the absolute speed・ The speed can be estimated. By using road information, steering start determination by the steering angle sensor 84, or the like, a trend of approaching vehicles on a curved road / corner ring can be determined by processing such as setting a reference point before the entrance of the curve. In this example, it is not necessary to calculate the amount of FOE movement during cornering. The relative distance / relative speed of the object OJ in the image is obtained by the distorted camera 12, and the background and the approaching vehicle can be distinguished from the combined vector of the moving speed / turning direction of the own vehicle MT with respect to the reference point.

  In the conventional method, for approaching vehicle discrimination / risk level prediction, “vector synthesis on the two-dimensional image plane in the image with FOE as the origin (moving vector for the speed and turning of the vehicle MT and the background“ point vector due to stationary ” ”And movement vectors of surrounding vehicles). However, in the camera 12 with distortion, the concept of vectors based on FOE on a two-dimensional image is not valid due to the influence of refraction due to lens aberration. On the other hand, in the second embodiment, using the measurement map 38 is based on vector synthesis in a three-dimensional space, and even a distorted camera 12 can cope with a discrimination process during cornering.

<3 Distorted image correction device>
Next, a distortion image correction apparatus according to Embodiment 3 is disclosed. As shown in FIG. 27, the distorted image correction apparatus includes a storage unit 14 and a correction processing unit 100 as main components. The storage unit 14 stores external parameters r and t and an internal parameter a of the camera 12 having a wide-angle lens. As in the first and second embodiments, the storage unit 14 includes a coordinate system definition 18 that defines a plurality of coordinate systems, a plurality of coordinate relational expressions 20 that represent the relationships among the coordinate systems, and coefficients of the coordinate relational expressions. The following relational expression parameter group 22 is stored.
In the third embodiment, the coordinate system defined by the coordinate system definition 18 is unique to the third embodiment, in addition to the two-dimensional image coordinate system UV that uses the pixels of the wide-angle image 10 as in the first and second embodiments. It has an enlarged corrected image coordinate system, an enlarged ideal plane coordinate system, a distorted plane coordinate system, and a display coordinate system.

The enlarged corrected image coordinate system handles the enlarged corrected image 102 whose size is enlarged according to the degree of the wide angle. For example, the vertical and horizontal lengths are each twice the size of the wide-angle image 10. When the number of pixels of the wide-angle image 10 is 640 × 480, the number of pixels in the enlarged corrected image coordinate system is set to 1280 × 960. The enlargement according to the degree of wide angle is an enlargement factor that is necessary when the same maximum field of view is imaged linearly without using the wide angle lens 28.
As shown in FIG. 23, if the size of the pixels per unit area is the same, the size of the enlarged corrected image 102 is larger than that of the wide-angle image 10.

  The enlarged ideal plane coordinate system is an image plane similar to the ideal plane XcmYcm of the first and second embodiments, is enlarged to the same size as the enlarged corrected image coordinate system, and the image coordinate system is an ideal two-dimensional image without distortion. This is a coordinate system projected onto an enlarged ideal plane of dimensions. That is, the coordinate system is an enlarged size of the ideal plane XcmYcm. The coordinate values of the enlarged corrected image coordinate system correspond linearly to the coordinate values of the world coordinate system XYZ.

  The distortion plane coordinate system is an image plane corresponding to the distortion plane XcdYcd of the first and second embodiments, and is a coordinate system in which distortion of the wide-angle image by the wide-angle lens 28 is projected onto the enlarged ideal plane to form a distortion plane. is there. The coordinate value of the enlarged ideal plane coordinate system and the coordinate value of the distorted plane coordinate system can correct the distortion caused by the wide-angle lens 28 by a coordinate relational expression (formula (25), formula (27), etc.). The third embodiment has a display coordinate system for displaying with higher resolution than the display unit 90 and the display control unit 92 (FIG. 14) of the second embodiment.

  The correction processing unit 100 corrects distortion of the wide-angle image 10 input from the camera 12. As shown in FIG. 27, the correction processing unit 100 includes an enlarged ideal planarization process 110, a distortion planarization process 112, an enlarged corrected imaging process 114, and a distorted image correction control process 116, as shown in FIG. I have.

Enlarge ideal plane processing 110 converts the coordinate value of the expansion correction image 102 [Uma _a, Vma _a] coordinate values of the enlargement ideal plane [Xcma _a, Ycma _a] to. The enlarged ideal planarization processing 110 mainly corrects the wide-angle image 10 of the camera 12 by an image plane by correcting the internal parameter a of the camera 12 and the center positions Cxm and Cxy (formula (22) described later). Project onto an enlarged ideal plane.

The distortion flattening process 112 converts the coordinate value [Xcma _a , Ycma _a ] of the enlarged ideal plane into the coordinate value [Xcd, Ycd] of the distortion plane. The distortion flattening process 112 mainly corrects distortion caused by the wide-angle lens 28 (formula (25) and formula (27) described later), thereby correcting the distortion of the wide-angle image 10 using the enlarged image plane. .

  The enlarged corrected imaging process 114 converts the coordinate value [Xcd, Ycd] of the distorted plane into the coordinate value [Ud, Vd] of the wide-angle image 10. The enlargement correction imaging process 114 associates the coordinate values [Xcd, Ycd] obtained by correcting the wide-angle distortion with the image plane with the coordinate values [Ud, Vd] of the wide-angle image 10 (formula (28) described later), The coordinate values [Xcd, Ycd] of the enlarged image plane with distortion are associated with the coordinate values [Ud, Vd] of the wide-angle image 10 captured by the camera 12.

  The distorted image correction control processing 116 refers to the value of the coordinate value [Ud, Vd] of the wide-angle image 10 corresponding to the coordinate value [Xcd, Ycd] of the corresponding distorted plane for each pixel of the enlarged corrected image 102. Thus, the generation of the enlarged corrected image 102 in the enlarged corrected image coordinate system is controlled.

The wide-angle image 10 and the enlarged corrected image 102 have different image center positions as shown in FIG. If the number of pixels in the horizontal direction of the wide-angle image 10 is Ud and the number of pixels in the horizontal direction of the enlarged corrected image 102 is Um, the respective pixel centers have a relationship represented by the following equation (20a). The vertical direction is as shown in the following equation (20b).
Then, from the relationship of the following equation (21), the following equations (22a) and (22b) can be derived. Therefore, using the equation (22b), each coordinate value [Uma _a , Vma _a ] of the enlarged corrected image 102 is obtained. Can be converted into coordinate values [Xcma _a , Ycma _a ] of the enlarged ideal plane.

[Cxd, Cyd]: Center position of wide-angle image 10
Ud, Vd: Number of pixels of wide-angle image 10
[Cxm, Cym]: Center position of the enlarged corrected image 102
Um, Vm: Number of pixels of the enlarged corrected image 102
[Xcma _a , Ycma _a ]: Expanded ideal plane coordinate value (two-dimensional plane) in the expanded ideal plane coordinate system (linear image plane)
[Uma _a , Vcma _a ]: Coordinate values of the enlarged corrected image 102 (two-dimensional plane)
a _nn : Linear distortion with the internal parameters of the camera 12 (same values as in the equations (1) and (6a))

As the internal parameter a in the equations (21) to (22), the same values as those in the first and second embodiments can be used. Each pixel [Uma _a , Vma _a ] of the enlarged corrected image 102 is converted to the coordinate value [Xcma _a , Ycma _a ] of the enlarged ideal plane on the image plane by the internal parameter a and the corrected image centers Cxm, Cym. Can be converted. The coordinate values [Xcma _a , Ycma _a ] estimated by the equation (22b) are connected to the world coordinate values [Xw, Yconst, Zw] in the real space by the equations (5b) and (5c). It can be handled as an image. In the case of an image without distortion by the wide-angle lens 28, the values of the k1, k2, k3 terms in the equations (8a) and (8b) and the p1, p2, and p3 terms in the equations (10a) and (10b) are all 0. Therefore, Xcm = Xcd and Ycm = Ycd. However, in the third embodiment, the distortion caused by the wide-angle lens 28 must be taken into consideration as in the first and second embodiments. For this reason, a distorted plane coordinate system is defined and associated with the enlarged ideal plane coordinate system.

  And in Example 3, Preferably, the said memory | storage part 14 is a radial distortion correction formula (25) which correct | amends radial distortion as said coordinate relational expression of the said distortion plane coordinate system and the said expansion ideal plane coordinate system, It is preferable to have a logarithmic distortion correction formula (27) for correcting logarithmic distortion. Then, the distortion flattening process 112 uses the radial distortion correction formula (25) inside a predetermined calibration circle Rck in the enlarged ideal plane coordinate system or the distortion plane coordinate system, and the calibration circle An enlargement switching process 113 for converting the distortion plane coordinate value into the ideal plane coordinate value using the logarithmic distortion correction formula (27) may be provided outside the Rck.

  In the third embodiment, the calibration circle Rck is a circle defined on the enlarged ideal plane, and is compared with the diameter at each coordinate value represented by the following equation (23). In the next equation group, the radial distortion correction equation (25) and the logarithmic distortion correction equation (27) are switched depending on whether or not the coordinate value on the enlarged ideal plane is inside the calibration circle Rcmak.

Rcma _a : radius of the coordinate value [Xcma _a , Ycma _a ] of the enlarged ideal plane coordinate system
Rck: Calibration circle Rck (radius) in the enlarged ideal plane coordinate system
[Xcda _a , Ycda _a ]: Distorted plane coordinate value (two-dimensional plane) in the distorted plane coordinate system (distorted image plane)
k _n : Radial distortion correction parameter for specifying the relationship between the enlarged ideal plane and the distortion plane
p _n : Logarithmic distortion correction parameter for specifying the relationship between the enlarged ideal plane and the distortion plane
[Ud, Vd]: Coordinate value of wide-angle image 10 (same as equation (1), equation (9), etc.)
The subscript T refers to the transpose of the matrix.

After associating the coordinate value of the enlarged corrected image 102 with the coordinate value of the enlarged ideal plane using equation (22b), it is included in the coordinate value of this enlarged ideal plane using equations (24) to (27). Correct the distortion. Here, the formula to be used is switched depending on whether or not the diameter based on the coordinate values [Xcma _a , Ycma _a ] of the enlarged ideal plane is within a predetermined calibration circle Rck. If it is inside the calibration circle Rck, the radial distortion correction formula (25) is used. With the radial distortion correction parameter k and the diameter Rcma, the coordinate value [Xcma _a , Ycma _a ] in the enlarged ideal plane is expanded to the coordinate value [Xcd, Ycd] of the distortion plane by this equation (25). While being reduced, distortion caused by the wide-angle lens 28 is corrected. The logarithmic distortion correction parameter p (27) of the logarithmic distortion correction formula also plays the same role.

  Expression (28) is the same as Expression (9). The coordinate value [Xcd, Ycd] of the distorted plane and the coordinate value [Ud, Vd] of the wide-angle image 10 are represented by the internal parameter a and the center position [Cxd, Cyd. ] To match.

FIG. 29 is a flowchart illustrating an example of the distortion-corrected image generation process.
The enlarged ideal planarization process 110 secures a distortion corrected image buffer in the storage unit 14 (step S71). This distortion-corrected image buffer secures an area with the same size as the enlarged corrected image 102. Then, the calibration circle Rck is input (step S72). Next, the enlarged ideal planarization process 110 specifies the coordinate values [Uma _a i, Vma _a i] of the enlarged corrected image 102 and uses the equation (22b) to express the coordinate values [Xcma _a , Ycma _a ] is specified (step S73). The distortion flattening process 112 calculates the diameter Rcma _a of the coordinate values [Xcma _a , Ycma _a ] of the enlarged ideal plane and compares it with the calibration circle Rck (step S74). If within the calibration circle Rck, the distortion caused by the wide-angle lens 28 is corrected using the radial distortion formula (25) (step S77), and if not within the calibration circle Rck, the distortion is calculated using the logarithmic distortion correction formula (27). Correction is performed (step S77). The coordinate value corrected for this distortion is the coordinate value [Xcd, Ycd] of the distortion plane on the image plane.

The enlarged corrected image processing 114 calculates the coordinate value [Ud, Vd] of the wide-angle image 10 corresponding to the coordinate value [Xcd, Ycd] of the distorted plane by the equation (28). In step S78 from the step S73, it is possible to identify the coordinate value _{[Uma a i, Vma a j} ] of the expansion correction image 102 coordinate values of the wide-angle image 10 corresponding to [Ud, Vd] (steps S79). Then, step S73 to step S79 are repeated for the entire enlarged corrected image 102 (steps S80, S81, S82, S83). In the example shown in FIG. 28, scanning is started from the upper left in the drawing of the right-side enlarged corrected image, and the coordinate values [Ud of the wide-angle image 10 corresponding to the respective coordinate values [Uma _a i, Vma _a i] of the enlarged corrected image. , Vd] is specified, the coordinate value [Ud, Vd] of the wide-angle image is read (for example, the density value if the wide-angle image 10 is grayscale), and the coordinate value [Uma _a i, Vma _a i]. By referring to the coordinate value of the wide-angle image 10, an enlarged corrected image is generated. In addition, in the calculation from the coordinate value of the enlarged corrected image, the outside of the wide-angle image 10 may be referred to. In this case, the reference is invalidated, for example, black.

  The correction processing unit 100 displays the generated enlarged corrected image 102 on, for example, a driver display unit 90. When switching and displaying the wide-angle image 10 and the enlarged corrected image 102, the display of the enlarged corrected image 102 may be controlled as a high-resolution image.

  In the method shown in FIG. 27 to FIG. 29, the reference from the enlarged corrected image 102 to the same coordinate value of the wide angle image 10 occurs from the different coordinate values of the enlarged corrected image 102 when the enlarged corrected image 102 is referred to the wide angle image 10. Due to this overlap, the accuracy of the enlarged corrected image 102 deteriorates.

In the example shown in FIG. 27, a simple map generation process 118 is provided. The simple map generation process 118 generates a simple measurement map that associates a world coordinate value with each pixel of the wide-angle image 10 using an enlarged ideal plane or a distorted plane. The measurement map is a table in which the coordinate values [Ud, Vd] of the wide-angle image 10 are associated with the corresponding world coordinate values [Xw, Yconst, Zw]. Since the world coordinate values [Xw, Yconst, Zw] correspond to the coordinate values [Xcma _a , Ycma _a ] of the enlarged ideal plane, the coordinate values of the wide-angle image 10 and the coordinate values [Xcma _a , Ycma _{a of the} enlarged ideal plane] ], It is possible to generate a simple measurement map.

  In the first and second embodiments, as shown in FIG. 5, the coordinate value [Xcd, Ycd] of the distorted plane is specified from the coordinate value [Ud, Vd] of the wide-angle image 10, and the equation (2) or the equation (3) is obtained. Is used to convert to the coordinate value [Xcm, Ycm] of the ideal plane, and the world coordinate value [Xwm, Ywm, Zwm] is specified using equation (4). Thereby, the world coordinate values [Xwm, Yconst, Zwm] for each coordinate value [Ud, Vd] of the wide-angle image 10 are obtained and used as a measurement map.

In the third embodiment, the coordinate values [Uma _a , Vma _a ] of the enlarged corrected image 102 obtained by the procedure shown in FIG. 29 or the coordinate values [Xcma _a , Ycma _a ] of the enlarged ideal plane are used as the world coordinate values [Xwm, Yconst, Zwm] is obtained and used as a simple measurement map. The enlarged corrected image 102 is linear and has a one-to-one correspondence with the world coordinate values [Xwm, Yconst, Zwm]. However, the enlarged corrected image 102 has coordinate values with overlapping reference objects, and this error becomes an error of the world coordinate value.

  As described above, pixel omission that becomes a data gap or overlap occurs due to the difference in image size between the wide-angle image 10 and the enlarged corrected image 102. Then, according to the procedure shown in FIG. 29 and the like, duplication occurs, but a worm-eaten image cannot be obtained without taking measures against filling in a worm-eaten portion by missing pixels such as bilinear interpolation.

  The coordinate values of the wide-angle image 10 are integer values, while the coordinate values of the enlarged ideal plane and the distortion plane obtained by projecting the enlarged corrected image 102 on the image plane are real values. For this reason, when the coordinate values [Ud, Vd] of the wide-angle image 10 are referred to from the coordinate values [Xcd, Ycd] of the distorted plane, a rounding error from a real number to an integer occurs. This also causes duplication in the enlarged corrected image 102.

  In the first embodiment, the method of generating a measurement map starting from integer coordinate values of the wide-angle image 10 does not cause rounding to an integer, and a highly accurate measurement map can be created.

FIG. 30 is a diagram showing an example of a simple map in which the assumed ground height Yconst = 345 [mm]. FIG. 30 (A) shows an overlapping portion, FIG. 30 (B) shows a simple Xmap error, and FIG. (C) shows a simple Zmap error. FIG. 31 is a diagram showing an example of a simple map in which the assumed ground height Yconst = 0, FIG. 31A shows an overlapping portion, FIG. 31B shows an error of the simple Xmap, and FIG. Indicates the error of the simple Zmap. In FIGS. 30 and 31, the white background is out of the measurement range, and the more overlapping the number, the darker the black.
FIGS. 32A to 32E show an example in which the overlapping portions shown in FIG. 30A are individually displayed according to the overlapping number. The gray background is not subject to measurement. In the example shown in FIG. 32, the number of overlaps of the white portion differs depending on each figure.

  An overlapping portion in FIG. 30A will be described. The white portion shown in FIG. 32A is a highly accurate portion with zero overlap. The white portion shown in FIG. 32 (B) has one overlap, the white portion shown in FIG. 32 (C) has two overlaps, and the white portion shown in FIG. 33 (D) has three overlaps. The white portion shown in FIG. 32E has four or more overlapping times. Thus, the number of times of overlap increases as the distortion increases due to the wide-angle lens.

Compared with FIG. 30A, in the simple measurement maps shown in FIGS. 30B and 30C, the measurement accuracy of the world coordinate system decreases toward the outside where the density decreases. The same applies to the measurement map on the road surface (Yconst = 0) shown in FIG.
However, according to this method, it is possible to easily create a measurement map for a relatively short distance with few overlapping reference points by the conventional calculation method such as the distortion correction device creation means as compared with the means of the first embodiment. is there. For example, a parking assistance camera is fixed to the vehicle in such a direction that the lower foot of the host vehicle is located at the center of the screen. For the system under such a configuration, the left and right white lines of the foot of the traveling vehicle are obtained by image processing such as binarization, and the detection point coordinates on the image plane are obtained from the simple measurement map at the position of Yconst = 0. By converting to spatial coordinates, it is possible to confirm whether or not the vehicle is traveling near the center of the travel lane, and can be applied to the development of a departure lane warning device. In this case, by limiting the detection area to a range where there are few overlapping points of the simple measurement map, obstacles such as other white lines, backgrounds, and subsequent vehicles that are not detected can be simultaneously excluded from the image. Therefore, it is possible to avoid the influence of these other obstacles related to the detection work of the left and right white lines at the foot, and the detection effect can be improved at the same time.
Therefore, it is efficient by adopting the measurement map creation method according to Example 1 when detecting a wide range of objects in real space, and when detecting a relatively short distance object according to Example 3. Device development can be performed.

DESCRIPTION OF SYMBOLS 10 Wide-angle image 12 Camera 14 Memory | storage part 16 Generation | occurrence | production process part 28 Wide-angle lens 30 Distortion planarization process 32 Ideal planarization process 34 World coordinate process 36 Map generation process 38 Measurement map 38A X measurement map 38B Z measurement map 40 Assumption ground height process 42 Switching process 44 Measurement frame 46 Measurement frame process 48 Ground height upper limit 50 Overall measurement map 52 Measurement frame measurement map 60 Confirmation processing unit 62 Tracking process 64 Position reference process 66 Effective flow extraction process 68 Speed range 70 Template 72 Grouping process 74 Determination processing

Claims (10)

A storage unit that stores external parameters and internal parameters of a camera having a wide-angle lens, and a generation processing unit that generates a predetermined measurement map for use in image processing of a wide-angle image input from the camera,
The storage unit stores a coordinate system definition that defines a plurality of coordinate systems, a plurality of coordinate relational expressions that represent the relationship of each coordinate system, and a relational expression parameter that is a coefficient of each coordinate relational expression,
The coordinate system defined by the coordinate system defines a two-dimensional image coordinate system in units of pixels of the wide-angle image and a three-dimensional camera coordinate system of the camera projected on an ideal two-dimensional ideal plane without distortion. An ideal plane coordinate system, a distorted plane coordinate system in which distortion of the wide-angle image by the wide-angle lens is projected onto the ideal plane to form a distorted plane, and a world three-dimensional world coordinate system to be imaged by the camera Yes,
The generation processing unit
For each coordinate value of the coordinate system, using the coordinate relational expression and the respective parameters, a distortion planarization process for converting the image coordinate value of the wide-angle image into a distortion plane coordinate value, and the distortion plane coordinate value An ideal planarization process for converting the ideal plane coordinate value into a world coordinate value, and an ideal plane conversion process for converting the ideal plane coordinate value into a world coordinate value,
A map generation process for storing the world coordinate value of each image coordinate value as the measurement map in the storage unit;
A measurement map generation device characterized by that. The world coordinate processing includes presumed ground height processing for presetting an assumed ground height of a characteristic part of an object in the world coordinate system and converting the ideal plane coordinate value to the world coordinate value.
The measurement map generating apparatus according to claim 1, wherein: The storage unit has, as the coordinate relational expression between the distortion plane coordinate system and the ideal plane coordinate system, a radial distortion correction expression for correcting radial distortion, and a logarithmic distortion correction expression for correcting logarithmic distortion,
The ideal planarization processing uses the radial distortion correction formula inside a predetermined calibration circle in the ideal plane coordinate system or the distorted plane coordinate system, and the logarithmic distortion correction outside the calibration circle. A switching process for converting the distorted plane coordinate value into the ideal plane coordinate value using an equation;
The measurement map generating apparatus according to claim 1 or 2, wherein The storage unit stores in advance a measurement frame that divides a measurement range to be subjected to image processing by the maximum value and the minimum value of the world coordinate values,
The generation processing unit includes a measurement frame process for extracting a value of an image coordinate value in which the world coordinate value is included in the measurement frame in the measurement map.
The measurement map generation device according to claim 1, 2, or 3. A storage unit that stores external parameters and internal parameters of a camera having a wide-angle lens, and a generation processing unit that generates a predetermined measurement map for use in image processing of a wide-angle image input from the camera,
The storage unit stores a coordinate system definition that defines a plurality of coordinate systems, a plurality of coordinate relational expressions that represent the relationship of each coordinate system, and a relational expression parameter that is a coefficient of each coordinate relational expression,
As the coordinate system, a two-dimensional image coordinate system in units of pixels of the wide-angle image, and an ideal plane coordinate system obtained by projecting the three-dimensional camera coordinate system of the camera onto an ideal two-dimensional ideal plane without distortion And a distortion plane coordinate system in which distortion of the wide-angle image by the wide-angle lens is projected onto the ideal plane to form a distortion plane, and a three-dimensional world coordinate system of the world to be imaged by the camera,
Measurement is performed using a measurement map generation device having a radial distortion correction expression for correcting radial distortion and a logarithmic distortion correction expression for correcting logarithmic distortion as the coordinate relational expression between the distortion plane coordinate system and the ideal plane coordinate system. A method of generating a map,
For each coordinate value of the coordinate system, using the coordinate relational expression and the respective parameters, a distortion planarization step of converting the image coordinate value of the wide-angle image into a distortion plane coordinate value;
The radial distortion correction formula is used for the inside of a predetermined calibration circle in the distortion plane coordinate system, and the distortion plane coordinate value is calculated using the logarithmic distortion correction formula for the outside of the calibration circle. An ideal planarization process for converting to an ideal plane coordinate value;
A world coordinate conversion step of converting the ideal plane coordinate value into the world coordinate value;
A map generation step of generating the measurement map by repeating each step until the world coordinate value of each image coordinate value is generated,
A measurement map generation method characterized by that. A camera that has a wide-angle lens and shoots the driving environment of the vehicle;
A storage unit that stores a predetermined measurement map;
A confirmation processing unit that confirms the traveling environment of the host vehicle based on the wide-angle image input from the camera and the measurement map;
The measurement map is a map that stores in advance the correspondence between the image coordinate value of the image coordinate system and the world coordinate value of the world coordinate system for a range of the measurement frame in the world coordinate system of the driving environment,
The confirmation processing unit
A tracking process for tracking feature points in the wide-angle image within the measurement frame using the continuous wide-angle image;
A position reference process that refers to the measurement map from the image coordinate value of the flow that is the tracked feature point and identifies the position of the flow in the world coordinate system,
A traveling environment confirmation device characterized by that. The measurement map is a map obtained by projecting the image coordinate value of the image coordinate system onto the world coordinate value of the world coordinate system, assuming the ground height of the characteristic part of the object in the world coordinate system in advance,
The storage unit stores a speed range predetermined in the world coordinate system for the assumed ground height of the driving environment;
The confirmation processing unit includes an effective flow extraction process for extracting a flow within the speed range among the flows that have identified the position as an object in which the characteristic portion having a height of the assumed ground height moves.
The traveling environment confirmation device according to claim 6. The storage unit stores a measurement map for each of a plurality of assumed ground heights,
The confirmation processing unit extracts a flow for each of a plurality of assumed ground heights.
The traveling environment confirmation device according to claim 7. A camera having a wide-angle lens, a storage unit storing external parameters and internal parameters of the camera, a correction processing unit that corrects distortion of a wide-angle image input from the camera, and a corrected image corrected by the correction processing unit And a display unit for displaying
The storage unit stores a coordinate system definition that defines a plurality of coordinate systems, a plurality of coordinate relational expressions that represent the relationship of each coordinate system, and a relational expression parameter that is a coefficient of each coordinate relational expression,
The coordinate system defined by the coordinate system definition is a two-dimensional image coordinate system in units of pixels of a wide-angle image captured by the camera, and an enlarged corrected image coordinate that handles an enlarged corrected image whose size is enlarged according to the degree of the wide angle. System, an enlarged ideal plane coordinate system in which the image coordinate system is projected onto an ideal two-dimensional enlarged ideal plane having no distortion and the same size as the enlarged corrected image coordinate system, and distortion of the wide-angle image by the wide-angle lens. A distortion plane coordinate system that is projected onto the enlarged ideal plane to be a distortion plane, and a display coordinate system of the display unit that is enlarged in size than the wide-angle image,
The correction processing unit
For each coordinate value of the coordinate system, using the coordinate relational expression and each parameter, an enlarged ideal planarization process that converts the coordinate value of the enlarged corrected image into a coordinate value of the enlarged ideal plane; A distortion flattening process for converting the coordinate value of the enlarged ideal plane into the coordinate value of the distortion plane; and an enlarged correction imaging process for converting the coordinate value of the distortion plane into the coordinate value of the enlarged correction image;
The generation of the enlarged corrected image in the enlarged corrected image coordinate system is controlled by referring to the value of the coordinate value of the wide-angle image corresponding to the coordinate value of the corresponding distortion plane for each pixel of the enlarged corrected image. With distortion image correction control processing
A distorted image correction apparatus characterized by the above. The storage unit has a radial distortion correction formula for correcting radial distortion and a logarithmic distortion correction formula for correcting logarithmic distortion, as the coordinate relational expression of the distortion plane coordinate system and the enlarged ideal plane coordinate system,
The distortion planarization process uses the radial distortion correction formula inside a predetermined calibration circle in the enlarged ideal plane coordinate system or the distortion plane coordinate system, and the logarithmic distortion outside the calibration circle. Using a correction formula, comprising an enlargement switching process for converting the distorted plane coordinate value to the ideal plane coordinate value,
The distorted image correction apparatus according to claim 9.