JP2021081368A - Distance image generator, model generator, inter-vehicle distance calculation device, distance image generation method, and program therefor - Google Patents

Abstract

To provide a distance image generator that generates a distance image from the photographed image of a camera.SOLUTION: A distance image generator 1 comprises: an input unit 10 for inputting moving-image data photographed by a camera mounted to a vehicle and the position or speed data of the vehicle at the time the moving-image is photographed; a beam calculation unit 11 for calculating a beam of light extending from the optical center of the camera to the image pixels of a prescribed frame of the moving-image; a motion parameter calculation unit 12 for finding the motion parameter of the camera on the basis of the moving-image data and the position or speed data of the vehicle; a road surface restoration unit 13 for finding on the basis of the motion parameter the shape of a road surface on which the vehicle has traveled; a distance calculation unit 14 for calculating the distance to the pixels corresponding to the beam of light in the image of the prescribed frame, on the basis of an intersection of the beam of light and the road surface; and a distance image generation unit 15 for generating, on the basis of the distance calculated by the distance calculation unit 14, a distance image that indicates the distance to the road surface in the image of the prescribed frame.SELECTED DRAWING: Figure 2

Description

The present invention relates to a distance image generation device, a model generation device, an inter-vehicle distance calculation device, and the like.

As a sensor used in a vehicle or the like, a sensor that images distance information is known. One such sensor is LIDAR (light detection and ranging). Further, since the stereo camera can also calculate the distance to the object by using the parallax of the two cameras, it is possible to generate a distance image. By imaging the distance information, for example, there is an advantage that the distance between the vehicle and the preceding vehicle can be measured, and an object can be detected even at night when the visibility is poor. On the other hand, LIDAR and stereo cameras have an industrial disadvantage of high cost.

Non-Patent Document 1 proposes unsupervised depth estimation with a monocular camera, but when training a model for estimation, access the left and right color images from a calibrated stereo camera. (Non-Patent Document 1, page 272, right column), a stereo camera is still required.

Godard, Clement, Oisin Mac Aodha, and Gabriel J. Brostow. "Unsupervised monocular depth estimation with left-right consistency." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.

If a distance image can be generated from an image taken by a monocular camera, a lidar or a stereo camera can be omitted. As one method for this, it is conceivable to use an inference model for estimating a distance image from a monocular camera image, but in order to generate such a model, an image taken by the camera and a corresponding distance image can be used. A large amount of teacher data is required as a set.

Therefore, in view of the above background, an object of the present invention is to provide a distance image generation device or the like that generates a distance image from an image captured by a camera.

The distance image generation device of the present invention has an input unit for inputting moving image data taken by a camera mounted on a vehicle and data on the position or speed of the vehicle when the moving image is taken, and the moving image. A light ray calculation unit that calculates a light beam extending from the optical center of the camera with respect to a pixel of an image of a predetermined frame, and a motion parameter that obtains a motion parameter of the camera based on the moving image data and the position or speed data of the vehicle. Based on the calculation unit, the road surface restoration unit that obtains the shape of the road surface on which the vehicle travels based on the motion parameters, and the intersection of the light beam and the road surface, up to the pixel corresponding to the light beam in the image of the predetermined frame. It includes a distance calculation unit that calculates a distance, and a distance image generation unit that generates a distance image indicating a distance to a road surface in an image of the predetermined frame based on the distance calculated by the distance calculation unit. In this way, the distance to each pixel of the road surface can be obtained from the information of the intersection of the light ray to each pixel of the image of the predetermined frame and the road surface restored by traveling. This makes it possible to generate a distance image showing the distance to the road surface based on the moving image taken by the camera and the position or speed of the vehicle.

The distance image generator restores the scale of the error included in the distance obtained by the distance calculation unit, (i) the pitch angle of the camera when the image of the predetermined frame is taken, and the road surface corresponding to the pixel. An error model storage unit that stores an error model in which the displacement of the camera with respect to the pitch angle and (ii) the length of the predetermined frame from the optical center of the camera to the pixel in the image are expressed as variables. And an error scale calculation unit that calculates the scale of the error with respect to the distance obtained by the distance calculation unit with reference to the error model. With this configuration, in addition to the distance to each pixel, the scale of the error included in the obtained distance can be obtained.

The model generation device of the present invention infers a distance image from a captured image using a set of a distance image generated by the distance image generation device and an image of a predetermined frame that is the source of the distance image as training data. It has a configuration to generate a model. With this configuration, it is possible to generate a model that generates a distance image from an image taken by a camera using a distance image generated from a moving image as teacher data.

The model generation device of the present invention uses a set of a distance image generated by the distance image generation device, an image of a predetermined frame that is the source of the distance image, and a scale of the error as teacher data, and captures an image. It is a model generation device that learns a model for generating a distance image from an image and generates a model, and has a configuration in which the larger the scale of the error, the smaller the influence on the loss function is to perform the learning. With this configuration, the influence of the teacher data having a large error on the model generation is reduced, so that a model capable of highly accurate estimation can be generated.

The inter-vehicle distance calculation device of the present invention has a storage unit that stores a model generated by the model generation device, an input unit that inputs an image taken by a camera mounted on the vehicle, and the input unit. A vehicle detection unit that detects a preceding vehicle from the input image, and when the vehicle detection unit detects a preceding vehicle, the model is read from the storage unit, and the image is applied to the model to obtain a distance image. It includes a distance image generation unit to be generated, and a distance determination unit that superimposes the preceding vehicle on the distance image and determines the distance to the pixel at the lower end of the preceding vehicle as the inter-vehicle distance. With this configuration, the inter-vehicle distance to the preceding vehicle can be calculated using the image taken by the monocular camera.

The distance image generation method of the present invention is a method of generating a distance image showing a distance to a road surface reflected in a moving image taken by a camera mounted on a vehicle, and is a moving image taken by a camera mounted on the vehicle. A step of inputting image data and data of the position or speed of the vehicle when the moving image is taken, and a step of calculating a light beam extending from the optical center of the camera with respect to the pixels of the image of a predetermined frame of the moving image. A step of obtaining the motion parameter of the camera based on the moving image data and the data of the position or speed of the vehicle, a step of obtaining the shape of the road surface on which the vehicle traveled based on the motion parameter, and the light beam and the above. Based on the step of calculating the distance to the pixel corresponding to the light beam in the image of the predetermined frame based on the intersection with the road surface, and to the road surface in the image of the predetermined frame based on the distance calculated by the distance calculation unit. It includes a step of generating a distance image showing the distance of.

In the distance image generation method of the present invention, the scale of the error included in the distance calculated in the step of calculating the distance is (i) the pitch angle of the camera when the image of the predetermined frame is taken and the pixel. An error model in which the displacement of the camera from the pitch angle when the corresponding road surface is restored and (ii) the length of the predetermined frame from the optical center of the camera to the pixel in the image are expressed as variables. It includes a step of reading from the error model storage unit and a step of calculating the scale of the error with respect to the distance obtained in the step of calculating the distance with reference to the error model.

The program of the present invention is a program for generating a distance image showing the distance to the road surface reflected in a moving image taken by a camera mounted on a vehicle, and is photographed by a computer and a camera mounted on the vehicle. The step of inputting the moving image data and the data of the position or speed of the vehicle when the moving image was taken, and the light beam extending from the optical center of the camera for the pixels of the image of the predetermined frame of the moving image are calculated. The step of obtaining the motion parameter of the camera based on the moving image data and the data of the position or speed of the vehicle, the step of obtaining the shape of the road surface on which the vehicle traveled based on the motion parameter, and the light beam. In the image of the predetermined frame, based on the step of calculating the distance to the pixel corresponding to the light beam in the image of the predetermined frame based on the intersection of the and the road surface, and the distance calculated by the distance calculation unit. A step of generating a distance image showing the distance to the road surface is executed.

The program of the present invention tells a computer the scale of the error included in the distance calculated in the step of calculating the distance to (i) the pitch angle of the camera when the image of the predetermined frame is taken and the pixel. An error model in which the displacement of the camera from the pitch angle when the corresponding road surface is restored and (ii) the length of the predetermined frame from the optical center of the camera to the pixel in the image are expressed as variables. The step of reading from the error model storage unit and the step of calculating the scale of the error with respect to the distance obtained in the step of calculating the distance with reference to the error model are further executed.

According to the present invention, it is possible to generate a distance image to a road surface based on a moving image taken by a camera and a position or speed of a vehicle.

It is a figure which shows the relationship of the distance image generation apparatus, the model generation apparatus, and the inter-vehicle distance calculation apparatus. It is a figure which shows the structure of the distance image generation apparatus of 1st Embodiment. It is a figure which shows the installation position of a camera. It is a figure which shows the point in the image which is the object of distance calculation. (A) It is a figure which described the light beam up to the point P with Z = 0 (m) as the position of the camera when the image of a predetermined frame was taken. (B) It is a figure which showed the road surface restored from the motion parameter. It is a flowchart which shows the operation of the distance image generation apparatus of 1st Embodiment. It is a figure which shows the structure of the model generation apparatus of 1st Embodiment. It is a figure which shows the structure of the inter-vehicle distance calculation device. It is a flowchart which shows the operation of the inter-vehicle distance calculation device. It is a figure which shows the structure of the distance image generation apparatus of 2nd Embodiment. It is a flowchart which shows the operation of the distance image generation apparatus of 2nd Embodiment. It is a figure which shows the structure of the model generation apparatus of 2nd Embodiment. It is a figure which shows the structure of the error model generation apparatus which generates the error model used in the inter-vehicle distance calculation apparatus. It is a figure which shows the main sensor value used when determining the inter-vehicle distance D. It is a flowchart which shows the operation of an error model generator. The concept of error scale calculation in another example will be described.

Hereinafter, the distance image generation device, the model generation device, and the inter-vehicle distance calculation device according to the embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing the relationship between the distance image generation device, the model generation device, and the inter-vehicle distance calculation device described below. The distance image generation device accepts input of moving image data and vehicle speed data when the moving image is taken, and generates a distance image with respect to an image (captured image) of a predetermined frame of the moving image data. According to this embodiment, a distance image can be obtained without using a lidar or a stereo camera. Since the distance image can be generated from the moving image data and the vehicle speed data, a large amount of distance images can be easily prepared.

The model generation device uses the captured image and the distance image generated by the distance image generation device as teacher data to generate a model for inferring the distance image from the captured image. Here, the model may be a model of a neural network.

The inter-vehicle distance calculation device uses the model generated by the model generation device to obtain the inter-vehicle distance to the preceding vehicle. As an outline of the process, the distance between vehicles is obtained by applying the captured image to the model to generate a distance image and reading the distance information of the pixels at the lower end position of the preceding vehicle. Hereinafter, the configuration of each device will be described in detail.

(First Embodiment)
[Distance image generator]
FIG. 2 is a diagram showing the configuration of the distance image generation device 1 according to the first embodiment. The distance image generation device 1 generates a distance image of the road surface reflected in the image of a predetermined frame of the moving image data. The distance image generation device 1 includes an input unit 10, a light ray calculation unit 11, a motion parameter calculation unit 12, a road surface restoration unit 13, a distance calculation unit 14, a distance image generation unit 15, and an output unit 16. doing. The input unit 10 receives input of moving image data taken by a camera mounted on the vehicle and vehicle speed data when the moving image is taken.

The distance image generation device 1 generates a distance image for an image of a predetermined frame of moving image data. The predetermined frame may be the entire frame of the moving image data, or may be, for example, a frame sampled from the moving image data at a predetermined interval (for example, 100 ms).

The light ray calculation unit 11 calculates light rays extending from the optical center of the camera for all pixels of the image of a predetermined frame. The motion parameter calculation unit 12 obtains the motion parameters of the camera based on the moving image data and the vehicle speed data. Specifically, the motion parameter calculation unit 12 obtains the rotation parameters (yaw, pitch, roll) and the moving direction of the camera based on the moving image data. Further, the motion parameter calculation unit 12 associates the moving distance in the image with the actual moving distance of the camera based on the vehicle speed data. In the present embodiment, the example of using the vehicle speed data when the moving image is taken is given, but the position data of the vehicle when the moving image is taken may be used instead of the vehicle speed data. Even if the position data of the vehicle is used, the movement distance in the image can be associated with the actual movement distance, and the motion parameter of the camera can be obtained.

The road surface restoration unit 13 restores the movement of the camera based on the motion parameters, thereby restoring the movement of the vehicle equipped with the camera (that is, the vehicle that has taken the moving image), thereby restoring the road surface on which the vehicle has traveled. Restore. For example, the undulations of the road surface can be obtained from the displacement of the pitch angle. The difference in height in the width direction of the road surface can be obtained from the displacement of the roll angle.

The distance calculation unit 14 obtains the intersection of the light rays up to each pixel obtained in the image of the predetermined frame and the restored road surface, and calculates the distance to the pixel corresponding to the light rays obtained at the intersections. Prior to the explanation of the specific distance calculation, the orientation of the camera installed in the vehicle and the coordinate system used below will be described.

FIG. 3 is a diagram showing an installation position of the camera 10. The camera 10 is installed at the tip of the vehicle (front bumper). The height of the camera 10 from the ground is H, and in this document, the ground in the direction vertically below the optical center of the camera 10 is referred to as a ground point. The distance image generator 1 has data on the height H and is known.

When installing the camera 10 in the vehicle, the optical axis (Z axis) of the camera 10 coincides with the traveling direction of the vehicle, and the horizontal axis (X axis) of the image taken by the camera 10 is parallel to the ground. The installation angle of the camera 10 and the like are adjusted. As a result, the vertical axis (Y axis) of the image coincides with the direction perpendicular to the ground.

Next, the calculation of the distance to the pixel in the image will be described. Here, an example of calculating the distance to one point in the image, for example, the point P in the image shown in FIG. 4 will be described, but the distance calculation unit 14 has the same description as the following for all the pixels in the image. Process.

FIG. 5A is a diagram showing light rays up to point P with the position of the camera when an image of a predetermined frame is taken as Z = 0 (m). From the position of point P in the image, the light beam from the optical center of the camera to point P can be obtained. The point P exists on this ray, but since the depth is unknown, it is not possible to specify where the road surface is at this stage.

FIG. 5B is a diagram showing a road surface restored from the motion parameters. When the road surface is restored by the motion parameters based on the moving image data and the vehicle speed data, if the point P is a point on the road surface, the road surface and the light rays up to the point P intersect. As a result, the distance to the point P can be obtained in the image of the predetermined frame. If the point P is not a point on the road surface, or if the vehicle does not pass on the point P even if the point P is a point on the road surface, the restored road surface and the point P do not intersect. As described above, the distance to point P can be obtained. The distance calculation unit 14 obtains the distance to each pixel of the road surface in the image of the predetermined frame by performing the above processing on all the pixels in the image of the predetermined frame.

Although only the YZ direction has been described here for ease of understanding, the light ray to the point P actually has a value in the X direction as well. It is clear that it is possible to find the intersection of the light beam and the restored road surface even in the case of three dimensions by extending the above explanation to three dimensions.

The distance image generation unit 15 generates a distance image based on the distance data to each pixel obtained by the distance calculation unit 14. Specifically, the distance image generation unit 15 generates a distance image representing the distance by color by coloring the pixels according to the distance to each pixel.

The output unit 16 has a function of outputting the distance image data generated by the distance image generation unit 15. The output unit 16 may be, for example, a monitor that displays a distance image to the user, or a communication interface that outputs distance image data to another in-vehicle device.

FIG. 6 is a flowchart showing the operation of the distance image generation device 1. The distance image generation device 1 accepts input of moving image data and vehicle speed data when the moving image is taken (S10). Each frame of the moving image is associated with the vehicle speed data, and it is possible to know how many kilometers per hour the vehicle was at when each frame was photographed.

The distance image generator 1 calculates light rays extending from the optical center of the camera for all pixels of the image of a predetermined frame (S11). The distance image generation device 1 calculates the motion parameters of the camera based on the motion image data and the vehicle speed data (S12), and restores the road surface based on the motion parameters (S13). Next, the distance image generation device 1 determines whether or not the light rays to each pixel and the restored road surface intersect (S14), and if they do not intersect (NO in S14), the moving image data and the vehicle speed. Return to data input (S10).

When the light ray and the road surface intersect (YES in S14), the distance to the pixel corresponding to the light ray is calculated based on the intersection (S15). The distance image generation device 1 determines whether or not the processing is completed for all the pixels of the image of the predetermined frame (S16), and if the processing is completed (YES in S16), the distance image generation device 1 is the distance. A distance image is generated using the distance data to each pixel for which is calculated (S17). If the processing of all pixels is not completed (NO in S16), the process returns to step S10 for inputting the moving image data and the vehicle speed data.

Here, the determination of the completion of pixel processing will be described. When the light ray and the road surface intersect, the road surface image generation device 1 determines that the processing of the pixels applied to the light ray is completed. In addition to this, the road surface image generation device 1 determines that the processing of the pixels of the image is completed when the image travels by a predetermined distance from the place where the image of the predetermined frame is taken. This is because some light rays do not intersect the road surface, so if the process continues to wait for the light rays to intersect the road surface, the process will not end.

Although the distance image generation device 1 of the embodiment has been described above, the distance image generation device 1 is composed of a microcomputer including a CPU, ROM, RAM, etc. (not shown). The distance image generation device 1 generates a distance image for an image of a predetermined frame cut out from the moving image data according to, for example, a program stored in the ROM. Such programs are also included in the scope of the present invention.

[Model generator]
FIG. 7 is a diagram showing the configuration of the model generation device 20. The model generation device 20 has an input unit 22 that receives input of teacher data. The teacher data is a photographed image and a distance image thereof. The teacher data can be generated in large quantities by using the moving image data by the distance image generation device 1 described above. The model generation device 20 has a model generation device 23 that generates a model for inferring a distance image from an image taken by a camera using teacher data, and a storage unit that stores the generated model. There is. The model generated by the model generation device 20 is, for example, a neural network model.

[Inter-vehicle distance calculation device]
FIG. 8 is a diagram showing the configuration of the inter-vehicle distance calculation device 30. When the inter-vehicle distance calculation device 30 detects the preceding vehicle from the image taken by the camera 40 mounted on the vehicle, it superimposes the detected preceding vehicle on the distance image generated from the captured image and reaches the pixel at the lower end of the preceding vehicle. By acquiring the distance of, the inter-vehicle distance to the preceding vehicle is calculated.

The inter-vehicle distance calculation device 30 includes an input unit 31, a vehicle detection unit 32, a distance image generation unit 33, and a distance determination unit 34. Hereinafter, each configuration included in the inter-vehicle distance calculation device 30 will be described with reference to FIG. 9 which describes the flow of inter-vehicle distance calculation.

The input unit 10 is connected to a camera 36 mounted on the vehicle, and image data taken by the camera 36 is input (S20). The vehicle detection unit 32 detects the preceding vehicle from the input captured image (S21). For example, HOG features are extracted from the image, and a pattern classifier such as a support vector machine is applied to the obtained features. HOG features are introduced in detail in N. Dalal and B. Triggs "Histograms of oriented gradients for human detection" CVPR 2005.

When the preceding vehicle is not detected (NO in S22), the inter-vehicle distance calculation device 30 returns to the step of inputting the captured image from the camera (S20). When the preceding vehicle is detected (YES in S22), the distance image generation unit 33 generates a distance image (S23).

The distance image generation unit 33 is connected to a storage unit 35 that stores a model for inferring a distance image from the captured image. This model is a model generated by the model generation device 20 described above. The distance image generation unit 33 reads a model from the storage unit 35 and applies a captured image in which the preceding vehicle is detected to the read model to generate a distance image corresponding to the captured image.

The distance determination unit 34 superimposes the preceding vehicle and the distance image, and determines the distance to the pixel at the lower end of the preceding vehicle as the inter-vehicle distance (S24). Since the distance image has information on the distance to each pixel, the inter-vehicle distance to the preceding vehicle can be obtained as soon as the pixel at the lower end of the preceding vehicle is obtained.

Here, an example of generating a distance image using the model generated by the model generation device 20 described above has been given, but a model generated by the model generation device 21 described later can also be used.

(Second Embodiment)
[Distance image generator]
FIG. 10 is a diagram showing the configuration of the distance image generation device 2 of the second embodiment. The distance image generation device 2 of the second embodiment performs a process of calculating the scale of the error included in the distance image in addition to generating the distance image.

The basic configuration of the distance image generation device 2 of the second embodiment is the same as that of the distance image generation device 1 of the first embodiment, but the distance image generation device 2 of the second embodiment is , It is provided with a configuration for calculating an error scale of how much error the distance calculated by the distance calculation unit 14 includes. The distance image generation device 2 of the second embodiment has an error scale calculation unit 17 and an error model storage unit 18.

The error model is
(I) When the pitch angle of the camera when the image of the predetermined frame is taken (the pitch angle when Z = 0 mm in FIG. 5B) and the road surface where the pixel to be calculated the distance is located are restored. Displacement with the camera pitch angle (pitch angle when Z = 38 mm in FIG. 5B),
(Ii) The length of a predetermined frame from the optical center of the camera to the pixel in the image,
It is a model of the scale of the error with the variable. In this embodiment, the error model is a look-up table having the displacement of the pitch angle and the value of the error scale with respect to the length in the image.

In the error model, learning is performed using the error between the distance to the pixel obtained by calculation and the true value, the displacement of the pitch angle of (i) and the length of (ii) of the image used in the calculation as teacher data. Can be generated by. As the teacher data, data actually acquired under various conditions may be used, but learning may be performed using data generated under different conditions by Monte Carlo simulation. An example of error model generation will be described later.

The error scale calculation unit 17 referred to the error scale model stored in the error model storage unit 18, and (i) restored the road surface corresponding to the pitch angle and pixels of the camera when the image of the predetermined frame was taken. The scale of the error corresponding to the deviation from the pitch angle of the camera at that time and (ii) the length of a predetermined frame from the optical center of the camera to the pixel in the image is calculated. Since the look-up table is used in the present embodiment, the error scale corresponding to the displacement of the pitch angle of (i) and the length of (ii) may be read from the look-up table.

FIG. 11 is a flowchart showing the operation of the distance image generation device 2 according to the second embodiment. The basic operation of the distance image generation device 2 of the second embodiment is the same as that of the distance image generation device 1 of the first embodiment, and steps S30 to S35 are steps S10 to 10 described with reference to FIG. Corresponds to S15. The distance image generation device 2 of the second embodiment has a step S36 of calculating the distance to each pixel (S35) and then calculating the error scale with respect to the distance. The distance image generation device 1 generates a distance image based on the distance to each pixel and stores the error scale data up to each pixel.

[Model generator]
FIG. 12 is a diagram showing the configuration of the model generation device 21 of the second embodiment. The model generation device 21 includes an input unit 22 that receives input of teacher data. The teacher data is information on the captured image, its distance image, and the error scale. The teacher data can be generated in large quantities by using the moving image data by the distance image generation device 2 described above. Further, the model generation device 21 has a model generation unit 23 that learns a model using the teacher data and generates a model, and a storage unit 24 that stores the generated model. The model generated by the model generation device 21 is, for example, a neural network model.

In the model generation device 21 of the second embodiment, the model generation unit 23 has a loss function setting unit 25. The model generation unit 23 generates a learning model using the captured image, the distance image thereof, and the error scale as teacher data, and stores the generated learning model in the storage unit 35. If the estimator with the learning parameter θ is _{represented by the function f θ} (・), the input variable is x, and the target value up to the pixel is Z, the following formulation is possible as a regression problem.

The model generation unit 23 treats the distance image as a correct answer, but the distance to each pixel of the distance image includes an error. The model generation unit 23 performs learning with the larger the scale of the error, the smaller the influence on the loss function. Specifically, in the present embodiment, learning is performed using the following equation so that distance data having a large error scale is not regarded as important.

[Error model generator]
Next, a method of generating an error model used in the distance image generation device of the second embodiment will be described.

(First example of error model generation)
FIG. 13 is a diagram showing a configuration of an error model generation device 40 that generates an error model used in the distance image generation device 2. First, an outline of the error model generation device 40 will be described. FIG. 14 is a diagram showing main sensor values used when determining the distance D to the pixel P on the road surface. There are the position of the own vehicle (X, Y, Z), the pitch angle φ of the camera, and the position coordinate y of the pixel P on the road surface reflected in the image. Note that f is the focal length and H is the height of the camera.

Hereinafter, the configuration of the error model generation device 40 will be specifically described.
As shown in FIG. 13, the error model generation device 40 includes a road structure information input unit 41 for inputting road structure information, a simulation unit 42 for simulating the distance to the pixel P reflected in the captured image, and a simulation unit 42. It has an error analysis unit 47 that performs error analysis using the distance and error data obtained by the error analysis unit 47, and a storage unit 48 that stores an error model generated based on the analysis result by the error analysis unit 47.

The simulation unit 42 includes a true value calculation unit 43, an error setting unit 44, a distance calculation unit 45, and an error scale calculation unit 46. The true value calculation unit 43 has a function of calculating the true value of the distance to the pixel P. The distance calculation unit 45 has a function of performing a simulation of moving the own vehicle to the position of the pixel P, which is the target of the distance calculation, and calculating the distance to the pixel P.

The distance calculation unit 45 detects the position of the own vehicle (X, Y, Z), the posture of the own vehicle (Oroll, Opitch, Oyaw), and the detection position of the pixel P while the own vehicle moves to the position of the preceding vehicle in the calculation of the distance. A random error is set with respect to the true value of (Lx, Ly), and the distance to the pixel P is calculated using the value to which the error is added. The error setting unit 44 has a function of generating a random error and inputting it to the distance calculation unit 46. When determining the distance to the pixel P based on the speed of the own vehicle and the inter-vehicle time, the error setting unit 44 randomly sets not only the error of each sensor value but also the speed of the own vehicle and the inter-vehicle time. You may.

The error scale calculation unit 46 has a function of calculating an error based on the difference between the distance calculated by the distance calculation unit 45 and the true value. The simulation unit 42 changes the added error and repeatedly calculates the distance by the distance calculation unit 45 to obtain a large number of simulation results.

The error analysis unit 47 has a function of generating an error model based on the simulation result of the distance and the error with respect to the distance. With respect to the obtained simulation result, the error analysis unit 47 uses the displacement of the pitch angle Opitch of the own vehicle and the detection position Ly of the pixel P as input variables, and obtains the error from the regression problem of the variance of the error as shown in the following equation. Model the variance.

FIG. 15 is a flowchart showing the operation of the error model generation device 40. The error model generator 40 first accepts input of information on the road structure to be simulated (S40). As the road structure information, input the structure information conforming to the provisions of the Road Structure Ordinance. The error model generation device 40 determines the own vehicle and the pixel P for distance calculation target (S41), and calculates the true value of the distance (S42).

Next, the error model generation device 40 performs a simulation of moving from the position of the own vehicle to the position of the pixel, and calculates the distance to the pixel (S43). Specifically, the distance is calculated by adding a randomly generated error to the true value of each sensor value while moving from the position of the own vehicle to the position of pixel P, and from the calculated true value of the distance. The error of (S43) is calculated.

The error model generation device 40 sets a random error and repeatedly calculates the distance and the error. That is, it is determined whether or not to end the simulation at the pixel (S44), and if it is determined not to end (NO in S44), the error model generator 40 sets a random error and distances. And the error is calculated (S43). When it is determined that the simulation with the pixel is completed (YES in S44), it is determined whether or not the distance and error simulation is performed for another pixel in the image (S45). If it is determined that the simulation is also performed for another pixel (YES in S45), the error model generator 40 returns to the process of determining the pixel to be calculated (S41).

If it is determined that the simulation is not performed for another pixel (NO in S45), the error model generator 40 determines whether or not to perform the simulation in another road structure (S46). When performing a simulation with another road structure (YES in S46), the process returns to the input of road structure information (S40), accepts the input of new road structure information, and performs the above simulation (S41 to S45). .. When the simulation with another road structure is not performed (NO in S46), the distance and the error obtained by the simulation are analyzed to generate an error model, and the generated error model is stored (S47).
Here, as an example, an example of polynomial regression of the error model is shown.

The error model generator 40 repeats pseudo-distance estimation by simulation and collects samples of the measurement results of the distance, so that the error due to polynomial regression using each sample and the governing term of the error identified analytically is used. Modeling is possible.

(Second example of error model generation)
First, the concept of error scale calculation in the second example will be described. The road surface on which the own vehicle is placed and the road surface on which the pixel P for which the distance is to be obtained are locally flat. The present inventors have noted that the error in the distance to the pixel P depends on the inclination of the road surface with the pixel P as seen from the road surface on which the own vehicle is mounted.

これをＺについて解くと、

となる。計算対象ピクセルの観測誤差を

とすれば、１次のテイラー展開により、距離の誤差のスケールは、次式（４）で表される。

FIG. 16 is a diagram showing the principle of finding the distance to the pixel P in the second example. The inclination of the road surface with the pixel P with respect to the plane on which the own vehicle is placed is “a”. The position of the pixel P is (i) the intersection of the straight line connecting the optical center of the camera of the own vehicle and the pixel P and (ii) the road surface on which the pixel P is located. (I) The straight line connecting the optical center of the camera of the own vehicle and the pixel to be calculated is represented by the following equation (1), where f is the focal length of the camera, and (ii) the road surface is represented by the following equation (2). To.

If you solve this for Z,

Will be. Observation error of the pixel to be calculated

If so, the scale of the error of the distance is expressed by the following equation (4) by the first-order Taylor expansion.

In the second example, an example in which a pinhole camera model is adopted has been described, but it is possible to obtain an error scale in a camera other than the pinhole camera in the same manner as described above.

The present invention is useful as a distance image generator that generates a distance image from a captured image.

1, 2 Distance image generator 10 Input unit 11 Light ray calculation unit 12 Motion parameter calculation unit 13 Road surface restoration unit 14 Distance calculation unit 15 Distance image generation unit 16 Output unit 17 Error scale calculation unit 18 Error model storage unit 20, 21 Model generation Device 22 Input unit 23 Model generation unit 24 Model storage unit 25 Loss function setting unit 30 Inter-vehicle distance calculation device 31 Input unit 32 Vehicle detection unit 33 Distance image generation unit 34 Distance determination unit 35 Model storage unit 36 Camera 37 Output device 40 Error model Generator 41 Road structure information input unit 42 Simulation unit 43 True value calculation unit 44 Error setting unit 45 Distance calculation unit 46 Error scale calculation unit 47 Error analysis unit 48 Storage unit

Claims (9)

An input unit for inputting moving image data taken by a camera mounted on the vehicle and data on the position or speed of the vehicle when the moving image was taken.
A ray calculation unit that calculates a ray extending from the optical center of the camera with respect to the pixels of the image of a predetermined frame of the moving image.
A motion parameter calculation unit that obtains motion parameters of a camera based on the motion image data and the position or speed data of the vehicle, and a motion parameter calculation unit.
A road surface restoration unit that obtains the shape of the road surface on which the vehicle traveled based on the motion parameters,
A distance calculation unit that calculates the distance to the pixel corresponding to the light ray in the image of the predetermined frame based on the intersection of the light ray and the road surface.
A distance image generation unit that generates a distance image indicating the distance to the road surface in the image of the predetermined frame based on the distance calculated by the distance calculation unit.
Distance image generator equipped with. The scale of the error included in the distance obtained by the distance calculation unit is (i) the pitch angle of the camera when the image of the predetermined frame is taken and the pitch angle of the camera when the road surface corresponding to the pixel is restored. And (ii) an error model storage unit that stores an error model expressing the length of the predetermined frame from the optical center of the camera to the pixel in the image as variables.
With reference to the error model, an error scale calculation unit that calculates the scale of the error with respect to the distance obtained by the distance calculation unit, and
The distance image generator according to claim 1. A model for inferring a distance image from a captured image is generated using a set of a distance image generated by the distance image generator according to claim 1 and an image of a predetermined frame that is the source of the distance image as teacher data. Model generator. A distance image from a captured image using a set of a distance image generated by the distance image generator according to claim 2, an image of a predetermined frame that is the source of the distance image, and a scale of the error as teacher data. A model generator that generates a model that infers
A model generator that performs learning with a smaller effect on the loss function as the scale of the error increases. A storage unit that stores the model generated by the model generator according to claim 3 or 4.
An input unit for inputting images taken by a camera mounted on the vehicle,
A vehicle detection unit that detects the preceding vehicle from the image input by the input unit, and
When the vehicle detection unit detects the preceding vehicle, the distance image generation unit reads the model from the storage unit and applies the image to the model to generate a distance image.
A distance determining unit that superimposes the preceding vehicle on the distance image and determines the distance to the pixel at the lower end of the preceding vehicle as the inter-vehicle distance.
An inter-vehicle distance calculation device equipped with. It is a method of generating a distance image showing the distance to the road surface reflected in a moving image taken by a camera mounted on a vehicle.
A step of inputting moving image data taken by a camera mounted on the vehicle and data of the position or speed of the vehicle when the moving image was taken, and
A step of calculating a light ray extending from the optical center of the camera with respect to the pixel of the image of a predetermined frame of the moving image, and
A step of obtaining the motion parameters of the camera based on the moving image data and the position or speed data of the vehicle, and
A step of obtaining the shape of the road surface on which the vehicle traveled based on the motion parameters, and
A step of calculating the distance to a pixel corresponding to the light ray in the image of the predetermined frame based on the intersection of the light ray and the road surface.
A step of generating a distance image showing the distance to the road surface in the image of the predetermined frame based on the distance calculated by the distance calculation unit, and
Distance image generation method with. The scale of the error included in the distance calculated in the step of calculating the distance is (i) the pitch angle of the camera when the image of the predetermined frame is taken and the camera when the road surface corresponding to the pixel is restored. (Ii) A step of reading an error model representing the length of the predetermined frame from the optical center of the camera to the pixel in the image as variables from the error model storage unit.
With reference to the error model, a step of calculating the scale of the error with respect to the distance obtained in the step of calculating the distance, and a step of calculating the error scale.
The distance image generation method according to claim 6. It is a program for generating a distance image showing the distance to the road surface reflected in a moving image taken by a camera mounted on a vehicle, and is a program for a computer.
A step of inputting moving image data taken by a camera mounted on the vehicle and data of the position or speed of the vehicle when the moving image was taken, and
A step of calculating a light ray extending from the optical center of the camera with respect to the pixel of the image of a predetermined frame of the moving image, and
A step of obtaining the motion parameters of the camera based on the moving image data and the position or speed data of the vehicle, and
A step of obtaining the shape of the road surface on which the vehicle traveled based on the motion parameters, and
A step of calculating the distance to a pixel corresponding to the light ray in the image of the predetermined frame based on the intersection of the light ray and the road surface.
A step of generating a distance image showing the distance to the road surface in the image of the predetermined frame based on the distance calculated by the distance calculation unit, and
A program that executes. On the computer
The scale of the error included in the distance calculated in the step of calculating the distance is (i) the pitch angle of the camera when the image of the predetermined frame is taken and the camera when the road surface corresponding to the pixel is restored. (Ii) A step of reading an error model representing the length of the predetermined frame from the optical center of the camera to the pixel in the image as variables from the error model storage unit.
With reference to the error model, a step of calculating the scale of the error with respect to the distance obtained in the step of calculating the distance, and a step of calculating the error scale.
8. The program according to claim 8.

Images