JP7461160B2 - Three-dimensional information estimation system, three-dimensional information estimation method, and computer-executable program - Google Patents

Description

The present invention relates to a three-dimensional information estimation system, a three-dimensional information estimation method, and a program executable by a computer.

Conventionally, techniques proposed for estimating 3D information include using a stereo camera alone to exploit the parallax between left and right images, and using LiDAR (Laser Imaging Detection and Ranging) for measurement. When using the above-mentioned stereo camera or LiDAR, the sensor itself is expensive, resulting in a high system cost.

On the other hand, in recent years, there has been active research into surrounding environment recognition technology using millimeter wave radar and monocular cameras, but most of the research has focused only on dynamic movement, and no method has been proposed for accurately recognizing three-dimensional information about structures. If three-dimensional information about structures could be estimated using inexpensive radar and monocular cameras, it would be advantageous in terms of cost.

JP 2010-286926 A

The present invention has been made in view of the above, and provides a three-dimensional information estimation system, a three-dimensional information estimation method, and a computer-executable three-dimensional information estimation system capable of estimating three-dimensional information of a structure with a low-cost configuration. The purpose of this program is to provide a comprehensive program.

In order to solve the above problems and achieve the object, the present invention is characterized by comprising a radar that transmits radar waves to the surroundings and receives reflected waves reflected by an object to obtain a radar signal, a camera that captures an image of the surroundings to obtain image information, a contour information extraction means that extracts contour information of the image information output from the camera, a target detection means that searches for the highest intensity point in each direction for the radar signal output from the radar and detects the nearest reflection point on the image information output from the camera, a distance information assignment means that assigns distance information of the nearest reflection point in each direction to the contour information of the image information in the same direction, a height information assignment unit that assigns height information to the contour information of the image information, and a three-dimensional information estimation means that estimates three-dimensional information according to the contour information, the distance information, and the height information.

According to one aspect of the present invention, the camera may be a monocular camera.

According to one aspect of the present invention, the distance information providing means may assume that the distance information between adjacent orientation bins for the nearest reflection point changes linearly, and may interpolate the missing orientation distance information.

According to one aspect of the present invention, the target detection means searches each direction for the radar signal output from the radar, determines the highest intensity point in each direction, plots the highest intensity point in each direction on the image information output from the camera, calculates the brightness characteristic value near each highest intensity point on the image information, and determines whether each highest intensity point on the image information is a real image or a false image based on the calculated brightness characteristic value. If it is determined to be a real image, the highest intensity point is set as the nearest reflection point, and if it is determined to be a false image, the next highest intensity signal point in that direction is obtained, and the process is repeated until it is determined to be a real image. If the target signal disappears, it is determined that there is no real image in that direction.

Further, according to one aspect of the present invention, the radar may be a millimeter wave radar.

In order to solve the above-mentioned problems and achieve the object, the present invention is characterized by including a step of a radar transmitting radar waves to the surroundings and receiving the reflected waves reflected by an object to obtain a radar signal, a step of a camera capturing an image of the surroundings to obtain image information, a step of extracting contour information of the image information output from the camera, a step of searching for the maximum intensity point in each direction for the radar signal output from the radar and detecting the nearest reflection point on the image information output from the camera, a step of adding distance information of the nearest reflection point in each direction to the contour information of the image information in the same direction, a step of adding height information to the contour information of the image information, and a step of estimating three-dimensional information according to the contour information, the distance information, and the height information.

Furthermore, in order to solve the above-mentioned problems and achieve the objects, the present invention provides a radar system that transmits radar waves to the surroundings of a computer, receives reflected waves reflected by an object, and acquires radar signals. Detecting the target object by searching for the highest intensity point in each direction for the radar signal output from the camera and detecting the nearest reflection point on the image information output from the camera. a contour information extracting means for extracting contour information of the image information output from the camera; and a contour information adding means for adding distance information of the nearest reflection point in each direction to the contour information of the image information in the same direction. and height information adding means for adding height information to contour information of the image information, and three-dimensional information estimating means for estimating three-dimensional information according to the contour information, the distance information, and the height information. It is characterized in that it is a computer-executable program for functioning.

According to the present invention, it is possible to estimate three-dimensional information of a structure with a low-cost configuration.

FIG. 1 is a diagram showing an example of a schematic configuration of a three-dimensional information estimation system according to the present embodiment. FIG. 2 is a diagram illustrating an example of a flow illustrating the general operation of the three-dimensional information system according to the present embodiment. FIG. 3 is a diagram for explaining a method of calculating the resolution. FIG. 4 is a diagram showing an example in which the nearest reflection point of a radar signal and the brightness dispersion value are plotted on an image captured by a camera. FIG. 5 is a diagram for explaining signals to be used in the analysis and signals to be excluded. FIG. 6 is a diagram for explaining the association of image contour information and distance information. FIG. 7 is a diagram showing an example of the created three-dimensional map (first angle). FIG. 8 is a diagram showing an example of the created three-dimensional map (second angle). FIG. 9 is a diagram showing an example of the created three-dimensional map (third angle). FIG. 10 is a diagram showing an example of the created three-dimensional map (fourth angle).

DESCRIPTION OF THE PREFERRED EMBODIMENTS Below, embodiments of a three-dimensional information estimation system, a three-dimensional information estimation method, and a computer-executable program according to the present invention will be described in detail based on the drawings. Note that the present invention is not limited to this embodiment. Furthermore, the constituent elements in the embodiments described below include those that can be easily imagined by those skilled in the art or those that are substantially the same. Although the components of the invention are generally illustrated in the drawings herein, it will be readily appreciated that they may be arranged and designed in a wide variety of different configurations. Accordingly, the following more detailed description of embodiments of systems and apparatus of the invention does not limit the scope of the invention as set forth in the claims, but merely represents an example of selected embodiments of the invention. It is something. This specification incorporates known technology by reference. Therefore, those skilled in the art will appreciate, through the use of known techniques, that the invention may be practiced without one or more of the specific details, or with other methods, components, or materials.

[1. Principle of the present invention]
In the present invention, in order to estimate three-dimensional information of a structure with a low-cost configuration, a radar and a monocular camera are used. We propose a method for estimating three-dimensional information of surrounding structures using height information.

When extracting radar signals, radar signals from stationary structures are more susceptible to noise components such as clutter and false images than signals from moving pedestrians and vehicles, making it difficult to select radar signals. Therefore, in this invention, as a method for extracting the desired structure signal, image information is used to suppress the clutter components of the radar signal.

Specifically, the detected radar signals from each direction are plotted on the image, taking advantage of the property that there is a low possibility that unnecessary components will appear on the image at the same position as the false image position detected by the radar signal. By doing so, it is determined whether each signal is a signal from a target object. More specifically, when plotting radar signals on captured images, it is assumed that each radar signal is a reflection signal from a height of zero at each reflection distance, and when the target is in contact with the road surface. Plot the radar signal at the point on the image. The brightness dispersion value of the image is obtained near each radar signal plotted on the image, and each radar signal is classified as a real image or a false image based on the obtained brightness dispersion value. When the radar signal is a real image, the brightness of the transition portion between the target object and the road surface on the image is evaluated, so the brightness variance value tends to be large. On the other hand, even if a radar signal of a false image is plotted, the probability of evaluating the brightness of a road surface area where there is nothing on the image increases, so the brightness variance value tends to be small. In this way, by evaluating the brightness dispersion value on the image associated with each radar signal, it becomes possible to distinguish between a real image and a false image. In this way, the distance to the nearest object can be obtained with high accuracy for each direction.

At this time, signals are not detected (excluded) in directions where no object exists. Three-dimensional information of the structure in each direction is calculated using the distance information of the direction in which the signal was detected.

In this way, this invention focuses on the fact that images have the characteristic of having azimuth information on the horizontal axis, and therefore objects with the same horizontal coordinates can be considered to be in the same orientation. The image azimuth information and radar azimuth information are integrated, and the signal distance in the direction in which the radar detected the signal is linked to the image contour information in the same direction. This makes it possible to assign distance information to image information in a certain direction.

However, since the horizontal angular resolution of the radar is very low compared to the horizontal angular resolution of the image, it is necessary to estimate the distance information of the azimuths between the azimuth bins of the radar. Therefore, it is assumed that the distance information between adjacent azimuth bins changes linearly, and the missing azimuth distance information is interpolated.

Distance information is assigned to each pixel component on the horizontal axis (ground) of the image, and the resolution is calculated for each distance corresponding to the direction. The resolution calculated here is then used to calculate the height information for each contour component in each direction. By applying the above method, it is possible to estimate three-dimensional information about surrounding structures.

Furthermore, if the technology for estimating three-dimensional information of surrounding structures of this embodiment is applied to a system in which dynamic detection is performed using radar and a monocular camera, static information will be added, which is different from the conventional method. It may even be possible to achieve self-position estimation, which has been difficult. With the development of GNSS, typified by GPS, it has become possible to recognize only one's own position with high precision, but it is not possible to instantly derive the absolute direction. If the positional relationship with surrounding structures can be accurately determined, it becomes possible to recognize the observer's absolute orientation with a single measurement.

2. Configuration and Operation Examples of the Three-Dimensional Information Estimation System of the Present Embodiment
An example of the configuration and operation of a three-dimensional information estimation system according to the present embodiment will be described with reference to Figures 1 to 3. Figure 1 is a diagram showing an example of the schematic configuration of a three-dimensional information estimation system 1 according to the present embodiment. Figure 2 is a diagram showing an example of a flow showing the schematic operation of the three-dimensional information estimation system 1 according to the present embodiment.

With reference to FIG. 1, the configuration of a three-dimensional information estimation system 1 according to the present embodiment will be described. As shown in FIG. 1, a three-dimensional information estimation system 1 according to the present embodiment includes a three-dimensional information estimation device 10, a radar 20, and a camera 30.

The radar 20 transmits radar waves to the surroundings, receives reflected waves reflected by objects, and outputs them to the three-dimensional information estimation device 10 as radar signals. The radar 20 is, for example, an FM-CW millimeter wave radar (79 GHz band, 4 GHz bandwidth, MIMO radar (3 transmitting antennas, 6 receiving antennas), and the detection range is from -75 degrees to 75 degrees horizontally. The radar 20 transmits radar waves to the surroundings from a transmitter equipped with a plurality of transmitting antennas, and receives reflected waves reflected by an object using a receiver equipped with a plurality of receiving antennas. This is a sensor for detecting the distance, relative velocity, angle, and signal strength of an object by receiving it with a device.

The radar signal output from the radar 20 may be, for example, a signal consisting of a point or a sequence of points indicating one or more representative positions of the object whose reflected waves have been processed in the signal processing circuit included in the radar 20, Alternatively, it may be a signal indicating an unprocessed reflected wave. In this example, the radar signal output from the radar 20 includes information on the distance, direction, relative speed, and signal strength of the object calculated by the known FM-CW method in the signal processing circuit described above. shall be. Note that when unprocessed received waves are used as radar signals, these signal processes are executed in the three-dimensional information estimation device 10.

The camera 30 outputs image information of the surroundings to the three-dimensional information estimation device 10. The camera 30 is a sensor that receives visible light and infrared light and outputs the image information, which is information on the outer shape of the object. The camera 30 is, for example, a monocular camera, and can be configured, for example, by combining a CMOS global shutter sensor (for example, effective pixel count: 1920 pixels x 1200 pixels, frame rate: 41.6 fps, interface: USB 3.0) with a megapixel-compatible CCTV lens (for example, angle of view: 86.71 degrees x 61.44 degrees, focal length 6 mm), and has a detection range of, for example, a horizontal angle of -43.36 degrees to 43.36 degrees and an elevation angle of -30.72 degrees to 30.72 degrees.

The image information output from the camera 30 is composed of multiple frame images that are successive in time series, and each frame image is represented by pixel data. The pixel data is monochrome pixel data or color pixel data.

The three-dimensional information estimation device 10 includes a target detection unit 11, a contour extraction unit 12, a distance information assignment unit 13, a height information assignment unit 14, and a three-dimensional information estimation unit 15.

The target detection unit 11 searches for the maximum intensity point in each direction for the radar signal output from the radar 20, and detects the nearest reflection point on the image information output from the camera 30. The target detection unit 11 does not detect moving components (pedestrians, etc.) as the nearest reflection point in each direction.

The target detection unit 11 includes, for example, a threshold processing unit 40, a peak search unit 41, a plot unit 42, a brightness characteristic value calculation unit 43, and a real image/false image determination unit 44.

The threshold processing unit 40 performs threshold processing such as CFAR on the radar signal output from the radar 20. The peak search unit 41 determines the highest intensity point in each direction for the radar signal after threshold processing. When the real image/false image determination unit 44 determines that the highest intensity point is a false image, the next highest intensity signal point in the relevant direction is acquired.

The plot unit 42 plots the maximum intensity point of the radar signal in each direction on the image output from the camera 30. When plotting the maximum intensity point on the image, it assumes that the radar signal is a reflected signal from a height of zero at each reflection distance (because the radar elevation angle is small), calculates the corresponding coordinates on the image where the target is in contact with the road surface, and plots the radar signal point (maximum intensity point). If the peak search unit 41 acquires a signal point with the next highest intensity in the direction, the plot unit 42 plots the signal point with the next highest intensity in the direction.

The luminance characteristic value calculation unit 43 calculates the luminance characteristic value (e.g., luminance variance, luminance gradient) in the vicinity of each maximum intensity point on the image. When the plot unit 42 plots a signal point with the next highest intensity in the relevant direction, the luminance characteristic value calculation unit 43 calculates the luminance characteristic value in the vicinity of the signal point with the next highest intensity.

The real image/false image determining unit 44 determines whether the highest intensity point on the image is a real image or a false image based on the brightness characteristic value, and outputs the result of the real image. Specifically, when the real image/false image determination unit 44 determines that the image is a real image, it sets the highest intensity point as the nearest reflection point, and when it determines that it is a false image, it sets the next highest intensity point in the direction. The process is repeated until a high signal point is determined and determined to be a real image. However, if the target signal disappears, it is determined that there is no real image in that direction.
For example, if the brightness characteristic value is greater than or equal to the threshold value, the highest intensity point is determined to be a real image, and if the brightness characteristic value is not greater than the threshold value, the highest intensity point is determined to be a false image. be able to.

The contour extraction unit 12 extracts contour information from the image information output from the camera 30 and outputs it to the distance information assignment unit 13 and the height information assignment unit 14.

The distance information adding unit 13 adds distance information for each direction to the contour information of the image in the same direction. The distance information providing unit 13 may assume that distance information between adjacent azimuth bins changes linearly with respect to the nearest reflection point, and may interpolate the missing azimuth distance information.

The height information adding unit 14 adds height information (Z) to the outline information of the image.

The three-dimensional information estimation unit 15 estimates (draws) three-dimensional information by three-dimensional plotting on the contour information of the image according to the calculated distance information (X, Y) and height information (Z), and creates a three-dimensional map. Output.

The three-dimensional information estimation device 10 is equipped with a CPU (processor), a DSP, a RAM, a nonvolatile memory such as a ROM or a flash memory, an I/O, a monitor, and the like. Each function of the three-dimensional information estimating device 10 is realized by the CPU executing a program stored in a nonvolatile memory such as a ROM or a flash memory. By executing this program, a method corresponding to the program is executed.

When the CPU executes the program, the functions of the target object detection section 11, contour extraction section 12, distance information addition section 13, height information addition section 14, and three-dimensional information estimation section 15 are realized.

The method of realizing these elements constituting the three-dimensional information estimation device 10 is not limited to software, but it is also possible to realize some or all of the elements using hardware that combines logic circuits, analog circuits, etc. You can.

The overall processing of the three-dimensional information estimation system 1 of FIG. 1 will be explained with reference to FIG. 2.

In FIG. 2, the radar 20 transmits radar waves to the surroundings, receives the reflected waves, acquires a radar signal, and outputs it to the three-dimensional information estimation device 10 (step S1). The camera 30 captures the surroundings, acquires two-dimensional image information, and outputs it to the three-dimensional information estimation device 10 (step S2). It is assumed that the radar signal and the image information frames are synchronized.

The target detection unit 11 detects the nearest reflection point on the image in each direction of the radar signal based on the image information output from the camera 30 and the radar signal output from the radar 20 (step S3). The target detection unit 11 does not detect dynamic components (pedestrians, etc.) as the nearest reflection point in each direction. Specifically, the target detection unit 11 searches each direction for the radar signal output from the radar 20, determines the highest intensity point in each direction, and displays each direction on the image information output from the camera 30. Plot the highest intensity points in the azimuth, calculate the brightness characteristic values in the vicinity of each highest intensity point on the image information, and determine whether each highest intensity point on the image information is a real image or a false image based on the calculated brightness characteristic values. If it is determined to be a real image, the highest intensity point is set as the nearest reflection point, and if it is determined to be a false image, the signal point with the next highest intensity in the direction is acquired and the real image is determined. The process is repeatedly executed until it is determined that (the same process is repeatedly executed until the highest intensity point in each direction is determined to be a real image (until the nearest reflection point is obtained)). However, if the target signal disappears, it is determined that there is no real image in that direction.

The contour extraction unit 12 extracts contour information from the image information output from the camera 30 (step S4). The contour extraction unit 12 does not extract contour information for moving components (pedestrians, etc.). Moving components in an image can be detected by known methods, so a description of this is omitted.

The distance information adding unit 13 adds the distance information of each direction to the contour information of the image of the same direction (step S5). Specifically, for example, the distance information adding unit 13 converts the nearest reflection point of each direction from polar coordinate system data (r, θ) to Cartesian coordinate system data (X, Y), checks the signals of adjacent direction bins, interpolates between the adjacent signals, and then adds the distance information of each direction to the contour information of the image of the same direction. Here, the reason for interpolating between adjacent signals is that the horizontal angle resolution of the radar is very coarse compared to the horizontal angle resolution of the image, so it becomes necessary to estimate the distance information of the direction between each direction bin of the radar signal. Therefore, it is assumed that the distance information between adjacent direction bins changes linearly, and the missing distance information of the direction is interpolated.

In addition, distance measurement errors can be suppressed by smoothing the distance to each target by taking into account distance information in the same direction in past frames (e.g., the past five frames).

The height information adding unit 14 adds height information (Z) to the outline information of the image (step S6). Here, an example of a method for calculating height information (Z) will be explained. Since the distance to the contour component of the image in each direction has been calculated, the resolution (m/pix) of the image at the calculated distance is calculated, and height information is calculated using the resolution information. FIG. 3 is a diagram for explaining a resolution calculation method.

(1) The vertical resolution of the image at a distance Y (m) is calculated using the following equation (1).
Vertical resolution vReso (m/pix) = Y * 7.13/6/1200
... (1)
Here, 7.13 (mm) is the CMOS sensor size, 6 (mm) is the focal length of the CCTV lens, and 1200 (pix) is the number of pixels.

(2) Find the road surface coordinates on the image at the road surface height of distance Y (m) using the following formula (2).
Road surface coordinates rCol (pix) = ColCross + (CamHgt/vReso)
...(2)
Here, ColCross(pix) is the vanishing point coordinate of the vertical component, and CamHgt(m) is the camera installation height.

In perspective, a vanishing point is the point where lines that are actually parallel intersect when they are drawn as non-parallel. For example, parallel passageways converge at a single point on the image. The vanishing point refers to this convergence point. If the image is pre-calibrated so that the vanishing point and the center of the image coincide, the equation vanishing point = coordinates of the camera installation height at infinite distance holds true. Based on this premise, by subtracting the camera installation height (pix) at each distance from the vanishing point coordinates, the road surface coordinates rCol (pix) at that distance can be calculated using equation (2).

(3) The height to each contour component is calculated using the following equation (3).
Attention point height Z (m) = (TgtCol (pix) - rCol (pix)) * vReso
...(3)
Here, TgtCol(pix) is the vertical component coordinate of the pixel for which height is to be calculated.

Returning to FIG. 2, the three-dimensional information estimation unit 15 estimates three-dimensional information by three-dimensional plotting according to the contour information, the calculated distance information, and the height information, and creates a three-dimensional map (step S7).

[3. Verification example (example) of the three-dimensional information estimation device of this embodiment]
A verification example (example) of the three-dimensional information estimation system 1 of FIG. 1 according to the present embodiment will be described with reference to FIGS. 4 to 10. An example of creating a three-dimensional map in the surrounding environment as shown in FIG. 4 will be described.

In Figure 4, the diagram on the left shows an example in which the nearest reflection point and brightness variance value are plotted on an image captured by the camera 30, and the diagram on the right is a diagram for explaining a moving target in the image. In Figure 4, the broken line indicates the brightness variance value, the vertical dotted line indicates the direction of the radar 20, and the square indicates the nearest reflection point. The nearest reflection point is detected according to step S3 of the flow in Figure 2 above.

In Fig. 4, in the direction where a moving target such as a pedestrian exists, the signal from the target is greatly affected, so the signal excluding the direction in which the moving target exists is used to detect structures (stationary objects in general). ) to create a 3D map.

FIG. 5 is a diagram for explaining signals used for analysis and signals to be excluded. In the figure, structures are used for analysis and pedestrians are excluded. The distances and directions to all points of the signal used for analysis are calculated, and the contour information of the image is extracted according to steps S4 and S5 in the flow of FIG. granted to.

FIG. 6 is a diagram for explaining the association of image contour information and distance information. In FIG. 6, since the horizontal axis of the image represents the orientation, it is assumed that all the contour pixel groups in a vertical row are distance components in the relevant orientation. Distance information is linked to each vertical axis of image contour information. Note that although a moving target (pedestrian) is shown in FIG. 6, this is shown for reference, and contour information of the moving target (pedestrian) is not actually extracted.

The height information of the contour information of the image is calculated according to step S6 of the flowchart of FIG. 2 described above. Then, according to step S7 of the flowchart of FIG. 2 described above, a three-dimensional map is created by three-dimensional plotting according to the contour information and the calculated distance and height. FIGS. 7 to 10 are diagrams showing the created three-dimensional map from various angles. A short-distance component and a long-distance component may be distinguished and displayed (for example, a short-distance component may be displayed in red, and a long-distance component may be displayed in green). Note that in FIGS. 7 to 10, a moving target (pedestrian) is displayed in three dimensions, but this is shown for reference, and the moving target (pedestrian) is not actually drawn. As shown in Figures 7 to 10, by using the information of the highest intensity point detected in each direction of the radar signal, and by taking into account the estimated information (interpolation information) that the object distance changes linearly, It was possible to estimate 3D information of stationary objects such as buildings and warehouses.

The above verification results confirm that the 3D information estimation system 1 of this embodiment can estimate 3D information of surrounding structures by combining an inexpensive monocular camera and radar, and is useful for realizing recognition of the surrounding environment.

As described above, the three-dimensional information estimation system 1 of this embodiment includes a radar 20 that transmits radar waves to the surroundings and receives reflected waves reflected by an object to acquire a radar signal, a monocular camera 30 that captures images of the surroundings and acquires image information, a contour extraction unit 12 that extracts contour information from the image information output from the camera 30, a target detection unit 11 that searches for the highest intensity point in each direction for the radar signal output from the radar 20 and detects the nearest reflection point on the image information output from the camera 30, a distance information assignment unit 13 that assigns distance information of the nearest reflection point in each direction to the contour information of the image information in the same direction, a height information assignment unit 14 that assigns height information to the contour information of the image information, and a three-dimensional information estimation unit 15 that estimates three-dimensional information according to the contour information, distance information, and height information, making it possible to estimate three-dimensional information of a structure with a low-cost configuration.

The three-dimensional information estimation system 1 of the present embodiment can be suitably used in various applications such as, for example, an in-vehicle radar system, an advanced driving assistance system (ADAS), and a surrounding monitoring system.

1 3D information estimation system 10 3D information estimation device 11 Target detection section 12 Contour extraction section 13 Distance information addition section 14 Height information addition section 15 3D information estimation section (3D map generation section)
20 Radar 30 Camera 41 Peak search unit 42 Plot unit 43 Brightness characteristic value calculation unit 44 Real image/false image determination unit

Claims (7)

A radar that transmits radar waves to the surrounding area and receives reflected waves reflected by objects to obtain radar signals;
A camera that captures images of the surrounding area and obtains image information;
Contour information extraction means for extracting contour information of image information output from the camera;
When searching for the highest intensity point in each direction for the radar signal output from the radar and plotting the radar signal on the image information output from the camera, each radar signal has a height of zero at each reflection distance. By plotting the highest intensity point in each direction of the radar signal at the point on the image information where the target is in contact with the road surface, we can calculate the reflected signal from the camera. target detection means for detecting the nearest reflection point at the
Distance information adding means that converts the nearest reflection point in each direction from polar coordinate system data (r, θ) to orthogonal coordinate system data (X, Y) and adds distance information in each direction to contour information of an image in the same direction. and,
Calculate the resolution (m/pix) of the image at the distance to the contour component of the image in each calculated direction, calculate height information using the resolution information, and add height information to the contour information of the image information. height information providing means for
3D information estimating means for estimating 3D information according to the contour information, the distance information, and the height information;
A three-dimensional information estimation system characterized by comprising: The three-dimensional information estimation system according to claim 1, characterized in that the camera is a monocular camera. The three-dimensional information estimation system according to claim 1 or 2, characterized in that the distance information providing means assumes that the distance information between adjacent orientation bins for the nearest reflection point changes linearly, and interpolates the missing orientation distance information. The target object detection means includes:
Searching each direction for the radar signal output from the radar and determining the highest intensity point in each direction,
Plotting the highest intensity points in each direction on image information output from the camera, calculating brightness characteristic values in the vicinity of each highest intensity point on the image information,
Based on the calculated brightness characteristic value, it is determined whether each highest intensity point on the image information is a real image or a false image, and when it is determined that it is a real image, the highest intensity point is set as the nearest reflection point,
If the image is determined to be a false image, the signal point with the next highest intensity in the relevant direction is acquired, and the process is repeated until it is determined to be a real image. If the target signal disappears, it is determined that there is no real image in the relevant direction. The three-dimensional information estimation system according to any one of claims 1 to 3, characterized in that: The three-dimensional information estimation system according to any one of claims 1 to 4, wherein the radar is a millimeter wave radar. A step in which the radar transmits radar waves to the surrounding area and receives reflected waves reflected by the target object to obtain a radar signal;
A step in which the camera images the surrounding area and obtains image information;
extracting contour information of image information output from the camera;
When searching for the highest intensity point in each direction for the radar signal output from the radar and plotting the radar signal on the image information output from the camera, each radar signal has a height of zero at each reflection distance. By plotting the highest intensity point in each direction of the radar signal at the point on the image information where the target is in contact with the road surface, it is possible to calculate the reflected signal from the camera. detecting the nearest reflection point at;
Converting the nearest reflection point in each direction from polar coordinate system data (r, θ) to orthogonal coordinate system data (X, Y), and adding distance information in each direction to contour information of an image in the same direction;
Calculate the image resolution (m/pix) at the distance to the contour component of the image in each calculated direction, calculate height information using the resolution information, and add height information to the contour information of the image information. The process of
estimating three-dimensional information according to the contour information, the distance information, and the height information;
A three-dimensional information estimation method characterized by comprising: Computer,
a target detection means for searching for the maximum intensity point in each direction for a radar signal output from a radar which transmits radar waves into the surroundings and receives the reflected waves reflected by an object to obtain a radar signal , and for plotting the radar signal on image information output from a camera which captures an image of the surroundings to obtain image information, said target detection means assumes that each radar signal is a reflected signal from a height of zero at each reflection distance, and plots the maximum intensity point in each direction of the radar signal at a point on the image information where the target is in contact with the road surface, thereby detecting the nearest reflection point on the image information output from said camera;
distance information adding means for converting the nearest reflection point in each direction from polar coordinate system data (r, θ) to orthogonal coordinate system data (X, Y) and adding the distance information of each direction to the contour information of the image in the same direction;
a height information adding means for calculating an image resolution (m/pix) at the calculated distance to the contour component of the image in each direction, calculating height information using the resolution information, and adding the height information to the contour information of the image information;
a three-dimensional information estimating means for estimating three-dimensional information according to the contour information, the distance information and the height information;
A program that a computer can execute to make the device function as a processor.

Images