JP2021135061A - Three-dimensional information estimation system, three-dimensional information estimation method, and computer-executable program - Google Patents

Abstract

To estimate three-dimensional information of a structure with a low-cost structure.SOLUTION: A three-dimensional information estimation system includes: a radar that transmits a radar wave to surroundings and receives a reflection wave reflected by an object to acquire a radar signal; a camera that images the surroundings and acquires image information; contour information extraction means for extracting contour information of the image information output from the camera; target detection means for searching a maximum intensity point in each direction for the radar signal output from the radar and detecting nearest neighbor reflection point on the image information output from the camera; distance information giving means for giving distance information of the nearest neighbor reflection point in each direction to the contour information of the image information in the same direction; height information giving means for giving height information to the contour information of the image information; and three-dimensional information estimation means for estimating three-dimensional information according to the contour information, the distance information, and the height information.SELECTED DRAWING: Figure 1

Description

The present invention relates to a three-dimensional information estimation system, a three-dimensional information estimation method, and a computer-executable program.

Conventionally, in the estimation of three-dimensional information, a technique of using the parallax of the left and right images with a stereo camera alone and a technique of measuring using LiDAR (Larser Imaging Detection and Ranking) have been proposed. When the above-mentioned stereo camera or LiDAR is used, the cost of the system becomes high because the sensor itself is expensive.

On the other hand, recently, research on surrounding environment recognition technology using millimeter-wave radar and monocular camera has been actively conducted, but many researches target only dynamics, and accurately recognize 3D information of structures. No method has been proposed. If three-dimensional information of a structure can be estimated using an inexpensive radar and a monocular camera, it will be advantageous in terms of cost.

Japanese Unexamined Patent Publication No. 2010-286926

The present invention has been made in view of the above, and can be executed by a three-dimensional information estimation system, a three-dimensional information estimation method, and a computer capable of estimating three-dimensional information of a structure with a low-cost configuration. The purpose is to provide a good program.

In order to solve the above-mentioned problems and achieve the object, the present invention transmits a radar wave to the periphery, receives the reflected wave reflected by the object and acquires a radar signal, and images the periphery. The highest intensity point in each direction is searched for the camera that acquires the image information, the contour information extracting means that extracts the contour information of the image information output from the camera, and the radar signal output from the radar. , The target detecting means for detecting the nearest reflection point on the image information output from the camera, and the distance information for adding the distance information of the nearest reflection point in each direction to the contour information of the image information in the same direction. The giving means, the height information adding unit that gives height information to the contour information of the image information, and the three-dimensional information estimating means that estimates three-dimensional information according to the contour information, the distance information, and the height information. It is characterized by having.

Further, according to one aspect of the present invention, the camera may be a monocular camera.

Further, according to one aspect of the present invention, the distance information imparting means assumes that the distance information between adjacent directional bins changes linearly with respect to the nearest reflection point, and the distance in the insufficient directional direction. The information may be interpolated.

Further, according to one aspect of the present invention, the target detection means searches each direction for the radar signal output from the radar, determines the maximum intensity point in each direction, and outputs the radar signal from the camera. The highest intensity points in each direction are plotted on the image information, the brightness characteristic values in the vicinity of the highest intensity points on the image information are calculated, and each of the brightness characteristic values on the image information is calculated based on the calculated brightness characteristic values. When it is determined whether the highest intensity point is a real image or a false image and it is determined to be a real image, the highest intensity point is set as the nearest reflection point, and when it is determined to be a false image, the intensity is next to the direction. The process may be repeated until a high signal point is acquired and a real image is determined, and when the target signal disappears, it may be determined that there is no real image in the direction.

Further, according to one aspect of the present invention, the radar may be a millimeter wave radar.

Further, in order to solve the above-mentioned problems and achieve the object, the present invention is a step in which a radar transmits a radar wave to the periphery, receives a reflected wave reflected by an object, and acquires a radar signal. The maximum intensity of each direction with respect to the process in which the camera captures the surroundings to acquire image information, the process in which the contour information of the image information output from the camera is extracted, and the radar signal output from the radar. A step of searching for a point to detect the nearest reflection point on the image information output from the camera and adding distance information of the nearest reflection point in each direction to the contour information of the image information in the same direction. It is characterized by including a step, a step of adding height information to the contour information of the image information, and a step of estimating three-dimensional information according to the contour information, the distance information, and the height information.

Further, in order to solve the above-mentioned problems and achieve the object, the present invention transmits a radar wave to the periphery of the computer, receives the reflected wave reflected by the object, and acquires a radar signal. The target detection that detects the nearest reflection point on the image information output from the camera that searches the highest intensity point in each direction for the radar signal output from and acquires image information by imaging the periphery. Means, contour information extracting means for extracting contour information of image information output from the camera, and contour information giving means for imparting distance information of the nearest reflection point in each direction to the contour information of the image information in the same direction. As a height information giving means for giving height information to the contour information of the image information, and a three-dimensional information estimating means for estimating three-dimensional information according to the contour information, the distance information, and the height information. It is characterized in that the computer for functioning is an executable program.

According to the present invention, it is possible to estimate the three-dimensional information of the structure with a low-cost configuration.

FIG. 1 is a diagram showing a schematic configuration example of a three-dimensional information estimation system according to the present embodiment. FIG. 2 is a diagram showing an example of a flow showing a schematic operation of the three-dimensional information system according to the present embodiment. FIG. 3 is a diagram for explaining a method of calculating the resolution. FIG. 4 is a diagram showing an example in which the reflection point and the luminance dispersion value in the nearest vicinity of the radar signal are plotted on the image captured by the camera. FIG. 5 is a diagram for explaining signals used for analysis and signals to be excluded. FIG. 6 is a diagram for explaining the association between the contour information of the image and the distance information. FIG. 7 is a diagram showing an example of the created three-dimensional map (first angle). FIG. 8 is a diagram showing an example of the created three-dimensional map (second angle). FIG. 9 is a diagram showing an example of the created three-dimensional map (third angle). FIG. 10 is a diagram showing an example of the created three-dimensional map (fourth angle).

Hereinafter, a three-dimensional information estimation system according to the present invention, a three-dimensional information estimation method, and an embodiment of a computer-executable program will be described in detail with reference to the drawings. The present invention is not limited to this embodiment. In addition, the components in the following embodiments include those that can be easily assumed by those skilled in the art or those that are substantially the same. Although the components of the present invention are generally shown in the drawings of the present specification, it is easy to understand that they may be designed in a wide variety of arrangements in various configurations. Therefore, the following more detailed description of embodiments of the system and apparatus of the present invention does not limit the scope of the invention as shown in the claims, but merely provides an example of selected embodiments of the present invention. It is a thing. The present specification incorporates known techniques by reference. Therefore, those skilled in the art can understand that the present invention can be realized without one or more specific details, or with other methods, parts, and materials by using known techniques.

[1. Principle of the present invention]
In the present invention, in order to estimate the three-dimensional information of the structure with a low-cost configuration, a radar and a monocular camera are used, and the distance / orientation information obtained by the radar and the orientation / orientation obtained from the image captured by the monocular camera are used. We propose a method for estimating the three-dimensional information of surrounding structures using height information.

When extracting radar signals, compared to dynamic pedestrian and vehicle signals, radar signals of static structures are more susceptible to noise components such as clutter and false images, and radar signal selection can be performed. It will be difficult. Therefore, in the present invention, as a method for extracting a desired structure signal, the clutter component of the radar signal is suppressed by using image information.

Specifically, the radar signals in each of the detected directions are plotted on the image by utilizing the property that it is unlikely that unnecessary components are reflected on the image at the same position as the false image position detected by the radar signal. By doing so, it is determined whether or not each signal is a signal from a target. More specifically, when plotting radar signals on an captured image, it is assumed that each radar signal is a reflected signal from a height of zero at each reflection distance, and the target is in contact with the road surface. Plot the radar signal at a point on the image. The luminance dispersion value of the image is acquired in the vicinity of each radar signal plotted on the image, and whether each radar signal is a real image or a false image is classified from the acquired luminance variance value. When the radar signal is a real image, the brightness of the switching portion between the target and the road surface is evaluated on the image, so that the brightness dispersion value tends to be large. On the other hand, even if a false radar signal is plotted, the probability of evaluating the brightness of a road surface portion that is empty on the image increases, so that the brightness dispersion value tends to be small. In this way, by evaluating the luminance dispersion value on the image associated with each radar signal, it is possible to distinguish between the real image and the false image. In this way, the distance to the nearest object can be accurately obtained for each direction.

At this time, the signal is not detected (excluded) for the direction in which the object does not exist. The three-dimensional information of the structure in each direction is calculated by using the distance information of the direction in which the signal is detected.

As described above, in the present invention, since the image has a characteristic of having orientation information on the horizontal axis, attention is paid to the point that if the horizontal coordinates are the same, it can be regarded as an object having the same orientation. The direction information of the image and the direction information of the radar are integrated, and the signal distance of the direction in which the signal is detected by the radar is linked to the contour information of the image of the same direction. This makes it possible to add distance information to image information in a certain direction.

However, since the horizontal angle resolution of the radar is very low with respect to the horizontal angle resolution of the image, it is necessary to estimate the distance information of the direction corresponding to each direction bin of the radar. Therefore, it is assumed that the distance information between adjacent directional bins changes linearly, and the distance information of the insufficient directional bins is interpolated.

Distance information is added to each pixel component on the horizontal axis (ground) of the image, and the resolution is calculated for each distance corresponding to the orientation. By using the resolution obtained here, the height information of each contour component in each direction is calculated. By applying the above method, it is possible to estimate the three-dimensional information of the surrounding structure.

Further, when the technique of estimating the three-dimensional information of the peripheral structure of the present embodiment is applied to the system in which the dynamics are detected by the radar and the monocular camera, the static information is added. There is a possibility that even the difficult self-position estimation can be realized. With the development of GNSS represented by GPS, it has become possible to recognize with high accuracy if it is only the self-position, but it is not possible to derive the absolute direction instantaneously. If the positional relationship with the surrounding structures can be accurately grasped, the absolute orientation of the observer can be recognized by a single measurement.

[2. Configuration example and operation example of the three-dimensional information estimation system of this embodiment]
A configuration example and an operation example of the three-dimensional information estimation system of the present embodiment will be described with reference to FIGS. 1 to 3. FIG. 1 is a diagram showing a schematic configuration example of the three-dimensional information estimation system 1 according to the present embodiment. FIG. 2 is a diagram showing an example of a flow showing a schematic operation of the three-dimensional information estimation system 1 according to the present embodiment.

The configuration of the three-dimensional information estimation system 1 according to the present embodiment will be described with reference to FIG. As shown in FIG. 1, the three-dimensional information estimation system 1 according to the present embodiment includes a three-dimensional information estimation device 10, a radar 20, and a camera 30.

The radar 20 transmits a radar wave to the surroundings, receives the reflected wave reflected by the object, and outputs the radar signal to the three-dimensional information estimation device 10. The radar 20 is, for example, an FM-CW type millimeter-wave radar (79 GHz band, bandwidth 4 GHz, MIMO radar (3 transmitting antennas, 6 receiving antennas), and the detection range has a horizontal angle of −75 degrees to 75 degrees. The degree and elevation angle are -3 degrees to 3 degrees. The radar 20 transmits radar waves to the surroundings from a transmitter having a plurality of transmitting antennas, and receives reflected waves reflected by an object with a plurality of receiving antennas. It is a sensor for detecting the distance, relative speed, angle, and signal strength of an object by receiving it with a device.

The radar signal output from the radar 20 may be, for example, a signal consisting of points or a sequence of points indicating one or a plurality of representative positions of an object whose reflected wave has been processed in the signal processing circuit included in the radar 20. Alternatively, it may be a signal indicating an unprocessed reflected wave. In this example, the radar signal output from the radar 20 includes information on the distance, orientation, relative velocity, and signal strength of the object calculated by the known FM-CW method in the above-mentioned signal processing circuit. And. When the unprocessed received wave is used as a radar signal, the three-dimensional information estimation device 10 executes these signal processes.

The camera 30 outputs the image information obtained by capturing the surroundings to the three-dimensional information estimation device 10. The camera 30 is a sensor for receiving visible light or infrared light and outputting it as image information which is external information of an object. The camera 30 is, for example, a monocular camera, for example, a CMOS global shutter sensor (for example, the number of effective pixels: 1920 pixels × 1200 pixels, a frame rate: 41.6 fps, an interface: USB3.0) and a megapixel compatible CCTV lens (for example). For example, it can be configured by a combination of an angle of view: 86.71 degrees × 61.44 degrees, a focal length of 6 mm). It is -30.72 degrees to 30.72 degrees.

The image information output from the camera 30 is composed of a plurality of frame images that are continuous in time series, and each frame image is represented by pixel data. The pixel data is monochrome pixel data or color pixel data.

The three-dimensional information estimation device 10 includes a target detection unit 11, a contour extraction unit 12, a distance information addition unit 13, a height information addition unit 14, and a three-dimensional information estimation unit 15.

The target detection unit 11 searches for the highest intensity point in each direction with respect to the radar signal output from the radar 20, and detects the nearest reflection point on the image information output from the camera 30. The target detection unit 11 does not detect dynamic components (pedestrians, etc.) as the nearest reflection points in each direction.

The target detection unit 11 includes, for example, a threshold value processing unit 40, a peak search unit 41, a plot unit 42, a luminance characteristic value calculation unit 43, and a real image / false image determination unit 44.

The threshold value processing unit 40 performs threshold value processing such as CFAR on the radar signal output from the radar 20. The peak search unit 41 determines the highest intensity point in each direction for the radar signal after the threshold processing. When the real image / false image determination unit 44 determines that the highest intensity point is a false image, the signal point having the next highest intensity for the direction is acquired.

The plotting unit 42 plots the highest intensity points in each direction of the radar signal on the image output from the camera 30. When plotting the highest intensity points on the image, it is assumed that the radar signal is a reflected signal from zero height at each reflection distance (because the radar's elevation angle is small), and the target is in contact with the road surface. Calculate the corresponding coordinates on the image and plot the radar signal points (highest intensity points). When the peak search unit 41 acquires the signal point having the next highest intensity in the direction, the plotting unit 42 plots the signal point having the next highest intensity in the direction.

The luminance characteristic value calculation unit 43 calculates the luminance characteristic value (for example, the luminance dispersion value, the luminance gradient) in the vicinity of each highest intensity point on the image. The luminance characteristic value calculation unit 43 calculates the luminance characteristic value in the vicinity of the signal point having the next highest intensity when the signal point having the next highest intensity in the direction is plotted by the plotting unit 42.

The real image / false image determination unit 44 determines whether the highest intensity point on the image is a real image or a false image based on the luminance characteristic value, and outputs the result of the real image. Specifically, when the real image / false image determination unit 44 determines that the real image is a real image, the highest intensity point is set as the nearest reflection point, and when it is determined that the image is a false image, the intensity is next to the direction. A high signal point is determined, and the process is repeatedly executed until it is determined to be a real image. However, when the target signal disappears, it is judged that there is no real image of the relevant direction.
For example, when the luminance characteristic value is equal to or more than the threshold value, it is determined that the highest intensity point is a real image, and when the luminance characteristic value is not equal to or more than the threshold value, it is determined that the highest intensity point is a false image. be able to.

The contour extraction unit 12 extracts the contour information of the image information output from the camera 30 and outputs it to the distance information addition unit 13 and the height information addition unit 14.

The distance information adding unit 13 adds the distance information of each direction to the contour information of the image of the same direction. The distance information imparting unit 13 may interpolate the distance information of the insufficient direction on the assumption that the distance information between the adjacent direction bins changes linearly with respect to the nearest reflection point.

The height information adding unit 14 adds height information (Z) to the contour information of the image.

The three-dimensional information estimation unit 15 estimates (draws) the three-dimensional information by three-dimensionally plotting the contour information of the image according to the calculated distance information (X, Y) and height information (Z), and the three-dimensional map. Is output.

The three-dimensional information estimation device 10 is equipped with a CPU (processor), a DSP, a RAM, a non-volatile memory such as a ROM or a flash memory, an I / O, a monitor, and the like. Each function of the three-dimensional information estimation device 10 is realized by the CPU executing a program stored in a non-volatile memory such as a ROM or a flash memory. When this program is executed, the method corresponding to the program is executed.

When the CPU executes the program, the functions of the target detection unit 11, the contour extraction unit 12, the distance information addition unit 13, the height information addition unit 14, and the three-dimensional information estimation unit 15 are realized.

The method for realizing these elements constituting the three-dimensional information estimation device 10 is not limited to software, and some or all of the elements are realized by using hardware that combines logic circuits, analog circuits, and the like. You may.

The entire process of the three-dimensional information estimation system 1 of FIG. 1 will be described with reference to FIG.

In FIG. 2, the radar 20 transmits a radar wave to the surroundings, receives the reflected wave, acquires a radar signal, and outputs the radar signal to the three-dimensional information estimation device 10 (step S1). The camera 30 takes an image of the surroundings, acquires two-dimensional image information, and outputs the two-dimensional image information to the three-dimensional information estimation device 10 (step S2). It is assumed that the radar signal and the frame of the image information are synchronized.

The target detection unit 11 detects the nearest reflection point on the image in each direction of the radar signal based on the image information output from the camera 30 and the radar signal output from the radar 20 (step S3). The target detection unit 11 does not detect dynamic components (pedestrians, etc.) as the nearest reflection points in each direction. Specifically, the target detection unit 11 searches each direction for the radar signal output from the radar 20, determines the maximum intensity point in each direction, and displays each on the image information output from the camera 30. Plot the highest intensity points in the orientation, calculate the brightness characteristic value in the vicinity of each highest intensity point on the image information, and based on the calculated brightness characteristic value, each highest intensity point on the image information is a real image or a false image. If it is determined that the image is a real image, the highest intensity point is set as the nearest reflection point, and if it is determined to be a false image, the signal point with the next highest intensity in the direction is acquired and the real image is obtained. The process is repeatedly executed until it is determined that the image is (until the highest intensity point in each direction is determined to be a real image (until the nearest reflection point is acquired)). However, when the target signal disappears, it is judged that there is no real image of the relevant direction.

The contour extraction unit 12 extracts the contour information of the image information output from the camera 30 (step S4). The contour extraction unit 12 does not extract contour information for dynamic components (pedestrians and the like). Since the dynamic components in the image can be detected by a known method, the description thereof will be omitted.

The distance information adding unit 13 adds the distance information of each direction to the contour information of the image of the same direction (step S5). Specifically, for example, the distance information adding unit 13 converts the nearest reflection point of each direction from polar coordinate system data (r, θ) to Cartesian coordinate system data (X, Y), and outputs a signal in the adjacent direction bin. After confirming and interpolating between adjacent signals, distance information in each direction is added to the contour information of the image in the same direction. Here, the reason why the adjacent signals are interpolated is that the horizontal angle resolution of the radar is very coarse with respect to the horizontal angle resolution of the image, so the distance information of the direction between each direction bin of the radar signal is estimated. Need arises. Therefore, it is assumed that the distance information between adjacent directional bins changes linearly, and the distance information of the insufficient directional bins is interpolated.

It should be noted that the distance measurement error may be suppressed by smoothing the distance to each target by adding the distance information of the same direction in the past frame (for example, the past 5 frames).

The height information adding unit 14 adds height information (Z) to the contour information of the image (step S6). Here, an example of the calculation method of the height information (Z) will be described. Since the distance to the contour component of the image in each direction has been calculated, the resolution (m / pix) of the image at the calculated distance is calculated, and the height information is calculated using the resolution information. FIG. 3 is a diagram for explaining a method of calculating the resolution.

(1) The vertical resolution of the image at a distance Y (m) is obtained by the following equation (1).
Vertical resolution vReso (m / pix) = Y * 7.13 / 6/1200
... (1)
Here, 7.13 (mm) is the CMOS sensor size, 6 (mm) is the focal length of the CCTV lens, and 1200 (pix) is the number of pixels.

(2) The road surface coordinates on the image at the road surface height at a distance Y (m) are obtained by the following equation (2).
Road surface coordinates rCol (pix) = ColCross + (CamHgt / vReso)
... (2)
Here, ColCross (pix) is the vanishing point coordinates of the vertical component, and CamHgt (m) is the camera installation height.

The vanishing point is the point where the lines intersect when drawing what is actually parallel lines in perspective in a non-parallel manner. For example, an originally parallel passage also converges at a certain point on the image. The vanishing point refers to this converging point. When calibrated in advance so that the vanishing point and the center of the image match, the equation of vanishing point = coordinates of the camera installation height at an infinity distance holds. From this premise, by subtracting the camera installation height (pix) at each distance from the vanishing point coordinates, the road surface coordinates rCol (pix) at the corresponding distance can be obtained by the equation (2).

(3) The height to each contour component is obtained by the following equation (3).
Point of interest Height Z (m) = (TgtCol (pix) -rCol (pix)) * vReso
... (3)
Here, TgtCol (pix) is the vertical component coordinates of the pixel whose height is to be calculated.

Returning to FIG. 2, the three-dimensional information estimation unit 15 estimates the three-dimensional information by plotting three-dimensionally according to the contour information, the calculated distance information, and the height information, and creates a three-dimensional map (step S7).

[3. Verification example of the three-dimensional information estimation device of this embodiment (Example)]
A verification example (example) of the three-dimensional information estimation system 1 of FIG. 1 of the present embodiment will be described with reference to FIGS. 4 to 10. An example of creating a three-dimensional map in the surrounding environment as shown in FIG. 4 will be described.

In FIG. 4, the figure on the left side is a diagram showing an example in which the nearest neighbor reflection point and the brightness dispersion value are plotted on the image captured by the camera 30, and the figure on the right side is a diagram for explaining a moving object in the image. Is. In FIG. 4, the polygonal line indicates the luminance dispersion value, the vertical dotted line indicates the direction of the radar 20, and □ indicates the nearest neighbor reflection point. The nearest neighbor reflection point is detected according to step S3 of the flow of FIG. 2 described above.

In FIG. 4, in the direction in which a moving object such as a pedestrian exists, the signal from the target is greatly affected. ) Create a 3D map.

FIG. 5 is a diagram for explaining signals used for analysis and signals to be excluded. In the figure, structures are used for analysis and pedestrians are excluded. The distances and directions to all points of the signal used for analysis are calculated, the contour information of the image is extracted according to steps S4 and S5 of the flow of FIG. 2 described above, and the distance information of each direction is used as the contour information of the image of the same direction. Give to.

FIG. 6 is a diagram for explaining the association between the contour information of the image and the distance information. In FIG. 6, since the horizontal axis of the image is the direction, it is assumed that all the contour pixel groups in a vertically arranged row are the components of the distance in the corresponding direction. Distance information is linked to each vertical axis of the contour information of the image. Although the moving object target (pedestrian) is shown in FIG. 6, this is shown for reference only, and the contour information of the moving object target (pedestrian) is not actually extracted.

The height information of the contour information of the image is calculated according to step S6 of the flow of FIG. 2 described above. Then, according to step S7 of the flow of FIG. 2 described above, a three-dimensional map is created by three-dimensional plotting according to the contour information, the calculated distance and the height. 7 to 10 are views showing the created three-dimensional map at each angle. The short-distance component and the long-distance component may be distinguished and displayed (for example, the short-distance component may be displayed in red and the long-distance component may be displayed in green). Although the moving object target (pedestrian) is displayed three-dimensionally in FIGS. 7 to 10, this is shown for reference only, and the moving object target (pedestrian) is not actually drawn. As shown in FIGS. 7 to 10, by using the information of the highest intensity point detected in each direction of the radar signal and adding the estimation information (interpolation information) that the object distance changes linearly, It was possible to estimate the three-dimensional information of stationary objects such as buildings and warehouses.

Based on the above verification results, the 3D information estimation system 1 of the present embodiment can estimate 3D information of surrounding structures by a combination of an inexpensive monocular camera and a radar, and is useful for realizing recognition of the surrounding environment. It was confirmed that there was.

As described above, according to the three-dimensional information estimation system 1 of the present embodiment, the radar 20 transmits a radar wave to the periphery, receives the reflected wave reflected by the object, and acquires a radar signal. A monocular camera 30 that captures an image of the surroundings to acquire image information, a contour extraction unit 12 that extracts contour information of image information output from the camera 30, and a radar signal output from the radar 20 in each direction. The target detection unit 11 that searches for the highest intensity point and detects the nearest reflection point on the image information output from the camera 30, and the distance information of the nearest reflection point in each direction is the image information in the same direction. The distance information adding unit 13 given to the contour information of the image information, the height information adding unit 14 giving height information to the contour information of the image information, and the three-dimensional information estimation according to the contour information, the distance information, and the height information 3 Since the dimensional information estimation unit 15 is provided, it is possible to estimate the three-dimensional information of the structure with a low-cost configuration.

The three-dimensional information estimation system 1 of the present embodiment can be suitably used in various applications such as an in-vehicle radar system, an advanced driver assistance system (ADAS), and a peripheral monitoring system.

1 3D information estimation system 10 3D information estimation device 11 Target detection unit 12 Contour extraction unit 13 Distance information addition unit 14 Height information addition unit 15 3D information estimation unit (3D map generation unit)
20 Radar 30 Camera 41 Peak search unit 42 Plot unit 43 Luminance characteristic value calculation unit 44 Real image / false image determination unit

Claims (7)

A radar that transmits radar waves to the surroundings, receives reflected waves reflected by an object, and acquires radar signals.
A camera that captures the surroundings and acquires image information,
A contour information extraction means for extracting contour information of image information output from the camera, and
A target detection means that searches for the highest intensity point in each direction of the radar signal output from the radar and detects the nearest reflection point on the image information output from the camera.
A distance information imparting means for imparting distance information of the nearest reflection point in each direction to the contour information of the image information in the same direction, and
A height information adding unit that adds height information to the contour information of the image information,
A three-dimensional information estimation means that estimates three-dimensional information according to the contour information, the distance information, and the height information.
A three-dimensional information estimation system characterized by being equipped with. The three-dimensional information estimation system according to claim 1, wherein the camera is a monocular camera. The claim is characterized in that the distance information imparting means interpolates the distance information of insufficient directions on the assumption that the distance information between adjacent direction bins changes linearly with respect to the nearest reflection point. 1 or the three-dimensional information estimation system according to claim 2. The target detection means is
With respect to the radar signal output from the radar, each direction is searched to determine the highest intensity point in each direction.
The highest intensity points in each direction are plotted on the image information output from the camera, and the luminance characteristic value in the vicinity of each highest intensity point on the image information is calculated.
Based on the calculated luminance characteristic value, it is determined whether each maximum intensity point on the image information is a real image or a false image, and if it is determined to be a real image, the maximum intensity point is set as the nearest neighbor reflection point.
If it is determined to be a false image, the signal point with the next highest intensity after the relevant direction is acquired, the process is repeated until it is determined to be a real image, and if the target signal disappears, it is determined that there is no real image in the relevant direction. The three-dimensional information estimation system according to any one of claims 1 to 3, wherein the three-dimensional information estimation system is characterized. The three-dimensional information estimation system according to any one of claims 1 to 4, wherein the radar is a millimeter-wave radar. The process in which the radar transmits radar waves to the surroundings, receives the reflected waves reflected by the object, and acquires the radar signal.
The process by which the camera captures the surroundings and acquires image information,
The process of extracting the contour information of the image information output from the camera and
A process of searching for the highest intensity point in each direction of the radar signal output from the radar and detecting the nearest reflection point on the image information output from the camera.
A step of adding the distance information of the nearest reflection point in each direction to the contour information of the image information in the same direction, and
The process of adding height information to the contour information of the image information and
A step of estimating three-dimensional information according to the contour information, the distance information, and the height information, and
A three-dimensional information estimation method characterized by including. Computer,
A radar wave is transmitted to the periphery, and the reflected wave reflected by the object is received to acquire the radar signal. For the radar signal output from the radar, the highest intensity point in each direction is searched and the periphery is imaged. A target detection means that detects the nearest reflection point on the image information output from the camera that acquires the image information.
A contour information extraction means for extracting contour information of image information output from the camera, and
A contour information imparting means for imparting distance information of the nearest reflection point in each direction to the contour information of the image information in the same direction, and
A height information imparting means for imparting height information to the contour information of the image information, and
A three-dimensional information estimation means that estimates three-dimensional information according to the contour information, the distance information, and the height information.
A computer-executable program to make it work.

Images