KR102557620B1 - Radar apparatus and method for classifying object - Google Patents

Abstract

A radar device mounted on a vehicle may include a transmitter/receiver unit that transmits a radar signal to the outside of the vehicle and receives a radar signal reflected by an object, a signal processor that processes the reflected radar signal to detect an object, a fusion data generator that generates fusion data based on radar data and camera data, and a classification unit that classifies objects sensed through an artificial intelligence module learned based on the generated fusion data.

Description

Radar device and method for classifying objects {RADAR APPARATUS AND METHOD FOR CLASSIFYING OBJECT}

The present invention relates to a radar apparatus and method for classifying objects.

The vehicle radar serves to provide the driver with location information of surrounding vehicles and obstacles existing in front, side, and rear of the vehicle.

These vehicle radars have high distance accuracy in detecting objects and provide stable object detection performance because they are not affected by vehicle driving conditions (eg, bad weather, night conditions, etc.). In addition, since the vehicle radar can measure the speed of an object, it can distinguish whether the object is moving or stationary.

However, it is difficult to estimate and classify the shape of an object using a vehicle radar because the vehicle radar has lower lateral resolution compared to a vehicle camera or lidar. In addition, even in the annotation process for securing the ground truth used for artificial intelligence learning of the vehicle radar, it is difficult for a person to judge an object based only on the measurement result of the object detected by the vehicle radar (e.g., the position, speed, etc.) of the vehicle. Therefore, it is difficult to classify objects detected by the vehicle radar without additional information from other sensors.

On the other hand, the vehicle camera shows higher resolution performance in the lateral direction than the vehicle radar and has good resolution technology, so it is useful for recognizing and classifying objects.

However, a vehicle camera can measure a distance to an object using stereo vision, etc., but has poor distance accuracy and difficulty in long-distance detection. In addition, since the vehicle camera is affected by the driving conditions of the vehicle, object detection performance is deteriorated in bad weather or at night.

Korean Patent Publication No. 2020-0132137 (published on November 25, 2020)

The present invention is to solve the above-mentioned problems of the prior art, to generate fusion data based on radar data and camera data, and to classify objects detected through an artificial intelligence module learned based on the generated fusion data.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problems described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, a radar device mounted on a vehicle according to a first aspect of the present invention includes a transceiver for transmitting a radar signal to the outside of the vehicle and receiving a radar signal reflected by an object; a signal processor processing the reflected radar signal to detect the object; a fusion data generating unit generating fusion data based on radar data and camera data; and a classification unit configured to classify the sensed object through an artificial intelligence module learned based on the generated fusion data.

A method for classifying an object by a radar device mounted on a vehicle according to a second aspect of the present invention includes transmitting a radar signal to the outside of the vehicle; receiving a radar signal reflected by an object; processing the reflected radar signal to detect the object; generating fusion data based on radar data and camera data; and classifying the sensed object through an artificial intelligence module learned based on the generated fusion data.

The above-described means for solving the problems is only illustrative and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, the present invention generates fusion data based on radar data and camera data, and classifies objects detected through the learned artificial intelligence module based on the generated fusion data.

In addition, the present invention can improve the performance of object detection and classification through the learned artificial intelligence module by enabling real-time learning between artificial intelligence modules using fusion data.

In addition, the present invention can generate fusion data through radar data showing stable object detection performance even in situations such as bad weather and night situations, and improve the object detection performance of the camera artificial intelligence module based on the fusion data.

In addition, the present invention can stably detect and classify objects with a radar device through an artificial intelligence module learned based on fusion data even in situations where it is difficult to identify a camera, such as in bad weather or at night.

In addition, the present invention can classify an object detected by a radar device as a radar device without a camera later through an artificial intelligence module learned based on fusion data.

1 is a block diagram of a radar device according to an embodiment of the present invention.
2A to 2C are diagrams for explaining data used to generate fusion data according to an embodiment of the present invention.
3 is a diagram for explaining a process of generating fusion data according to an embodiment of the present invention.
4 is a diagram for explaining a method of learning an artificial intelligence module according to an embodiment of the present invention.
5 is a diagram for explaining a method of generating fusion data according to a driving situation while driving a vehicle according to an embodiment of the present invention.
6 is a diagram for explaining a method of generating fusion data according to performance of a radar artificial intelligence module according to an embodiment of the present invention.
7 is a flowchart illustrating a method of classifying an object through a radar device mounted on a vehicle according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be “connected” to another part, this includes not only the case where it is “directly connected” but also the case where it is “electrically connected” with another element interposed therebetween. In addition, when a certain component is said to "include", this means that it may further include other components without excluding other components unless otherwise stated.

In this specification, a "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware.

In this specification, some of the operations or functions described as being performed by a terminal or device may be performed instead by a server connected to the terminal or device. Likewise, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the corresponding server.

Hereinafter, specific details for the implementation of the present invention will be described with reference to the accompanying configuration diagram or process flow chart.

1 is a block diagram of a radar device 10 according to an embodiment of the present invention.

Referring to FIG. 1 , the radar device 10 may include a transceiver 100, a signal processor 110, a fusion data generator 120, a classifier 130, a learner 140, and an autonomous driving unit 150. However, the radar device 10 shown in FIG. 1 is only one implementation example of the present invention, and various modifications are possible based on the components shown in FIG. 1 .

The transceiver 100 may transmit a radar signal to the outside of the vehicle and receive a radar signal reflected by an object. For example, the objects may include obstacles, moving objects, pedestrians, and the like located in front, side, and rear of the vehicle.

The signal processing unit 110 may process the reflected radar signal to detect an object. At this time, the reflected radar signal is a signal received while the vehicle equipped with the radar device 10 is driving.

Specifically, the signal processing unit 110 may perform a signal processing process on the reflected radar signal to detect an object. For example, referring to FIG. 2A , the signal processor 110 may extract a plurality of point cloud data 201 (small boxes) constituting at least one object based on the reflected radar signal, and derive a detection result value for the object using the extracted plurality of point cloud data 201. Here, the detection result value for the object may include position information of the object derived based on the plurality of point cloud data 201, speed information, and angle information between the object and the vehicle (vehicle equipped with the radar device 10).

The signal processing unit 110 may perform object recognition based on the object detection result. The signal processor 110 may determine whether or not the sensed object is a real object based on the object detection result value, and derive an object recognition result value based on the determination result. For example, referring to FIG. 2A , after clustering a plurality of point cloud data 201, the signal processing unit 110 can estimate the clustered point cloud set 203 (large square box) as a real object, which can be regarded as an object recognition result value. In addition, the signal processing unit 110 may track the clustered point cloud set and derive the tracking result as an object recognition result value. In addition, the signal processing unit 110 may derive the corrected result as an object recognition result value by correcting the moving speed of the real object corresponding to the clustered point cloud set according to whether the vehicle is moving.

The signal processor 110 may generate radar data based on a fast Fourier transform value of the reflected radar signal, a result value of detecting an object, and a result value of object recognition.

An image processing unit (not shown) may perform object detection and object classification based on image data received from a camera installed in a vehicle. For example, referring to FIG. 2B , an image processing unit (not shown) inputs image data to the camera artificial intelligence module 20, and then derives camera data including object detection result values derived through the camera artificial intelligence module 20 (e.g., location information of the object) and classification result values (e.g., vehicle type information of the object). At this time, the image processing unit (not shown) may search for the location of the recognized object by recognizing the object from the image data through the camera artificial intelligence module 20 and then setting a bounding box on the recognized object. In addition, the image processing unit (not shown) may determine to which category among a plurality of categories (eg, truck type, bus type, car type, etc.) the object (e.g., vehicle) recognized through the camera artificial intelligence module 20 belongs, and then classify the recognized object into the category to which the object belongs.

The fusion data generator 120 may generate fusion data based on radar data and camera data.

Here, the radar data may include data that can be derived from the radar signal, such as a fast Fourier transform value of a radar signal reflected from an object, a detection result value of an object, and a recognition result value of an object. Here, the camera data is derived through the camera artificial intelligence module, and may include a detection result value for an object and a classification result value for an object included in image data generated by a camera mounted on a vehicle.

The fusion data generation unit 120 may generate fusion data by projecting camera data into a radar coordinate system for the radar device 10 and matching the projected camera data and radar data to the radar coordinate system for each target. For example, referring to FIG. 2C , the fusion data generation unit 120 converts the location information of the object included in the camera data into a radar coordinate system for the radar device 10 mounted on the vehicle 205 (eg, x, y, z coordinate system centered on the vehicle 205), and may check whether the coordinates of the object included in the camera data converted to the radar coordinate system and the coordinates of the object included in the radar data are at similar positions. In this case, the conversion to the radar coordinate system may be performed to compensate for the position error when there is a position error between the camera data and the radar data. Subsequently, when the coordinates of the object included in the camera data converted to the radar coordinate system and the coordinates of the object included in the radar data are at similar positions, the fusion data generator 120 may recognize the object included in the camera data and the object included in the radar data as the same object and match them to each other.

On the other hand, when the position error between the camera data and the radar data is less than the threshold value, the fusion data generator 120 does not project the camera data into the radar coordinate system for the radar device 10, and matches the camera data and radar data for each target to generate fusion data.

Fusion data can be used to obtain highly reliable classification results through mutual matching between camera data and radar data and statistical accumulation results over time.

The advantage of the radar device 10 is that it derives high distance and speed accuracy and fast detection results, and the advantage of the camera is that it can distinguish high lateral resolution and vehicle type (eg, truck, bus, passenger car, motorcycle, etc.). As such, the present invention can generate fusion data having accuracy for an object detected by the camera and radar device 10 by using the advantages of each of the camera and radar device 10 and complementing each other to each other.

3 is a diagram for explaining a process of generating fusion data. Referring to FIG. 3 , the signal processing unit 110 may generate radar data based on a radar signal reflected by an object. The image processing unit (not shown) may generate camera data based on image data through the camera artificial intelligence module 20 . In this case, the coordinates of the camera data may be converted into a radar coordinate system for target-specific matching with radar data. The fusion data generation unit 120 may generate fusion data using radar data and camera data converted to a radar coordinate system.

Meanwhile, the fusion data generating unit 120 may generate fusion data by analyzing a driving situation while the vehicle is driving and assigning weights between camera data and radar data based on the analyzed driving situation. Here, the driving situation refers to external environmental factors that affect driving of the vehicle, and may include various driving environment information (eg, rainy weather, foggy weather, night driving, etc.). For example, referring to FIG. 5 , when the weather during vehicle driving is rainy (or foggy), detection of an object (e.g., vehicle) using camera data 50 is less accurate. In this case, the fusion data generator 120 can generate fusion data by assigning a higher weight to the radar data 52 than the camera data 50. At this time, the fusion data to which a high weight is assigned to the radar data 52 is used for additional learning by the learning unit 140, and the location information of the object can be estimated from the camera data 50 based on the location information of the object included in the radar data 52.

Meanwhile, the learning unit 140 may input radar data to the radar artificial intelligence module for the radar device 10 for learning and input image data to the camera artificial intelligence module for the camera for learning. Here, the radar data is data generated while the vehicle is driving based on a reflected radar signal received while the vehicle is driving, and the image data is data generated while the vehicle is driving when a camera is mounted on the vehicle.

The learning unit 140 may input radar data to the radar artificial intelligence module to train the radar artificial intelligence module so that a detection process and a classification process for an object may be performed through the radar artificial intelligence module.

The learning unit 140 may input image data to the camera artificial intelligence module and train the camera artificial intelligence module to derive camera data including object detection result values and object classification result values through the camera artificial intelligence module.

A radar device and a camera mounted on a vehicle may receive a radar signal while driving the vehicle and generate image data in real time. Accordingly, camera data and radar data used in the learning unit 140 may be generated in real time based on radar signals and image data generated in real time. In addition, camera data and radar data generated in real time can be used in real time for basic learning of an artificial intelligence module and for additional learning of an artificial intelligence module in real time.

When the fusion data is generated by the fusion data generation unit 120, the learning unit 140 may additionally input the fusion data to the radar artificial intelligence module and the camera artificial intelligence module to further train the radar artificial intelligence module and the camera artificial intelligence module.

The object detection result by the radar device 10 is compared with the object detection result of the camera to derive a stable object detection result in bad weather conditions and night situations. In addition, since the radar device 10 is capable of limiting and classifying objects with movement of a certain speed or higher detected on the road through ground speed information into a specific class, the radar data can be used for learning of the camera artificial intelligence module in bad weather conditions and night situations.

The fusion data used for learning includes, for example, longitudinal and lateral position information of detected objects, and object type information (e.g., vehicles, trucks, motorcycles, bicycles, pedestrians, etc.) classified by a previously learned camera artificial intelligence module.

For example, referring to FIG. 4 , the learning unit 140 inputs fusion data and radar data (radar data generated based on a reflected radar signal received in real time) to the radar artificial intelligence module 22, and additionally trains the radar artificial intelligence module 22 to derive a classification result value for an object through the radar artificial intelligence module 22. In addition, the learning unit 140 inputs the fusion data and real-time image data to the camera artificial intelligence module 20 to further train the camera artificial intelligence module to derive camera data including object detection result values and object classification result values through the camera artificial intelligence module 20.

Thereafter, when real-time image data is input to the additionally learned camera artificial intelligence module, real-time camera data may be derived through the camera artificial intelligence module.

That is, the present invention may perform primary learning of an artificial intelligence module based on radar data and camera data corresponding to basic data. This corresponds to basic learning for the AI module to have minimum performance. In addition, the present invention may perform secondary learning of the artificial intelligence module based on fusion data generated to compensate for the disadvantages of each of the radar data and the camera data, thereby improving the performance of the artificial intelligence module. In particular, the present invention can enable effective radar artificial intelligence module learning by presenting a classification learning method based on fusion data for radar data having difficulty in annotation.

The classification unit 130 may classify objects detected by the radar device 10 through the learned artificial intelligence module based on the generated fusion data. Specifically, the classification unit 130 may classify objects detected by the radar device 10 through the additionally learned radar artificial intelligence module.

Meanwhile, the performance measuring unit (not shown) may measure the performance of the additionally learned radar artificial intelligence module based on the classification result of the object derived through the additionally learned radar artificial intelligence module.

The fusion data generating unit 120 may generate fusion data further based on a classification result value of an object derived from the radar artificial intelligence module when the performance value of the radar artificial intelligence module exceeds a predetermined threshold.

For example, referring to FIG. 6 , the fusion data generation unit 120 may generate fusion data based on a classification result value of an object derived through the radar artificial intelligence module 22 and camera data derived through the camera artificial intelligence module 20 when the performance value of the radar artificial intelligence module exceeds a preset threshold.

When fusion data is generated further based on the classification result of the object derived from the radar artificial intelligence module, the learning unit 140 additionally inputs the fusion data to the radar artificial intelligence module and the camera artificial intelligence module to further train the radar artificial intelligence module and the camera artificial intelligence module.

The autonomous driving unit 150 may perform autonomous driving of the vehicle based on the object classification result value derived from the radar artificial intelligence module. For example, when a vehicle drives with the radar device 10 mounted without a camera, the vehicle transmits and receives a radar signal through the radar device 10, inputs a radar signal reflected from an object to a radar artificial intelligence module, and performs autonomous driving based on a classification result for an object derived from the radar artificial intelligence module.

On the other hand, those skilled in the art will fully understand that each of the transceiver 100, the signal processor 110, the fusion data generator 120, the classifier 130, the learner 140, and the autonomous driving unit 150 may be implemented separately, or one or more of them may be integrated and implemented.

7 is a flowchart illustrating a method of classifying an object through a radar device 10 mounted on a vehicle according to an embodiment of the present invention.

Referring to FIG. 7 , in step S701, the radar device 10 may transmit a radar signal to the outside of the vehicle.

In step S703, the radar device 10 may receive the radar signal reflected by the object.

In step S705, the radar device 10 may process the reflected radar signal to detect an object.

In step S707, the radar device 10 may generate fusion data based on the radar data and the camera data.

In step S709, the radar device 10 may classify the object detected through the learned artificial intelligence module based on the generated fusion data.

In the above description, steps S701 to S709 may be further divided into additional steps or combined into fewer steps, depending on the implementation of the present invention. Also, some steps may be omitted if necessary, and the order of steps may be changed.

An embodiment of the present invention may be implemented in the form of a recording medium including instructions executable by a computer, such as program modules executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustrative purposes, and those skilled in the art may understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts thereof should be construed as being included in the scope of the present invention.

10: radar device
100: transceiver
110: signal processing unit
120: fusion data generation unit
130: classification unit
140: learning unit
150: autonomous driving unit

Claims (12)

In the radar device mounted on the vehicle,
a transceiver for transmitting a radar signal to the outside of the vehicle and receiving a radar signal reflected by an object;
a signal processor processing the reflected radar signal to detect the object;
a fusion data generating unit generating fusion data based on radar data and camera data; and
Classification unit classifying the sensed object through an artificial intelligence module learned based on the generated fusion data.
Including,
A learning unit for inputting the radar data and the fusion data to the radar artificial intelligence module for the radar device for learning, and inputting the image data generated by the camera and the fusion data to the camera artificial intelligence module for the camera for learning. Radar device further comprising.
According to claim 1,
The signal processing unit performs a signal processing process on the reflected radar signal to detect the object;
A radar device for performing recognition of the object based on a result value of detecting the object.
According to claim 2,
The radar data includes at least one of a fast Fourier transform value of the reflected radar signal, a detection result value of the object, and a recognition result value of the object;
The camera data includes a detection result value for an object included in the image data and a classification result value for the object.
According to claim 3,
Wherein the fusion data generation unit generates the fusion data by matching the camera data and the radar data for each target.
According to claim 4,
The camera data is derived through the camera artificial intelligence module, the radar device.
delete According to claim 1,
The fusion data generating unit generates the fusion data further based on a classification result value for the object derived from the radar artificial intelligence module when the performance evaluation value for the radar artificial intelligence module exceeds a preset threshold. Radar device.
According to claim 1,
Wherein the classification unit classifies the sensed object through the learned radar artificial intelligence module.
According to claim 3,
The radar signal is received while the vehicle is driving,
Wherein the image data is generated during driving of the vehicle when the camera is mounted on the vehicle.
According to claim 9,
Wherein the fusion data generation unit analyzes a driving situation while the vehicle is driving, and generates the fusion data by assigning a weight between the camera data and the radar data based on the analyzed driving situation.
According to claim 1,
An autonomous driving unit that performs autonomous driving on the vehicle based on the classification result of the object.
Which further includes a radar device.
A method for classifying an object by a radar device mounted on a vehicle,
Transmitting a radar signal to the outside of the vehicle;
receiving a radar signal reflected by an object;
processing the reflected radar signal to detect the object;
generating fusion data based on radar data and camera data; and
Classifying the sensed object through an artificial intelligence module learned based on the generated fusion data.
Including,
Further comprising the steps of inputting the radar data and the fusion data to a radar artificial intelligence module for the radar device for learning, and inputting image data generated by the camera and the fusion data to a camera artificial intelligence module for the camera for learning.

Images