KR20220101540A - Object detection method based on radar-camera fusion and electronic apparatus - Google Patents

Abstract

The present invention relates to a method for detecting an object by converging image data and radar data and an electronic apparatus. The method for detecting the object by converging the image data and the radar data of the present invention comprises the following steps of: detecting, by the electronic apparatus of a vehicle, a specific object using first image data and first radar data at a specific time point; detecting, by the electronic apparatus, visual object information of the road surface from second image data of a viewpoint after the specific time point; setting, by the electronic apparatus, a target region of interest for object detection based on the visual object information; and detecting, by the electronic apparatus, the specific object by using the second image data or the second radar data of a later view of the target region of interest. Specific object detection at a later time point further uses a detection result of the specific object obtained at the specific time point.

Description

OBJECT DETECTION METHOD BASED ON RADAR-CAMERA FUSION AND ELECTRONIC APPARATUS

The technology to be described below relates to an object detection technique using a camera and a radar in a vehicle.

Modern vehicles are equipped with advanced driver assistance systems (ADAS). The driving assistance system obtains external information using various sensor devices and analyzes it to provide a function for safety and convenience of driving a vehicle. Representative sensor devices include a camera and a radar.

The camera creates an image containing an object located in the field of view (FOV). Cameras are useful for object classification and object classification. Radar is useful for recognizing distant objects located in the FOV and measuring relative speed. However, cameras are vulnerable to changes in illuminance or climate change (rainfall, fog, snow, etc.), and radar has limitations in that it is difficult to identify small objects or classify objects. Therefore, in order to complement the shortcomings of the camera and the radar, a convergence technology of the camera and the radar is being developed.

US Patent Publication No. 2020-0175315

Inter-sensor fusion consists of a low-level fusion method that combines image data and radar data at the raw data level (early fusion) and a high-level fusion method that combines the results of processing image data and radar data respectively (late fusion). have. Low-level fusion has a limitation that high computational power is required, and high-level fusion has a limit that additional computation is required to verify each data.

The technology to be described below is intended to provide a hybrid fusion method that complements the disadvantages of image data and radar data.

The method for detecting an object by fusing image data and radar data includes: extracting, by an electronic device of a vehicle, information on a specific object using each of first image data and first radar data at a specific time point; detecting visual object information of a road surface from second image data of a viewpoint after the viewpoint; setting, by the electronic device, a target region of interest for object detection based on the visual object information; and, by the electronic device, the target interest and detecting the specific object using second image data or second radar data of the later time point for the area. The specific object detection at the later time point further uses the detection result of the specific object acquired at the specific time point.

An electronic device for detecting an object by fusing image data and radar data includes an input device that receives first image data and first radar data at a specific time point and second image data and second radar data at a time point after the specific time point; A storage device for storing a program for detecting an object using data and radar data, and extracting information on a specific object using the first image data and the first radar data, respectively, and extracting information on a road surface from the second image data and a computing device configured to set a target ROI based on visual object information, and to fuse detection results using the second image data for the target ROI and the second radar data to detect the specific object. The computing device detects the specific object at the later point in time by further using the detection result of the specific object acquired at the specific point in time.

The technology to be described below reduces fusion target data by fusing image data and radar data based on an adaptive region of interest (ROI) set according to the driving environment of the vehicle. In addition, the technology to be described below fuses image data and radar data in the object detection step, but utilizes the prediction result of the heterogeneous data used just before, so that it is possible to quickly and accurately detect an object.

1 is an example of a flowchart for a process of detecting an object by fusing image data and radar data in a vehicle.
2 is another example of a flowchart for a process of detecting an object by fusing image data and radar data in a vehicle.
3 is an example of a vehicle having a built-in sensor device.
4 is an example of a process of setting an adaptive ROI.
5 is an example of a process of detecting an object by fusing image data and radar data.
6 is an example of an electronic device for detecting an object.

Since the technology to be described below can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the technology described below to specific embodiments, and it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the technology described below.

Terms such as first, second, A, and B may be used to describe various components, but the components are not limited by the above terms, and only for the purpose of distinguishing one component from other components. used only as For example, a first component may be named as a second component, and similarly, the second component may also be referred to as a first component without departing from the scope of the technology to be described below. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

In terms of terms used herein, the singular expression should be understood to include the plural expression unless the context clearly dictates otherwise, and terms such as "comprises" include the described feature, number, step, operation, element. , parts or combinations thereof are to be understood, but not to exclude the possibility of the presence or addition of one or more other features or numbers, step operation components, parts or combinations thereof.

Prior to a detailed description of the drawings, it is intended to clarify that the classification of the constituent parts in the present specification is merely a division according to the main function that each constituent unit is responsible for. That is, two or more components to be described below may be combined into one component, or one component may be divided into two or more for each more subdivided function. In addition, each of the constituent units to be described below may additionally perform some or all of the functions of other constituent units in addition to the main function it is responsible for. Of course, it can also be performed by being dedicated to it.

In addition, in performing the method or operation method, each process constituting the method may occur differently from the specified order unless a specific order is clearly described in context. That is, each process may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

A technique to be described below is a technique for detecting an object around a vehicle using data acquired by a sensor device of the vehicle. This is an object detection technique using a camera and radar to be described below. A vehicle may use multiple cameras, and may use more than one radar. Accordingly, the arrangement or number of sensor devices (cameras and radars) in the technology to be described below may vary. That is, the technology to be described below may be utilized for object detection in various directions, such as front, side, and rear. Accordingly, the following description will be based on a camera and a radar having the same or overlapping FOV. That is, the following description is based on one FOV. Of course, the techniques described below may be simultaneously applied to multiple FOVs.

The image data is data generated by the camera. Image data uses a basically two-dimensional image coordinate system. The starting point of X and Y may vary depending on the setting.

Radar data is data generated by radar. A description of the radar data generation process will be omitted. Radar data basically uses the world coordinate system. The world coordinate system consists of three-dimensional X, Y, and Z axes, and the direction of the axes or the starting point of each axis may vary depending on the settings.

A technique to be described below is a technique for detecting an object by fusing image data and radar data. The fusion method applied to the technology to be described below may be referred to as a hybrid fusion method. The hybrid fusion method can be called a hybrid in two respects. One criterion is that a specific ROI may be set using the image data, and the image data and radar data may be fused within the set ROI. Another criterion is that when detecting an object based on features extracted from image data or radar data, prediction results of other heterogeneous data at a previous time point are used. It will be described in detail below.

Hereinafter, an electronic device refers to a device for detecting an object in a vehicle using image data and/or radar data. For example, the electronic device may mean an electronic control unit (ECU) of a vehicle, a dedicated detection device, or the like.

1 is an example of a flowchart for a process 100 of detecting an object by fusing image data and radar data in a vehicle. 1 is an example of object detection at time t. Here, t means a specific time point. Based on the interval at which data is collected, t-1 denotes a time point immediately before t, and t+1 denotes a time point immediately after t.

The camera generates first image data at time t-1. The electronic device acquires first image data (110). The electronic device may extract object information by using the first image data ( 110 ). The object information is information about an object located in the FOV of the camera and may include at least one of a position, a speed, and a size of the object. Alternatively, the electronic device may detect the object using the first image data.

The radar generates first radar data at time t-1. The electronic device acquires first radar data ( 120 ). The electronic device may extract object information using the first radar data ( 120 ). The object information is information about an object located in the FOV of the radar, and may include at least one of a position, a speed, and a size of the object. Alternatively, the electronic device may detect the object using the first radar data.

Thereafter, the camera generates second image data at time t. The radar generates second radar data at time t. The electronic device acquires second image data and second radar data ( 130 ).

The electronic device sets the target region of interest based on the second image data ( 140 ). The target region of interest refers to a region for detecting an object by fusing image data and radar data. The target region of interest may include a general candidate region and a warning region having a higher priority than the candidate region according to risk or importance. The electronic device may first detect an object with respect to the warning area. The electronic device may adaptively set the target ROI based on information obtained from the second image data. The setting of the target region of interest will be described later.

The electronic device may detect the object by fusing the first image data and the second radar data for the target ROI ( 150 ). The electronic device may fuse the first image data and the second radar data by limiting the range to the target ROI. The electronic device may detect an object using the second radar data at time t by using object information of the first image data acquired at time t-1. In this case, the electronic device utilizes object information predicted at time t (at least one of an object location, speed, and size) based on the object information extracted from the first image data at time t-1, and the second radar at time t Detect objects in data. When object information at time t predicted based on the first image data at time t-1 is used, the electronic device can effectively detect an object that is difficult to identify only with the second radar data. As a result, the electronic device detects an object by fusing the first image data at time t-1 and the second radar data at time t. This method is different from the conventional low-level fusion, and it is also different from the high-level fusion in which each object is detected and the results are merged.

Also, the electronic device may detect the object by fusing the first radar data and the second image data for the target ROI ( 160 ). The electronic device may fuse the first radar data and the second image data by limiting the range to the target ROI. The electronic device may detect the object using the second image data of the time t by utilizing the object information of the first radar data acquired at the time t-1. In this case, the electronic device utilizes the object information predicted at time t (at least one of an object position, velocity, and size) based on the object information extracted from the first radar data at time t-1, and the second image at time t Detect objects in data. If object information at time t predicted based on the first radar data at time t-1 is used, the electronic device can effectively detect an object that is difficult to identify only with the second image data. As a result, the electronic device detects an object by fusing the first radar data at time t-1 and the second image data at time t. This method is different from the conventional low-level fusion, and it is also different from the high-level fusion in which each object is detected and the results are merged.

The electronic device checks whether the object detection is finished ( 170 ), and continues object detection in the same manner when the object detection is still required. The subsequent time point is a time point immediately after the current time t, and t = t + 1 (180). At time t+1, the electronic device retains object information extracted using image data and radar data at time t. Accordingly, the electronic device may detect each object using image data and radar data at time t+1.

Furthermore, at time t+1, the electronic device may estimate object information predicted at time t+1 by using the result (150 and/or 160) of detecting an object by fusing image data and radar data at time t. . In this case, the electronic device predicts the object information at time t+1 based on the object information predicted by the fusion method at time t, and uses the image data or radar data acquired at time t+1 to determine the object information at time t+1. object can be detected.

2 is another example of a flowchart for a process 200 of detecting an object by fusing image data and radar data in a vehicle. 1 is an example of object detection at time t.

The camera generates first image data at time t-1. The electronic device acquires first image data ( 210 ). The electronic device may extract object information by using the first image data ( 210 ). The object information is information about an object located in the FOV of the camera and may include at least one of a position, a speed, and a size of the object. Alternatively, the electronic device may detect the object using the first image data.

The radar generates first radar data at time t-1. The electronic device acquires first radar data (220). The electronic device may extract object information by using the first radar data ( 220 ). The object information is information about an object located in the FOV of the radar, and may include at least one of a position, a speed, and a size of the object. Alternatively, the electronic device may detect the object using the first radar data.

Thereafter, the camera generates second image data at time t. The radar generates second radar data at time t. The electronic device acquires second image data and second radar data ( 230 ).

The electronic device sets a target region of interest based on the second image data ( 240 ). The target region of interest refers to a region for detecting an object by fusing image data and radar data. The target region of interest may include a general candidate region and a warning region having a higher priority than the candidate region according to risk or importance. The electronic device may first detect an object with respect to the warning area. The electronic device may adaptively set the target ROI based on information obtained from the second image data. The setting of the target region of interest will be described later.

The electronic device may detect the object by fusing the first image data and the second radar data for the target ROI ( 250 ). The electronic device may fuse the first image data and the second radar data by limiting the range to the target ROI. The electronic device may detect an object using the second radar data at time t by using object information of the first image data acquired at time t-1. In this case, the electronic device utilizes object information predicted at time t (at least one of an object location, speed, and size) based on the object information extracted from the first image data at time t-1, and the second radar at time t Detect objects in data. When object information at time t predicted based on the first image data at time t-1 is used, the electronic device can effectively detect an object that is difficult to identify only with the second radar data. As a result, the electronic device detects an object by fusing the first image data at time t-1 and the second radar data at time t. This method is different from the conventional low-level fusion, and it is also different from the high-level fusion in which each object is detected and the results are merged.

Also, the electronic device may detect an object by fusing the first radar data and the second image data for the target ROI ( 260 ). The electronic device may fuse the first radar data and the second image data by limiting the range to the target ROI. The electronic device may detect the object using the second image data of the time t by utilizing the object information of the first radar data acquired at the time t-1. In this case, the electronic device utilizes the object information predicted at time t (at least one of an object position, velocity, and size) based on the object information extracted from the first radar data at time t-1, and the second image at time t Detect objects in data. If object information at time t predicted based on the first radar data at time t-1 is used, the electronic device can effectively detect an object that is difficult to identify only with the second image data. As a result, the electronic device detects an object by fusing the first radar data at time t-1 and the second image data at time t. This method is different from the conventional low-level fusion, and it is also different from the high-level fusion in which each object is detected and the results are merged.

The electronic device provides (i) a result of detecting an object by fusing the first image data and the second radar data (250) and (ii) a result of detecting an object by fusing the first radar data and the second image data (260). may be merged to finally detect the object ( 270 ). In this process, the electronic device predicts an object based on a result of merging the object detected in step 250 and the object detected in step 260 . For example, the electronic device may select an object detected overlapping in the object detection result of step 250 and the object detection result of step 260 as the final detection result.

The electronic device checks whether the object detection is finished ( 280 ), and if the object detection is still required, the object detection is continued in the same manner. The subsequent time point is a time point immediately after the current time t, and t = t + 1 (290). At time t+1, the electronic device retains object information extracted using image data and radar data at time t. Accordingly, the electronic device may detect each object using image data and radar data at time t+1.

Furthermore, at time t+1, the electronic device may estimate object information predicted at time t+1 by using the results (250 and/or 260) of detecting an object by fusing image data and radar data at time t. . In this case, the electronic device predicts the object information at time t+1 based on the object information predicted by the fusion method at time t, and uses the image data or radar data acquired at time t+1 to determine the object information at time t+1. object can be detected.

3 is an example of a vehicle 300 having a built-in sensor device. 3 is an example of a vehicle system 300 including a radar 310 , a camera 320 , and ECUs 330 , 340 , 350 , and 360 . The sensor device and ECU can exchange data through vehicle communication (CAN, etc.).

3A is an example of a vehicle system 300 including a radar 310 , a camera 320 , and a plurality of ECUs 330 , 340 and 350 . Radar 310 is typically disposed on the front side of the vehicle. Of course, unlike FIG. 3A , the radar 310 may be disposed at a different location, or a plurality of radars may be used.

3A is an example in which four cameras 321 , 322 , 323 , and 324 are arranged by way of example. The arrangement position or number of cameras may vary according to the needs of the driving assistance system.

The ECU 330 may receive the radar data generated by the radar 310 and extract object information.

The ECU 340 may receive image data generated by at least one of the plurality of cameras 321 , 322 , 323 and 324 , and extract object information.

The ECU 350 may receive object information at time t-1 from the ECU 330 and the ECU 340 , respectively. Also, the ECU 350 may receive image data and/or radar data at time t, respectively, from the ECU 330 and the ECU 340 . The ECU 350 may detect an object by fusing object information extracted from the image data at time t-1 and radar data at time t. Also, the ECU 350 may detect an object by fusing object information extracted from radar data at time t-1 and image data at time t. This process is the same as the process described in FIGS. 1 and 2 .

3B is an example of a vehicle system 300 including a radar 310 , a camera 320 , and an ECU 360 . The ECU 360 may be referred to as an ECU dedicated to the hybrid fusion method.

Radar 310 is typically disposed on the front side of the vehicle. Of course, unlike FIG. 3B , the radar 310 may be disposed at a different location, or a plurality of radars may be used.

3(B) is an example in which four cameras 321 , 322 , 323 and 324 are arranged. The arrangement position or number of cameras may vary according to the needs of the driving assistance system.

The ECU 360 may receive the radar data generated by the radar 310 and extract object information.

The ECU 360 may receive image data generated by at least one of the plurality of cameras 321 , 322 , 323 and 324 , and extract object information.

The ECU 360 may extract object information at time t-1 using data transmitted from at least one of the radar 310 and the plurality of cameras 321 , 322 , 323 , and 324 , respectively. Also, the ECU 360 may receive image data and/or radar data at time t from at least one of the radar 310 and the plurality of cameras 321 , 322 , 323 and 324 , respectively. The ECU 360 may detect an object by fusing object information extracted from the image data at time t-1 and radar data at time t. Also, the ECU 360 may detect an object by fusing object information extracted from radar data at time t-1 and image data at time t. This process is the same as the process described in FIGS. 1 and 2 .

4 is an example of a process of setting an adaptive ROI.

The electronic device may extract certain information by using the image data.

The electronic device may extract visual object information of the road surface from the image data. Here, the visual object information may include information on at least one of a lane, a type of a lane, a bending direction of the lane, a bending degree of the lane, and a stop line.

For example, the electronic device may classify lanes to identify a current driving lane of the vehicle and a driving lane of a neighboring vehicle. The electronic device may classify the types of lanes (center line, solid line, dotted line, etc.) to predict the proximity of a nearby vehicle. The electronic device may determine whether the road is a curved road based on the direction in which the lane is curved. The electronic device may determine the curvature information based on the degree of curvature of the lane. The electronic device may predict the possibility of speed deceleration of the current vehicle and/or whether the preceding vehicle is decelerated based on the stop line.

The visual information on the road surface includes various information in addition to the lane. Accordingly, the visual object information may further include at least one of a driving direction display, a speed limit display, a protected area display, a road information display, a crosswalk display, a prohibition display, and a stop display.

For example, the electronic device may predict the traveling direction of a vehicle in a current lane and a neighboring lane based on the driving direction display (arrow). The electronic device may predict whether the speed of the current vehicle or a nearby vehicle will increase or decrease based on the speed limit display. The electronic device can predict whether the current vehicle or nearby vehicles will slow down based on the protected area mark (children's area, animal shelter, etc.). The electronic device may predict whether the current vehicle and/or surrounding vehicles are decelerating or whether to change a course based on other road information display (deceleration, lane reduction, etc.). The electronic device may predict whether the current vehicle and/or nearby vehicles are decelerating or whether to walk on the crosswalk based on the crosswalk display. The electronic device may predict the driving direction of the current vehicle and/or surrounding vehicles based on the prohibition mark (for example, prohibition of going straight, prohibition of turning, prohibition of U-turn, etc.). The electronic autonomy may predict whether the current vehicle and/or surrounding vehicles will decelerate or stop based on the stop indication.

Furthermore, the electronic device may set the target ROI by further using at least one of information such as a current vehicle speed, a current time, and ambient illuminance. For example, the electronic device may adaptively change the length or size of the target ROI according to the speed of the vehicle. In addition, the electronic device may adaptively change the length or size of the target ROI when the surroundings are dark according to the current time or/and ambient illuminance.

4A is an example in which the current vehicle T sets a target ROI based on lane information of a straight road. For convenience of description, it will be described that the vehicle sets the target ROI. Specifically, the electronic device disposed in the vehicle sets the target ROI.

The vehicle T may set the target ROI by using image data acquired from the camera FOV and radar data acquired from the radar FOV. The vehicle T acquires lane information of the camera FOV. Vehicle T may distinguish a current driving lane and an adjacent lane on a straight course. As described above, the target ROI may include a warning region and a candidate region. In some cases, the target region of interest may consist of only a warning region or a candidate region.

The current driving lane of vehicle T may be set as a warning area. The vehicle T may set lane(s) adjacent to the current driving lane as a candidate area. In FIG. 4A , vehicle A is located in the candidate area, and vehicle C is located in the warning area. Meanwhile, vehicle B may not be a detection target because it is located outside the target ROI. In this situation, vehicle T may first detect whether there is an object (vehicle C) located in the warning area.

The vehicle T may adaptively set the length (based on the driving direction) of the warning area or the candidate area in consideration of the speed of the vehicle T. For example, if the speed of the vehicle T is high, the length of the warning area or the candidate area within the observation range of the camera or radar may be longer. Also, the vehicle T may expand or move the target ROI according to the traveling direction in consideration of the traveling direction of the vehicle T.

4B is an example in which the current vehicle T sets a target ROI based on lane information of a curved (curve) road. For convenience of description, it will be described that the vehicle sets the target ROI. Specifically, the electronic device disposed in the vehicle sets the target ROI.

The vehicle T may set a target ROI based on lane information (a lane, a bending direction of a lane, and information about a bending of a lane) obtained from image data. Vehicle T is currently meeting a curved road that curves to the right. The vehicle T may set the target ROI by moving the previous or default ROI to the right. Alternatively, the vehicle T may set the target ROI by extending the previous or basic ROI in the lateral direction (based on the driving direction). 4B is an example of setting a candidate area. In FIG. 4B , both vehicle A and vehicle C are located in the target ROI.

The vehicle T may adaptively set the lateral length of the target region of interest in consideration of the speed of the vehicle T. For example, if the speed of the vehicle T is high, the lateral length may be set longer within the observation range of the camera or radar. Also, the vehicle T may expand or move the target ROI according to the traveling direction in consideration of the traveling direction of the vehicle T.

4(C) is an example in which the current vehicle T sets a target region of interest in a situation where the vehicle meets a crosswalk. For convenience of description, it will be described that the vehicle sets the target ROI. Specifically, the electronic device disposed in the vehicle sets the target ROI.

It can be seen that a crosswalk appears in front of vehicle T based on the visual object information of the road surface included in the image data. The vehicle T may set the target ROI by extending the previous or basic ROI in the lateral direction (based on the driving direction). Also, the longitudinal direction of the vehicle T immediately before or of the basic ROI may be reduced uniformly. The vehicle T may set an area including the current driving lane as a warning area, and may set an area including a neighboring lane as a candidate area.

The vehicle T may adaptively set the lateral length of the target region of interest in consideration of the speed of the vehicle T. For example, if the speed of the vehicle T is high, the length of the target region of interest within the observation range of the camera or radar may be longer, and the length in the lateral direction may be set to about three lane widths. Also, the vehicle T may expand or move the target ROI according to the traveling direction in consideration of the traveling direction of the vehicle T.

4(D) is an example in which the current vehicle T sets a target ROI in a situation where the vehicle meets an intersection. For convenience of description, it will be described that the vehicle sets the target ROI. Specifically, the electronic device disposed in the vehicle sets the target ROI.

The vehicle T may know that an intersection appears in front based on the visual object information on the road surface included in the image data. The vehicle T may set the target ROI by extending the previous or basic ROI in the lateral direction (based on the driving direction). In the case of a three-way intersection, the vehicle T may set the target ROI by moving the previous or basic ROI in one direction.

Also, the longitudinal direction of the vehicle T immediately before or of the basic ROI may be reduced uniformly. The vehicle T may set an area including the current driving lane as a warning area, and may set an area including a neighboring lane as a candidate area. In FIG. 4(D) , vehicle A is located in the candidate area.

The vehicle T may adaptively set the lateral length of the target region of interest in consideration of the speed of the vehicle T. For example, if the speed of the vehicle T is high, the length of the target region of interest within the observation range of the camera or radar may be longer, and the length in the lateral direction may be set to about three lane widths. Also, the vehicle T may expand or move the target ROI according to the traveling direction in consideration of the traveling direction of the vehicle T.

The target region of interest described in FIG. 4 is an example. The vehicle (electronic device) may adaptively set the target ROI by using various visual object information different from that of FIG. 4 .

5 is an example of a process of detecting an object by fusing image data and radar data. FIG. 5 is an example illustrating a process of detecting an object based on the vehicle system of FIG. 3B .

The camera 310 generates first image data at time t-1 and second image data at time t. The radar 320 generates first radar data at time t-1 and second radar data at time t.

The ECU 360 extracts object information from the first image data. The ECU 360 may detect an object from the first image data (①).

The ECU 360 extracts object information from the first radar data. The ECU 360 may detect an object from the first radar data (②).

The ECU 360 may set the target ROI at time t based on the second image data. Setting the target region of interest is the same as described in FIG. 4 (③).

The ECU 360 may fuse the second image data and the first radar data to detect the object at time t (④).

When only the second image data is used, the ECU 360 uses the Kalman filter to detect an object in the image frame (second image data) at the time t based on the corrected result value up to the image frame at the time t-1. to extract A process in which the ECU 360 detects an object at time t may be expressed as "T frame = function {a, b, c}". Here, a, b, and c are features (variables) extracted from image data.

In the case of using the hybrid fusion method, the ECU 360 detects an object based on the image data at the time t by utilizing a result predicted using radar data up to time t-1. A process in which the ECU 360 detects an object at time t can be expressed as "T frame = function {a, b, c, Radar(T-1 frame)}. Here, Radar(T-1 frame) is t - Corresponds to object information predicted using radar data at the time of -1 (object location, object speed, object size, etc.).

The ECU 360 may fuse the second radar data and the first image data to detect the object at time t (⑤).

When only the second radar data is used, the ECU 360 uses the Kalman filter to detect objects in the radar frame (second radar data) at the time t based on the corrected result value up to the radar data at the time t-1. to extract A process in which the ECU 360 detects an object at time t may be expressed as “T frame = function {x, y, z}”. Here, x, y, and z are features (variables) extracted from radar data.

In the case of using the hybrid fusion method, the ECU 360 detects an object based on the radar data at the time t by utilizing a result predicted using the image data up to time t-1. A process in which the ECU 360 detects an object at time t may be expressed as "T frame = function {x, y, z, Image(T-1 frame)}". Here, Image (T-1 frame) corresponds to object information (position of an object, velocity of an object, size of an object, etc.) predicted using image data at time t-1.

Further, the ECU 360 detects a result of "T frame = function {a, b, c, Radar(T-1 frame)}" and "T frame = function {x, y, z, Image(T-1)" frame)}", and finally the object can be detected by merging the predicted results.

An example of the object detection result is shown at the bottom of FIG. 5 .

First, when the ECU 360 detects a crosswalk based on image data of a current viewpoint, the ECU 360 may horizontally expand the target ROI. In addition, the ECU 360 may expand the target ROI in the lateral direction or the like when identifying the school zone mark on the road surface. Also, the ECU 360 may identify the lane information to expand the target ROI in the longitudinal direction in the case of the current driving lane.

The ECU 360 may detect a person O1 crossing the crosswalk. Also, the ECU 360 may detect another vehicle O2 that is in front of the current vehicle in the striking direction. According to the detection result, the ECU 360 may transmit information to another ECU or infotainment ECU that controls the operation of the vehicle to reduce the speed of the vehicle or to warn the driver of certain information.

6 is an example of an electronic device for detecting an object. The electronic device may be implemented in the form of an ECU of a vehicle, an object detection-only device, a data processing-only chipset, or the like.

The electronic device 400 may include a storage device 410 , a memory 420 , an arithmetic device 430 , an interface device 440 , and a communication device 450 .

The storage device 410 may store a program for extracting object information from image data or for detecting an object.

The storage device 410 may store a program for extracting object information from radar data or detecting an object.

The storage device 410 may store a program for detecting an object by fusing image data and radar data.

The storage device 410 may store information on the detected object.

The memory 420 may store data and information generated by the electronic device 400 during an object detection process.

The interface device 440 is a device that receives predetermined data and commands from other objects connected through vehicle communication.

The interface device 440 may receive image data and/or radar data from a sensor device (camera and/or radar). The interface device 440 may transmit a predetermined command (data request, etc.) to the sensor device.

The interface device 440 may receive predetermined data or commands from other ECUs. The interface device 440 may transmit an object detection result or command to another ECU.

The communication device 450 means a configuration for receiving and transmitting certain information through a wireless network. Vehicle communication mainly exchanges data and information through an interface device connected by a bus network. However, in some cases, the vehicle may exchange information between devices inside the vehicle or with an object outside the vehicle through a separate wireless interface 450 .

The communication device 450 may receive image data and/or radar data from a sensor device (camera and/or radar). The communication device 450 may transmit a predetermined command (such as a data request) to the sensor device.

The communication device 450 may receive certain data or commands from other ECUs. The communication device 450 may transmit an object detection result or a command to another ECU.

The communication device 450 may transmit the object detection result and the extracted object information to an external object (RSU, surrounding vehicle).

The computing device 430 may extract object information from the first image data at time t-1 by using a program in the storage device 410 .

The computing device 430 may extract object information from the first radar data at time t-1 by using a program in the storage device 410 .

The computing device 430 may set the target ROI based on the second image data at time t by using a program in the storage device 410 . The computing device 430 may set the target ROI based on visual object information on the road surface. Setting the target region of interest is the same as described with reference to FIG. 4 .

The computing device 430 may detect an object by fusing the second image data at time t and the first radar data at time t for the target ROI. Object detection is the same as described with reference to FIG. 5 .

The computing device 430 may detect an object by fusing the second radar data at time t and the first image data at time t for the target ROI. Object detection is the same as described with reference to FIG. 5 .

In addition, the computing device 430 fuses the second image data at time t and the first radar data at time t to detect an object and fuses the second radar data at time t and the first image data at time t. An object may be finally detected by merging the results of detecting the object.

The computing device 430 may be a device such as a processor, an AP, or a program embedded chip that processes data and processes a predetermined operation.

In addition, the adaptive region-of-interest setting method, the camera and radar fusion method, and the object detection method as described above may be implemented as a program (or application) including an executable algorithm that can be executed in a computer. The program may be provided by being stored in a temporary or non-transitory computer readable medium.

The non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, etc., and can be read by a device. Specifically, the various applications or programs described above are CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM (read-only memory), PROM (programmable read only memory), EPROM (Erasable PROM, EPROM) Alternatively, it may be provided while being stored in a non-transitory readable medium such as an EEPROM (Electrically EPROM) or a flash memory.

Temporarily readable media include: Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (Enhanced) SDRAM, ESDRAM), Synchronous DRAM (Synclink DRAM, SLDRAM) and Direct Rambus RAM (Direct Rambus RAM, DRRAM) refers to a variety of RAM.

This embodiment and the drawings attached to this specification merely clearly show a part of the technical idea included in the above-described technology, and within the scope of the technical idea included in the specification and drawings of the above-described technology, those skilled in the art can easily It will be apparent that all inferred modified examples and specific embodiments are included in the scope of the above-described technology.

Claims (15)

extracting, by an electronic device of a vehicle, information on a specific object using each of first image data and first radar data at a specific time;
detecting, by the electronic device, visual object information of a road surface from second image data of a time point after the specific time point;
setting, by the electronic device, a target ROI for object detection based on the visual object information; and
and detecting, by the electronic device, the specific object using second image data or second radar data of the later time point for the target ROI, wherein the detection of the specific object at the later time point is performed at the specific time point. A method of detecting an object by fusing image data and radar data that further use the obtained detection result of the specific object. According to claim 1,
The visual object information is a method of detecting an object by fusing image data and radar data including a lane, a type of lane, a bending direction of a lane, a bending degree of a lane, and a stop line. 3. The method of claim 2,
The visual object information detects an object by fusing image data and radar data further comprising at least one of a driving direction display, a speed limit display, a protected area display, a road information display, a crosswalk display, a prohibition display, and a stop display. How to. According to claim 1,
The electronic device detects an object by fusing image data and radar data for setting the target region of interest by further using at least one of the vehicle speed, time, and ambient illuminance at the later time point. According to claim 1,
The method of detecting an object by fusing image data and radar data, wherein the target region of interest includes a warning region having a high priority for object tracking and a candidate region having a lower priority for object tracking than the warning region. According to claim 1,
The electronic device receives image data and radar data for setting the target region of interest by moving, expanding in at least one direction, or reducing the initial region of interest set before the later time in one direction based on the visual object information. A method for detecting objects by fusion. According to claim 1,
The electronic device further uses the position, velocity, and size of the object predicted in the time t frame with respect to the specific object based on the first radar data of the time t-1 frame, and the second image of the time t frame A method of detecting an object by fusing image data and radar data for detecting the object in data. According to claim 1,
The electronic device further uses the position, velocity, and size of the object predicted in the time t frame with respect to the specific object based on the first image data of the time t-1 frame, and the second radar of the time t frame A method of detecting an object by fusing image data and radar data for detecting the object in data. According to claim 1,
The electronic device fuses image data and radar data further comprising the step of finally detecting the specific object by merging results of detecting the specific object using the second image data and the second radar data, respectively. How to detect an object. an input device for receiving first image data and first radar data at a specific time point, and second image data and second radar data at a time point after the specific time point;
a storage device for storing a program for detecting an object using image data and radar data; and
information of a specific object is extracted using the first image data and the first radar data, respectively, and a target region of interest is set based on visual object information of a road surface from the second image data, and the target region of interest is And a computing device for detecting the specific object by fusing the detection result using the second image data and the second radar data,
The computing device further uses a detection result of the specific object acquired at the specific point in time to fuse image data for detecting the specific object at the later time point and radar data to detect the object. 11. The method of claim 10,
The visual object information is an electronic device that detects an object by fusing image data and radar data including a lane, a type of lane, a bending direction of the lane, a bending degree of the lane, and a stop line. 11. The method of claim 10,
The computing device detects an object by fusing image data and radar data for setting the target region of interest by further using at least one of the vehicle speed, time, and ambient illuminance at the later time point. 11. The method of claim 10,
The computing device performs image data and radar data for setting the target region of interest by moving an initial region of interest set before the subsequent time point in one direction based on the visual object information, expanding in at least one direction, or reducing the region in at least one direction. An electronic device that detects an object by fusion. 11. The method of claim 10,
The computing device further uses the position, velocity, and size of the object predicted in the time t frame with respect to the specific object based on the first radar data of the time t-1 frame, and the second image of the time t frame An electronic device for detecting an object by fusing image data and radar data for detecting the object from data. 11. The method of claim 10,
The computing device further uses the position, velocity, and size of the object predicted in the time t frame with respect to the specific object based on the first image data of the time t-1 frame, and the second radar of the time t frame An electronic device for detecting an object by fusing image data and radar data for detecting the object from data.

Images