KR102456151B1 - Sensor fusion system based on radar and camera and method of calculating the location of nearby vehicles - Google Patents

Abstract

The present invention relates to a sensor fusion system including radar and a camera and a method for calculating positions of nearby vehicles by using the sensor fusion system. In accordance with one embodiment of the present invention, the method for calculating positions of nearby vehicles, includes: a first method of calculating location information on a target estimated to be a nearby vehicle from a plurality of radar measurement points sensed through radar; a second method of recognizing the nearby vehicle included in a sensor fusion image between a camera and the radar as a target, and tracking location information of the target; calculating a cost function by using a center point of a target with respect to each target estimated or recognized based on at least one of the first method and the second method; and comparing the calculated cost function with respect to each target and selecting an optimal cost function value, and calculating location information about the target based on the selected cost function. Therefore, the present invention is capable of replacing high-priced and high-precision equipment for recognizing nearby vehicles.

Description

SENSOR FUSION SYSTEM BASED ON RADAR AND CAMERA AND METHOD OF CALCULATING THE LOCATION OF NEARBY VEHICLES

The present radar and camera-based sensor fusion system and a method for calculating the position of surrounding vehicles are related.

Radar (Radio Detection And Ranging) detects the position and direction of an object, and makes it possible to measure the distance and speed of the object. Radar is widely used for detecting the surrounding environment using radio waves, but in the case of vehicle radar, due to low angle recognition performance, an object is detected as a single point regardless of type, so there may be a limitation in distinguishing the type of object only by radar. .

In addition, with the recent development of deep learning technology, autonomous driving technology that recognizes the surrounding environment through multiple cameras in the vehicle and creates a driving route for the vehicle based on this is rapidly emerging. However, since various situations that can occur within the road traffic environment are variable, and the surrounding environment must be recognized and judged in real time, there may still be many difficulties related to autonomous driving. On the other hand, sensor fusion technology for detecting the surrounding environment by fusion of several sensors is also in the spotlight.

An object of the present disclosure is to provide a sensor fusion system including a radar and a camera, and a method for calculating a position of a surrounding vehicle using the sensor fusion system.

However, the objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

A first aspect of the present disclosure provides a method of calculating a location of a surrounding vehicle, the method comprising: a first method of calculating location information on a target estimated as a surrounding vehicle from a plurality of radar measurement points sensed through a radar; a second method of recognizing a surrounding vehicle included in a sensor fusion image between a camera and a radar as a target, and tracking location information about the target; calculating a cost function using a center point of the target for each target estimated or recognized based on at least one of the first method and the second method; and selecting an optimal cost function value by comparing the calculated cost functions for each target, and calculating location information about the target based on the selected cost function.

The first method includes: emitting radio waves from a radar, and acquiring a plurality of radar measurement points by detecting reflected waves that are reflected and incident on an object; clustering the plurality of radar measurement points into a predetermined group based on a target detection probability; and estimating the clustered group maintaining continuity as surrounding vehicles by updating the clustered group using a plurality of continuously acquired radar measurement points.

The second method may include detecting a surrounding vehicle by applying an object recognition algorithm to the sensor fusion image; setting a binding box for the detected surrounding vehicle area, and tracking the location information on the target using the area in which the binding box is set as a target.

The step of tracking the location information on the target may be a step of calculating continuous location information on the target reflecting real-time location change of the target by comparing a previous image frame and a current image frame in the sensor fusion image. .

The calculating of the cost function may include a first cost function obtained by summing horizontal distances between the virtual line and a plurality of radar measurement points based on an imaginary line perpendicular to the lateral center point of the target estimated by the first method. calculating ; Based on the virtual line corresponding to the heading direction (Yaw) of the vehicle passing through the center point of the binding box set in each target by targeting the surrounding vehicle recognized by the second method, the virtual line and the plurality of radar measurement points calculating a second cost function summing the horizontal distances between them; and selecting a target having a minimum cost function value by comparing the first cost function with the second cost function.

Calculating the cost function may include calculating a predetermined curve curve by accumulating previously determined target location information; estimating the location information of the target at the current time in consideration of the curve curve and the movable area of the vehicle; Calculating a third cost function summing the horizontal distance between the virtual line and the plurality of radar measurement points based on the virtual line corresponding to the heading direction (Yaw) of the vehicle passing through the expected center point of the target may include more.

A second aspect of the present disclosure provides a sensor fusion system based on a radar and a camera, comprising: a radar emitting radio waves to the surroundings and calculating lateral distance information with an object by detecting reflected waves that are reflected and incident on the object; a camera mounted on at least one of the front, rear, and side of the vehicle, the camera acquires the captured image and transmits it to the processor; The radar and the camera are calibrated, the sensor fusion image is generated by matching the lateral distance information with the object acquired from the radar and the captured image, and the location of the surrounding vehicle by analyzing the radar measurement points and the sensor fusion image It may include a processor, which calculates .

Calculating the location of the surrounding vehicle by the processor may include: a first method of calculating location information on a target estimated to be a surrounding vehicle from a plurality of radar measurement points sensed through a radar; a second method of recognizing a surrounding vehicle included in a sensor fusion image between a camera and a radar as a target, and tracking location information about the target; calculating a cost function using a center point of the target for each target estimated or recognized based on at least one of the first method and the second method; and selecting an optimal cost function value by comparing the calculated cost functions for each target, and calculating location information about the target based on the selected cost function.

In addition to this, another method for implementing the present invention, another system, and a computer-readable recording medium storing a computer program for executing the method may be further provided.

According to an embodiment of the present invention, since location tracking with a high level of precision is possible by processing the information collected by the radar and the camera, it can replace expensive, high-precision equipment for recognizing a nearby vehicle.

The sensor fusion system 100 according to an embodiment of the present invention compares and analyzes the cost functions calculated by various methods, so that the current value of the surrounding vehicle is higher than when the surrounding vehicle is located only by one method, respectively. Location can be tracked in real time.

1 is an exemplary diagram for explaining that a vehicle recognizes a surrounding vehicle using a camera and a radar in a road environment.
2 is a block diagram of a sensor fusion system according to an embodiment of the present invention.
3 is a flowchart illustrating a method of calculating the location of a nearby vehicle in the sensor fusion system 100 according to an embodiment of the present invention.
4 is an exemplary diagram for explaining calculating a cost function through various methods according to an embodiment of the present invention.
5 is an exemplary diagram for comparing and explaining a coordinate system used for each step according to an embodiment of the present invention.
6A to 6B are diagrams exemplarily illustrating location information of a target estimated based on radar measurement points acquired through a radar and a target tracking algorithm.
7 is a view for explaining obtaining location information of a surrounding vehicle through a camera according to an embodiment of the present invention.
8 is an exemplary view for explaining a method of calculating a first cost function according to an embodiment of the present invention.
9 is an exemplary view for explaining a method of calculating a second cost function according to another embodiment of the present invention.
10 to 11 are exemplary diagrams for explaining prediction of target location information of a current view from past target location information.

Advantages and features of the present invention, and a method for achieving them will become apparent with reference to the detailed description in conjunction with the accompanying drawings. However, it should be understood that the present invention is not limited to the embodiments presented below, but may be implemented in a variety of different forms, and includes all transformations, equivalents, and substitutes included in the spirit and scope of the present invention. . The embodiments presented below are provided to complete the disclosure of the present invention, and to completely inform those of ordinary skill in the art to the scope of the present invention. In describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented in various numbers of hardware and/or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors, or by circuit configurations for a given function. Also, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented as an algorithm running on one or more processors. Further, the present disclosure may employ prior art techniques for electronic configuration, signal processing, and/or data processing, etc. Terms such as "mechanism", "element", "means" and "configuration" will be used broadly. and is not limited to mechanical and physical configurations.

In addition, the connecting lines or connecting members between the components shown in the drawings only exemplify functional connections and/or physical or circuit connections. In an actual device, a connection between components may be represented by various functional connections, physical connections, or circuit connections that are replaceable or added.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

1 is an exemplary diagram for explaining that a vehicle recognizes a surrounding vehicle using a camera and a radar in a road environment.

Referring to FIG. 1 , a sensor fusion system that may be mounted on a vehicle 10 may identify situations around the vehicle based on external environment information recognized through various sensors and vehicle own information. For example, radar is relatively robust to external environments such as lighting and weather, and can directly acquire distance and speed, so that it is easy to acquire distance information between the vehicle and the vehicle. In addition, the radar can extract a Doppler component of an object, so that a stationary object and a moving object can be distinguished, and this Doppler spectrum can be used for object recognition or classification.

In addition, the camera mounted on the vehicle 10 may photograph the surrounding environment in real time and recognize the object included in the captured image in the form of a feature, thereby distinguishing the types of objects having various shapes and shapes.

Referring to FIG. 1 , in the sensor fusion system, the radar mounted on the vehicle 10 emits radio waves toward the surroundings and receives the reflected waves 26 , 27 , and 28 reflected by the surrounding vehicle 15 to determine the radar measurement points. can be obtained At this time, the acquired radar measurement points are, based on the target tracking algorithm of the vehicle radar, predetermined information necessary for the vehicle 10, for example, lateral distance information between the vehicle 10 and the surrounding vehicle 15, the surrounding vehicle ( 15) can be calculated.

Also, in the sensor fusion system, the camera mounted on the vehicle 10 may acquire the captured image 30 . As an example, the sensor fusion system extracts a feature class from the captured image 30 , determines how many vehicles are in the captured image 30 , and sets a binding box for each vehicle, thereby targeting nearby vehicles real-time tracking of

In addition, the sensor fusion system fuses the information acquired from the radar and the camera on the same coordinate system, so that the lateral distance information with the surrounding vehicle 15 located in the vicinity of the vehicle 10, coordinate information including the x-axis and the y-axis, Alternatively, data obtained by converting the corresponding coordinate information into a global coordinate system (x, y, z) may be obtained.

According to an embodiment of the present invention, the sensor fusion system generates a sensor fusion image by mapping the captured image acquired by the radar and the camera, and analyzes the radar measurement points in multiple versions to view the location information of the target. More precise positioning is possible.

2 is a block diagram of a sensor fusion system 100 according to an embodiment of the present invention. According to an example, the sensor fusion system 100 may include a radar 110 , a camera 120 , and a processor 130 .

The radar 110 may calculate lateral distance information with an object by emitting radio waves to the surroundings and detecting reflected waves that are reflected and incident on the object. At this time, the radar 110 may determine the state information of the vehicle through the antenna, the transceiver terminal, and the signal processing terminal. For example, the radar 110 may detect a distance, an angle, and a speed for each object using a transmit antenna and a plurality of receive antennas. Here, the radar 110 is relatively robust to the external environment, and can detect an accurate distance with the round trip time of radio waves. In addition, the radar 110 may directly acquire speed information of the surrounding vehicle by using the Doppler frequency reflected from the objects. However, since the angle detection performance of the radar 110 is low, all objects may be recognized as one point, and accordingly, a separate process for recognizing surrounding vehicles among the information acquired through the radar 110 may be required. .

The camera 120 may be mounted on at least one of the front, rear, and side of the vehicle, and may acquire a captured image and transmit it to the processor. Here, the camera 120 may extract a feature class from the captured image, set a binding box in the extracted feature group, and obtain tracking information for the corresponding object through real-time object recognition result update.

The processor may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processor by a storage medium or a communication module. For example, the processor may be configured to execute instructions received according to program code stored in a recording device such as a storage medium.

The processor 130 calibrates the radar 110 and the camera 120, and generates a sensor fusion image by matching the lateral distance information with the object acquired from the radar and the captured image, and the radar measurement points and the sensor. By analyzing the fusion image, it is possible to calculate the location of the surrounding vehicle included in the captured image.

Meanwhile, the sensor fusion system 100 according to an embodiment of the present invention includes a plurality of radars 110 and cameras 120 based on the installation positions of each radar 110 and each camera 120 . After calibration is applied, mapping of a one-to-many relationship is possible.

As an example, the processor 130 projects information obtained from the radar 110 into a camera coordinate system to map the lateral distance information between the vehicle obtained from the radar 110 and surrounding vehicles on the 2D image captured by the camera. can

In addition, the processor 130 targets the surrounding vehicle included in the first method of calculating the location information on the target estimated as the surrounding vehicle from a plurality of radar measurement points detected through the radar, and the sensor fusion image between the camera and the radar. , and through the second method of tracking the location information on the target, location information of each surrounding vehicle may be calculated.

As an example, the processor 130 acquires a plurality of radar measurement points by emitting radio waves from the radar, detecting reflected waves that are reflected and incident on an object, and selects the plurality of radar measurement points into a predetermined group based on a target detection probability. The first method of estimating the clustered group maintaining continuity as a neighboring vehicle may be performed by clustering the clusters to .

As another example, the processor 130 detects a surrounding vehicle by applying an object recognition algorithm to the sensor fusion image, sets a binding box for the detected surrounding vehicle area, and targets the area in which the binding box is set, A second method of tracking location information for a target may be performed.

Here, the processor 130 compares the previous image frame with the current image frame among the sensor fusion image to calculate continuous position information for the target reflecting the real-time position change of the target, thereby tracking the position information on the target. can

The processor 130 does not roughly estimate the location of the surrounding vehicle by just one method, but compares and analyzes several methods to select the optimal vehicle location information, so that high-precision positioning of the surrounding vehicle is possible.

The processor 130 may calculate a cost function using the center point of the target for each target estimated or recognized based on at least one of the first method and the second method.

For example, the processor 130 may be configured to calculate a first cost function obtained by summing horizontal distances between the virtual line and a plurality of radar measurement points based on an imaginary line perpendicular to the lateral center point of the target estimated by the first method. , and with each of the surrounding vehicles recognized by the second method as a target, based on an imaginary line corresponding to the heading direction Yaw of the vehicle passing through the center point of the binding box set for each target, the plurality of The cost function may be calculated by calculating a second cost function summing the horizontal distances between radar measurement points, and selecting a target having a minimum cost function value by comparing the first cost function and the second cost function. . In this case, the processor 130 may select an optimal cost function value by comparing the calculated cost functions for each target, and may calculate location information about the target based on the selected cost function.

On the other hand, the processor 130 calculates a predetermined curve curve by accumulating the previously determined location information of the target, estimates the location information of the target at the current time in consideration of the curve curve and the movable area of the vehicle, and Based on the virtual line corresponding to the heading direction (Yaw) of the vehicle passing through the center point of the target, a third cost function may be calculated by summing the horizontal distances between the virtual line and the plurality of radar measurement points. In this case, the processor 130 may further consider the third cost function and select a cost function having a minimum value among the first cost function, the second cost function, and the third cost function as the optimal cost function.

3 is a flowchart illustrating a method of calculating the location of a nearby vehicle in the sensor fusion system 100 according to an embodiment of the present invention.

In operation S110 , the sensor fusion system 100 may perform a first method of calculating location information on a target estimated as a nearby vehicle from a plurality of radar measurement points sensed through the radar. For example, it is possible to determine that a target exists at a specific location only by the reflected wave received through the radar, but it is not possible to determine which target exists at the corresponding location. However, if there are a plurality of radar measurement pointers, accurate location information on a target estimated as a nearby vehicle may be estimated by the first method as follows. In this case, the radar measurement pointers may estimate accurate coordinate values including longitudinal and lateral direction information between the vehicle including the radar and the surrounding vehicle on the two-axis coordinate system.

For example, the target tracking algorithm based on the radar 110 includes a Kalman filter, an extended Kalman filter, a probabilistic data-association filter (PDAF), an integrated probabilistic data-association filter (IPDAF), and a linear LM-IPDAF (Linear) filter. A future path may be estimated using the current and past paths of the target through filters or location tracking algorithms such as multi-target IPDAF), joint probabilistic data-association filter (JPDAF), and constant false alarm rate (CFAR).

According to an embodiment of the present invention, any method of the above-described filter or position tracking algorithm may be used, and the high level of precision required for an autonomous vehicle by a radar and a camera only by supplementing the prior art techniques make the output possible. According to the present invention, since location tracking with a high level of precision is possible by processing the information collected by the radar and the camera, it can replace expensive, high-precision equipment for recognizing a nearby vehicle. Although there may be several positioning techniques using radar, the following embodiment will be described with respect to one of them.

For example, the sensor fusion system 100 may calculate location information on a target estimated as a nearby vehicle through a first method, wherein the first method is a reflected wave that radiates a radio wave from a radar and is reflected by an object acquiring a plurality of radar measurement points by detecting estimating the clustered group maintaining continuity as surrounding vehicles by updating .

As an example, the sensor fusion system 100 uses a target tracking technology to t-1, t-2, . The location information of the target can be tracked by using the location information about the target acquired over time. In this case, the target is a real object having a predetermined volume in the three-dimensional coordinate system, but radar measurement points obtained through the radar may be acquired as predetermined points in the two-axis coordinate system. Here, the sensor fusion system 100 clusters a predetermined region estimated as a target through a linear model decision method (residual sum), and repeatedly performs prediction and correction on the clustered region to ensure continuity. Branches can distinguish clustering regions.

In addition, the information obtained through the radar in the sensor fusion system 100 may include the distance between the vehicle and the target (lateral distance information) and the speed information of the target, and one point of the clustering area is used as the location information of the target. can be estimated Here, the target may be a vehicle mainly located in the vicinity, and the sensor fusion system 100 may acquire location information and movement speed information of each of the surrounding vehicles located in the vicinity of the vehicle using radar through a multi-object tracking technology.

In step S120 , the sensor fusion system 100 may recognize a nearby vehicle included in the sensor fusion image between the camera and the radar as a target, and perform a second method of tracking location information on the target. For example, the sensor fusion image may be an image obtained by mapping vehicle-to-vehicle lateral distance information obtained from radar with an image on a two-dimensional coordinate system captured by a camera. Meanwhile, the sensor fusion image may be converted into a global coordinate system for an autonomous vehicle, or annotation information for additionally expanding and writing predetermined data in the corresponding coordinate system may be added.

As an example, the second method in the sensor fusion system 100 includes the step of detecting, by the sensor fusion system 100, an object recognition algorithm to the sensor fusion image to detect a surrounding vehicle, and binding to the detected surrounding vehicle area. It may include setting a box, and tracking location information about the target by using an area in which the binding box is set as a target. Here, the step of the sensor fusion system 100 tracking the location information on the target is a continuous location of the target reflecting the real-time location change of the target by comparing the previous image frame and the current image frame among the sensor fusion images. It may be a step of calculating information.

In operation S130 , the sensor fusion system 100 may calculate a cost function using a center point of the target for each target estimated or recognized based on at least one of the first method and the second method. For example, there may be several methods of calculating the location information of the surrounding vehicle based on information obtained by the sensor fusion system 100 using any one of a radar and a camera. For high-precision positioning, the sensor fusion system 100 may calculate location information of surrounding vehicles when multiple methods are applied, calculate a cost function for this, and compare and analyze the utility of each method. For example, the sensor fusion system 100 may perform the following cost function calculation for comparison between a plurality of methods.

As an example, the sensor fusion system 100 calculates a first cost function by summing the horizontal distance between the virtual line and the plurality of radar measurement points based on the virtual line perpendicular to the lateral center point of the target estimated by the first method. can be calculated For example, a plurality of radar measurement points are acquired, and a longitudinal imaginary line perpendicular to the lateral center point of the target and each radar measurement for position comparison between each laser measurement point within a predetermined area clustered through the first method The horizontal distance between points can be calculated. In this case, the sensor fusion system 100 may calculate an index of how many radar measurement points are concentrated based on the vertical virtual line as a cost function by summing the horizontal distances of the plurality of radar measurement points.

As another example, the sensor fusion system 100 targets each of the surrounding vehicles recognized by the second method, based on an imaginary line corresponding to the heading direction Yaw of the vehicle passing through the center point of the binding box set for each target. , a second cost function obtained by summing the horizontal distances between the plurality of radar measurement points may be calculated. The virtual line corresponding to the heading direction Yaw of the vehicle and the aforementioned vertical virtual line have different names, but are common in that they are virtual lines passing through the center point of the target set by each method in the sensor fusion system 100 . can be

As another example, the sensor fusion system 100 calculates a predetermined curve curve by accumulating previously determined location information of the target, and predicts the location information of the target at the current time in consideration of the curve curve and the movable area of the vehicle calculating a third cost function summing the horizontal distance between the virtual line and the plurality of radar measurement points based on the virtual line corresponding to the heading direction (Yaw) of the vehicle passing through the expected center point of the target may include the step of

As an example, the sensor fusion system 100 plots frame time points (t, t-1, t-2, ...) of the captured image on the X-axis and lateral distance information on the Y-axis, From the recording of , it is possible to estimate the lateral distance information of the target at the current time t corresponding to a predetermined curve curve. However, in this case, the degree of the predetermined curve curve may be adjusted in consideration of the tracking result of whether the corresponding surrounding vehicle goes straight, left/right turn, or lane change in consideration of driving lane information of the surrounding vehicle. For example, the sensor fusion system 100 collects the lateral distance information accumulated from the past t-nth frame to the t-1th frame in the left lane (1st lane) of the surrounding vehicle, the corresponding lane (2nd lane), the right lane ( Three prediction frames (1st lane, 2nd lane, 3rd lane) in the t-th frame at the current time point can be obtained through the predicted value of the curve curve predicted using the lateral distance information accumulated for the section (3 lane). .

As described above, the lateral distance information predicted for the t-th frame from the past record enables better tracking of the lateral distance information of the corresponding vehicle even if the corresponding neighboring vehicle changes lanes, and Records are updated in real time, enabling a precise level of continuous location tracking. That is, as the surrounding vehicle location method according to the present invention aims for real-time tracking of the surrounding vehicle, accurate location information tracking is possible even at the moment when the surrounding vehicle changes lanes.

In step S140 , the sensor fusion system 100 may compare and select a cost function for each calculated target and calculate location information including lateral distance information for the selected target. For example, the sensor fusion system 100 may calculate a plurality of cost functions through various methods. In this case, the sensor fusion system may select an optimal cost function by comparing several cost functions, and may calculate location information of the selected target based thereon.

For example, the sensor fusion system 100 may select a target having a minimum cost function value by comparing the first cost function and the second cost function, and calculate location information about the target. As another example, when the first to third cost functions are calculated, the sensor fusion system 100 may select a target having a minimum cost function value based on the calculated first cost function to the third cost function, and may calculate location information for the target. In this case, the location information of each target may include lateral distance information obtained through radar.

As described above, the sensor fusion system 100 compares and analyzes the cost function calculated by various methods, and tracks the current location of the surrounding vehicle in real time with higher accuracy than when the surrounding vehicle is located only by one method, respectively. can do.

4 is an exemplary diagram for explaining calculating a cost function through various methods according to an embodiment of the present invention.

Referring to FIG. 4 , the sensor fusion system 100 may acquire captured images 210 , 220 , and 230 through the camera 200 . Here, the camera 200 may be mounted on any one or more points of the front, rear, and side of the vehicle, and may acquire the captured images 210 , 220 , 230 from the camera through a calibration process for a plurality of cameras. .

Meanwhile, the sensor fusion system 100 may emit radio waves to a predetermined area through the radar 310 , and collect reflected waves that are reflected from the object and return ( 320 ). In this case, information obtained from the collected reflected wave may be referred to as a radar measurement point. The radar measurement point may include coordinates at which the object is estimated, lateral distance information between the vehicle and the object, speed information of the object, and the like.

In addition, the sensor fusion system 100 may perform a process of recognizing a target as a surrounding vehicle or calculating accurate location information of the surrounding vehicle through predetermined processing on the radar measurement points acquired through the radar 310 . have.

On the other hand, the sensor fusion system 100 is a sensor fusion image in which each captured image 210 , 220 , 230 obtained from the camera 200 and the radar 310 and the radar measurement point 320 are mapped on the same coordinate system. can be obtained (330). In this case, the sensor fusion image may include lateral distance information between a vehicle and a surrounding vehicle processed from radar measurement points in a two-dimensional coordinate system that is an image coordinate system.

The sensor fusion system 100 may calculate location information of surrounding vehicles through various methods, and may calculate a cost function for comparing and analyzing each method. For example, the sensor fusion system 100 estimates the location information of the target based on the radar measurement points 320 based on the target tracking algorithm of the radar 110 and uses the radar measurement points according to a predetermined process. A first cost function may be calculated ( 340 ).

As another example, the sensor fusion system 100 interprets the acquired sensor fusion image, and uses a binding box for a target extracted from the captured image and radar measurement points around the binding box according to a predetermined process according to a second cost function. can be calculated (350).

The sensor fusion system 100 compares the first cost function and the second cost function and selects the minimum cost function value, so that one of the location values of the surrounding vehicle location information by the first method or the second method is optimally selected. It can be determined based on the location information of the surrounding vehicle.

Meanwhile, according to another embodiment, the optimal surrounding vehicle location information determined through the above-described process generates a new predicted value from past data when determining the surrounding vehicle location information in the next frame, and generates the generated information according to a predetermined process. By calculating the third cost function for the predicted value, the first cost function, the second cost function, and the comparative example may be further generated.

For example, the sensor fusion system 100 accumulates surrounding vehicle location information determined as an optimal value in past (t-1, t-2, t-3, ??) frames as past data, and from this, It is possible to generate a predicted value of the position of the surrounding vehicle in the frame t. In this case, the sensor fusion system 100 further considers the possibility that the surrounding vehicle changes lanes between the first lane and the third lane for more accurate positioning of the current frame. Three more simulation values can be predicted for (t). This will be described later with reference to FIGS. 10A to 10C.

Meanwhile, the sensor fusion system 100 selects at least one cost function having a minimum cost function value from among the first cost function, the second cost function, and the third cost function, and sets the value positioned by the selected method to the optimal surrounding vehicle. It may be determined based on location information ( 380 ).

5 is an exemplary diagram for comparing and explaining a coordinate system used for each step according to an embodiment of the present invention. According to an embodiment of the present invention, since each sensor (radar, camera, etc.) included in the sensor fusion system 100 collects data based on a separate coordinate system, it is mapped to a single sensor fusion image or a global coordinate system. A conversion process may be required.

Referring to FIG. 5 , the radar coordinate system 400 is a coordinate system relative to the surrounding vehicle 15 with respect to the vehicle 10 on which the radar is mounted. . Referring to FIG. 5 , the surrounding vehicle 15 may be displayed as relative coordinate values (x, y) when viewed from the vehicle 10 . In this case, the lateral distance information may be calculated as linear distance information between the vehicle 10 and the surrounding vehicle 15 based on the coordinate values (x,y).

Referring to FIG. 5 , the camera coordinate system 410 uses a 3-axis coordinate system of the x-axis to the z-axis using a Cartesian coordinate system 410, but the image taken from the camera is an image plane (Image plane, 425). may be recorded using the pixel coordinate system 420 of the Accordingly, the lateral distance information between the vehicle and the surrounding vehicle cannot be clearly known only from the captured image taken by the camera, but the sensor fusion system 100 according to an embodiment of the present invention provides the lateral distance obtained through the radar. By mapping the information to the captured image, it is possible to generate a sensor fusion image further including lateral distance information in addition to 2D coordinates.

On the other hand, the information obtained from the sensing fusion system 100 as described above is a global coordinate system ( 430).

6A to 6B are diagrams exemplarily illustrating location information of a target estimated based on radar measurement points acquired through a radar and a target tracking algorithm.

According to an example, when the radar collects reflected waves reflected from an object and corresponds them on an image plane, a view as shown in FIG. 6A may appear. The radar measurement points 501 to 508 may be indicators indicating that there is an object at a specific location.

When the sensor fusion system 100 acquires a plurality of radar measurement points 501 to 508, the clustering area 500 may be set by estimating the location information of the target through predetermined clustering based on the probability that the target exists. . The sensor fusion system 100 may distinguish the clustering region 500 from other regions by updating the clustering region through prediction and correction for a predetermined period for the clustering region 500 . In this case, the clustering region may be a predetermined region having a circle, an ellipse, or an irregular region rather than a rectangular binding box as shown in FIG. 5 .

Referring to FIG. 6A , the sensor fusion system 100 may estimate ( 550 ) the location of the target based on a predetermined target tracking algorithm. In this regard, a target tracking algorithm will be described with reference to FIG. 6B below.

6B is an exemplary diagram for explaining a target tracking algorithm of a radar according to an embodiment of the present invention.

For example, a target to be estimated in the estimation algorithm may be referred to as a target, and X, which is a state vector of the target, is a state variable composed of a relative position, a relative velocity, and a relative acceleration. Here, the sensor fusion system 100 predicts the current state vector of the preceding vehicle from the past state vector of the preceding vehicle, and updates the state vector of the preceding vehicle using radar measurement points reflected and incident from the preceding vehicle. (Update) The target can be tracked using the dynamic model.

Referring to FIG. 6B , the sensor fusion system 100 may predict a state vector prediction value 600 of the preceding vehicle based on a predetermined target tracking algorithm. Here, the sensor fusion system 100 sets the gate 650 calculated with a constant probability at a predetermined distance around the state vector predicted value 600 , and the radar measurement points 610 and 620 entered in the gate 650 . ) can be used for updates. As an example, the sensor fusion system 100 filters the gate 650 region using a Kalman filter, and includes a measurement radar point z ₁ (k) 610 and another measurement radar point z ₂ (k). 620 can be used to update the corresponding predicted value 600. According to an embodiment, the sensor fusion system 100 calculates the predicted value of the state variable of the preceding vehicle by using a discrete-time dynamics model in a state space. can be calculated. For example, the sensor fusion system calculates the state vector 600 X(k-1) of the k-1th cycle of the preceding vehicle, the error covariance of the prediction error, and the gain of the filter (Kk, gain), and the radar By updating the state variable of the preceding vehicle through the measurement points z1(k) 610 and z2(k) 620 , the predicted state vector X(k|k-1) of the current time point in which the error is corrected is calculated. can do.

On the other hand, the sensor fusion system 100 may represent the state vector predicted value 600 of the preceding vehicle as the target detection probability, and the target location information can be estimated using a linear model discrimination method (Residual Sum) based on the target detection probability. have.

In this case, the sensor fusion system 100 may perform multi-tracking for tracking location information for each target when there are a plurality of vehicles in the vicinity.

7 is a view for explaining obtaining location information of a surrounding vehicle through a camera according to an embodiment of the present invention. Referring to FIG. 7 , a plurality of cameras may be mounted in a vehicle, and a plurality of captured images 710 , 720 , and 730 may be acquired according to positions of the respective cameras. As an example, the sensor fusion system 100 may determine the position of each camera in the vehicle through calibration for a plurality of cameras, and recognize an object included in the images 710 , 720 , 730 captured by each camera. have.

Referring to FIG. 7 , captured images 710 , 720 , and 730 of a plurality of viewpoints may be acquired through a camera mounted on a vehicle. For example, the sensor fusion system 100 may classify objects included in each image through a feature class based on an image interpretation algorithm, and recognize surrounding vehicles based on this.

For example, the sensor fusion system 100 recognizes surrounding vehicles included in the front left image 710, the front central image 720, and the front right image 730, and , it can be displayed superimposed in the captured image in the form of a binding box.

FIG. 8 is an example of analyzing radar measurement points obtained by the first method and FIG. 9 is an example of calculating a cost function based on the analysis points. 8 and 9 , the radar measurement points are the same (r01 to r06, R01 to R06). However, according to an embodiment of the present invention, in order to obtain more precise location information including the center point of the target, the radar measurement points may be analyzed based on different methods.

8 is an exemplary view for explaining a method of calculating a first cost function according to an embodiment of the present invention. The sensor fusion system 100 includes a first method for directly calculating the position information of a target estimated as a surrounding vehicle from the radar measurement points r01 to r06, and a lateral center point of the target estimated by the first method. A first cost function obtained by summing horizontal distances between the virtual line and the plurality of radar measurement points may be calculated based on the virtual line.

-----------------------------------수학식 (1)First cost function:

-----------------------------------Equation (1)

Among the plurality of radar measurement points, if about 6 radar measurement points are selected as shown in FIG. 8 , then the first cost function is d01 + d02 + d03 + d04 which is the horizontal distance between the virtual line 850 and each radar measurement point. It can be calculated as + d05 + d06.

9 is an exemplary view for explaining a method of calculating a second cost function according to another embodiment of the present invention. The sensor fusion system 100 may recognize a surrounding vehicle through object recognition from the sensor fusion image, and set a binding box 900 for the surrounding vehicle. At this time, the sensor fusion system 100 includes an imaginary line 950 corresponding to the heading direction Yaw of the vehicle passing through the center point of the binding box 900 , the imaginary line 950 , and the radar measurement points R01 . The sum of the horizontal distances to R06) can be calculated as a second cost function.

-------------------------------------수학식(2) A second cost function:

-------------------------------------Equation (2)

Among the plurality of radar measurement points, if about 6 radar measurement points are selected as shown in FIG. 9 , then the second cost function is D01 + D02 + D03 + D04 which is the horizontal distance between the virtual line 950 and each radar measurement point. It can be calculated as + D05 + D06.

8 and 9, since the center point of the target set by the sensor fusion system 100 in each method and an imaginary line perpendicular thereto (or imaginary line corresponding to the heading direction of the vehicle) are applied differently, based on this The calculated cost function may be different. In this case, when the cost functions calculated by each method are compared, it can be seen that more radar measurement points are distributed at a closer distance from the virtual line than when the comparison function has a minimum value. Based on this, the sensor fusion system 100 may calculate accurate location information about the target based on the center point of the target, which is the basis of the comparison function having the minimum value.

10 to 11 are exemplary diagrams for explaining prediction of target location information of a current view from past target location information.

10 is a t-th frame (current) of the vehicle 10 on which the sensor fusion system 100 is mounted, and a position in the t-1 th frame, at which time the surrounding vehicle 15 is at each time point (t, t-1). , t-2, ??) is exemplarily shown on the same plane. According to this, the surrounding vehicle 15 drives in the second lane in the t-2 th frame and attempts a lane change in the t-1 th frame to change the lane to the first lane in the t th frame. That is, since the surrounding vehicle 15 also has location information that varies dynamically in the road driving environment, if a predicted value for the motion of the surrounding vehicle 15 can be calculated, faster and more precise location information can be calculated based on the predicted value. can

In addition, the sensor fusion system 100 in the vehicle 10 provides lateral distance information Rt between the vehicle 10 and the surrounding vehicle 15 in the t-th frame, and the vehicle 10 and the surrounding vehicle 15 in the t-1 frame. ), the lateral distance information R(t-1) can be calculated.

11 is a graph for explaining prediction of current target location information from past target location information. In this case, the past target location information may be location information of a nearby vehicle located by a method having an optimal cost function among the above-described several methods.

The sensor fusion system 100 may calculate a predetermined curve curve by accumulating previously determined target position information. Even at this time, since the target, which is a nearby vehicle, can move at any time, it is assumed that the path on which the vehicle can move, for example, the location of the t-1th frame of the surrounding vehicle is the second lane, and the probability that the surrounding vehicle moves to the first lane or 3 Several versions of the simulation may be performed based on the probability of moving into a lane or the like.

Referring to FIG. 11 , for a two-dimensional graph in which an image frame corresponding to the time axis is set as the x-axis and lateral distance information is set as the y-axis, the surrounding vehicles maintain a straight lane (two lanes), respectively. The predicted value at the current time t of the surrounding vehicle may vary depending on the case 1020 , the case of moving to the first lane 1010 , the case of moving to the third lane 1030 , and the like. The sensor fusion system 100 may calculate lateral distance information for predicting the location information of the target at the current time in consideration of the curve curve and the movable area of the vehicle.

On the other hand, the sensor fusion system 100 adds up the horizontal distance between the virtual line and the plurality of radar measurement points based on the virtual line corresponding to the heading direction (Yaw) of the vehicle passing through the expected center point of the target. The method may further include calculating a third cost function. If the sensor fusion system 100 performs three simulations in consideration of the possibility of lane change of surrounding vehicles as shown in FIG. 11 , three versions of the third cost function may be further calculated as described above. In this case, the sensor fusion system 100 compares the first cost function, the second cost function, and the three third cost functions, selects a cost function having a minimum cost function value among a total of five cost functions, and uses the method according to the method. It is possible to determine the position information of the surrounding vehicle calculated based on the final position information.

Embodiments according to the present invention may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded in a computer-readable medium. In this case, the medium includes a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and a ROM. , RAM, flash memory, and the like, hardware devices specially configured to store and execute program instructions.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and used by those skilled in the computer software field. Examples of the computer program may include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

According to an embodiment, the method according to various embodiments of the present disclosure may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store (eg Play Store™) or between two user devices. It can be distributed directly or online (eg, downloaded or uploaded). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

The steps constituting the method according to the present invention may be performed in an appropriate order, unless the order is explicitly stated or there is no description to the contrary. The present invention is not necessarily limited to the order in which the steps are described. The use of all examples or exemplary terms (eg, etc.) in the present invention is merely for the purpose of describing the present invention in detail, and the scope of the present invention is limited by the examples or exemplary terms unless defined by the claims. it's not going to be In addition, those skilled in the art will appreciate that various modifications, combinations, and changes may be made in accordance with design conditions and factors within the scope of the appended claims or their equivalents.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the spirit of the present invention is not limited to the scope of the scope of the present invention. will be said to belong to

10: vehicle
15: surrounding vehicles
100: sensor fusion system
110: radar
120: camera
130: processor

Claims (9)

In the method of calculating the position of the surrounding vehicle,
a first method of calculating location information on a target estimated as a nearby vehicle from a plurality of radar measurement points sensed through a radar;
a second method of recognizing a surrounding vehicle included in a sensor fusion image between a camera and a radar as a target, and tracking location information about the target;
calculating a cost function using a center point of the target for each target estimated or recognized based on at least one of the first method and the second method; and
Comparing the calculated cost functions for each target, selecting an optimal cost function value, and calculating location information on the target based on the selected cost function;
Calculating the cost function comprises:
calculating a first cost function summing the horizontal distances between the virtual line and the plurality of radar measurement points based on an imaginary line perpendicular to the lateral center point of the target estimated by the first method;
Based on the virtual line corresponding to the heading direction (Yaw) of the vehicle passing through the center point of the binding box set in each target by targeting the surrounding vehicle recognized by the second method, the virtual line and the plurality of radar measurement points calculating a second cost function summing the horizontal distances between them; and
comparing the first cost function and the second cost function to select a target having a minimum cost function value. The method of claim 1,
The first method is
acquiring a plurality of radar measurement points by emitting radio waves from the radar and detecting reflected waves that are reflected and incident on an object;
clustering the plurality of radar measurement points into a predetermined group based on a target detection probability;
estimating the clustered group maintaining continuity as surrounding vehicles by updating the clustered group using a plurality of continuously acquired radar measurement points. According to claim 1,
The second method is
detecting a surrounding vehicle by applying an object recognition algorithm to the sensor fusion image;
Setting a binding box for the surrounding vehicle area, and tracking the location information on the target with the area in which the binding box is set as a target; 4. The method of claim 3,
The step of tracking the location information for the target,
Comparing a previous image frame and a current image frame among the sensor fusion image, calculating continuous position information about the target reflecting the real-time position change of the target, the method. delete The method of claim 1,
Calculating the cost function comprises:
calculating a predetermined curve curve by accumulating previously determined target position information;
estimating the location information of the target at the current time in consideration of the curve curve and the movable area of the vehicle;
Based on the virtual line corresponding to the heading direction (Yaw) of the vehicle passing through the expected center point of the target, calculating a third cost function summing the horizontal distance between the virtual line and the plurality of radar measurement points further Including method. In the radar and camera-based sensor fusion system,
a radar that emits radio waves to the surroundings and detects reflected waves that are reflected and incident on the object, thereby calculating lateral distance information with the object;
a camera mounted on at least one of the front, rear, and side of the vehicle, the camera acquires the captured image and transmits it to the processor;
calibrate the radar and camera;
generating a sensor fusion image by matching the lateral distance information with the object acquired from the radar and the captured image,
A processor that calculates a location of a nearby vehicle by analyzing the radar measurement points and the sensor fusion image; including a sensor fusion system 8. The method of claim 7,
Calculating the location of the surrounding vehicle by the processor,
a first method of calculating location information on a target estimated as a nearby vehicle from a plurality of radar measurement points sensed through a radar;
a second method of recognizing a surrounding vehicle included in a sensor fusion image between a camera and a radar as a target, and tracking location information about the target;
calculating a cost function using the center point of the target for each target estimated or recognized based on at least one of the first method and the second method;
Comparing the calculated cost functions for each target, selecting an optimal cost function value, and calculating location information about the target based on the selected cost function,
Calculating the cost function is
Based on an imaginary line perpendicular to the lateral center point of the target estimated by the first method, calculate a first cost function summing the horizontal distance between the imaginary line and a plurality of radar measurement points,
Based on the virtual line corresponding to the heading direction (Yaw) of the vehicle passing through the center point of the binding box set in each target by targeting the surrounding vehicle recognized by the second method, the virtual line and the plurality of radar measurement points Calculate a second cost function summing the horizontal distances between the
and selecting a target having a minimum cost function value by comparing the first cost function and the second cost function. A computer-readable recording medium in which a program for executing the method of claim 1 in a computer is recorded.

Images