KR20230072126A - System and method for driving control through fusion of sensors between vehicles - Google Patents

Abstract

The present invention relates to a driving control system through sensor fusion between vehicles and a method thereof. According to the present invention, sensor data is shared between the vehicles and the sensor data are fused to match the sensor data with an adjacent vehicle. Therefore, recognition of surroundings is improved, and driving is controlled according to environment change of the surroundings and a driving status of other vehicles. As a result, the sensor data from the other vehicles is received. Additionally, coordinates of the sensor data of the matched vehicle are converted and fused with own vehicle sensor information. Therefore, recognition performance and accuracy for the surrounding environment and an object are improved, and autonomous driving is performed stably. Additionally, stability of the vehicle is improved. The driving control system through the sensor fusion between the vehicles of the present invention comprises: a camera unit; a sensing unit; a video recognizing unit; a communicating unit; a data processing unit; a coordinates converting unit; and a controlling unit.

Description

Driving control system and method through sensor fusion between vehicles

The present invention shares sensor data through vehicle-to-vehicle communication, recognizes at least one adjacent vehicle using the fused sensor data, and controls driving in response to the sensor data and the recognized vehicle's driving through sensor fusion between vehicles. It relates to a driving control system and method therefor.

An autonomous vehicle is a vehicle that drives to a given destination by itself by recognizing the surrounding environment, determining the driving situation, and controlling the vehicle without driver intervention.

Vehicles perform autonomous driving by controlling the driving direction by controlling the steering angle based on map information and acquired information. A vehicle acquires other vehicles or road information located nearby while driving using a plurality of sensors, cameras, location information, etc., and controls driving based on the acquired data.

Since a car does not drive alone on the road, it is necessary to determine the location of surrounding vehicles and control driving while preventing collisions between vehicles according to changes in the location of surrounding vehicles.

In general, if location recognition through HDMap (High-Definition Map) and GPS is precise, an autonomous driving system can be implemented in a car without a sensor by sharing coordinate information of each object in real time through V2X (Vehicle to anything).

However, the HDMap maintenance process cannot occur in real time, and location recognition is difficult due to GPS coordinate error and shaded areas.

Moreover, although it is necessary to stably share information about objects in real time using V2X in various environments, there is a limit to stability and information about vehicles produced in the past cannot be known. Therefore, most autonomous vehicles recognize the surrounding environment through sensors and drive autonomously to control

Autonomous driving systems applied to automobiles generally recognize the surrounding environment using ultrasonic sensors, cameras, RADAR, and LiDAR.

Ultrasonic sensors, RADAR, and LiDAR transmit ultrasonic waves, electromagnetic waves, and light into the air, and then process the reflected signal to determine whether there are obstacles around. Therefore, there is a problem in that recognition performance changes according to a change in a medium in a driving situation.

Similar to human visual information, the camera can provide road information such as lanes as well as obstacles. However, it is not easy to recognize all of the various patterns appearing as a combination of an object and a background, and there is a problem in that recognition performance is degraded, especially when the object and the background are similar.

These self-driving cars are not yet perfect like humans driving normally, so unexpected situations may occur partially.

Korean Patent Registration No. 10-2205299 discloses a method for changing lanes by sharing data between vehicles. This technology describes a technology for moving a lane by predicting the position of another vehicle in consideration of communication delay time by transmitting and receiving a safety message through vehicle-to-vehicle communication.

However, there is a limit to the stability of shared data, and there is a limit because it does not recognize surrounding vehicles by reflecting data on adjacent vehicles within a certain distance.

Therefore, in order to overcome the limitations of recognition performance of a single sensor, an autonomous driving system needs a method of fusing information obtained by processing each sensor signal, generating higher reliability information, and using it for vehicle control.

Republic of Korea Patent No. 10-2205299

The present invention was created due to the above needs, and a driving control system through sensor fusion between vehicles that recognizes the surrounding environment by sharing sensor data between vehicles, fusing sensor data, and matching sensor data with adjacent vehicles, and Its purpose is to provide that method.

The present invention increases the number of sensors used for recognizing the driving environment through the fusion of sensor data, improves the recognition performance of the surroundings, and controls the driving according to changes in the surrounding environment and the driving state of other vehicles. Sensor fusion between vehicles Its purpose is to provide a driving control system and method through

In order to achieve the above object, a driving control system through vehicle-to-vehicle sensor fusion according to the present invention includes a camera unit including a plurality of cameras to capture images around the vehicle; a sensing unit that measures the position of the vehicle and detects objects around the vehicle; an image recognition unit for analyzing an image captured by the camera unit, recognizing a license plate of a second vehicle located nearby, and extracting a license plate number; A communication unit that communicates with a plurality of vehicles located within a certain distance, including the sensor data and vehicle number obtained from the sensor data and the V2X (Vehicle to anything) method in an option field of a safety message (BSM, Basic Safety Message); a data processing unit matching the plurality of vehicles and the plurality of safety messages based on at least one of the vehicle number and the location and operation of the plurality of vehicles; a coordinate conversion unit extracting sensor data from the safety message of the second vehicle for which matching has been completed and converting coordinates based on the vehicle; and a control unit controlling driving based on information about surroundings of the vehicle obtained by merging data of the sensing unit and sensor data of the second vehicle based on a matching result of the second vehicle. includes

The data processing unit compares the vehicle number of the second vehicle with the vehicle numbers included in the plurality of safety messages, matches a safety message having the same vehicle number with the second vehicle, and selects a number plate among the plurality of vehicles. For an unrecognizable third vehicle, the location and operation of the plurality of vehicles are analyzed based on the information included in the plurality of safety messages, and compared with data of the third vehicle detected through the sensor unit, Characterized in that the third vehicle is matched.

The data processing unit compares at least one of location information, operation information, and object information of the third vehicle detected through the sensor with data included in the plurality of safety messages to generate a safety message that matches within an error range. It is characterized by matching with a third vehicle.

The data processing unit creates and registers a new track for the matched vehicle, changes a matching candidate that has successfully matched for a certain period of time among at least one matching candidate included in the previously matched data to a confirmed match, and Among the confirmed matches, a confirmed match that has not been matched for a certain period of time is converted into a matching candidate, and a track for a matching candidate that has not been matched for the certain period of time is deleted from among the at least one matching candidate, so that the plurality of vehicles are matched. It is characterized in that it manages the matching result for.

The coordinate conversion unit acquires the coordinates of the fourth vehicle detected by the second vehicle from sensor data included in the safety message of the second vehicle, and converts the coordinates of the fourth vehicle based on the own vehicle. to be characterized

When the coordinate conversion unit includes a geomagnetic sensor, based on the traveling direction angle of the vehicle based on the north (N pole) detected by the geomagnetic sensor and the traveling direction angle of the second vehicle, in the second vehicle Coordinate conversion of the detected coordinate value of the fourth vehicle, and when the geomagnetic sensor is not included, the coordinate conversion of the fourth vehicle based on a difference between a lane and a driving direction of a lane in which each vehicle is traveling Characterized in that do.

A method of operating a driving control system through inter-vehicle sensor fusion according to the present invention includes the steps of photographing an image around a vehicle through a plurality of cameras, and sensing a location of the vehicle and an object around the vehicle; Recognizing the license plate of the second vehicle from the image and extracting the vehicle number; In the optional field of the safety message (BSM, Basic Safety Message), communicating with a plurality of vehicles within a certain distance, including sensor data and vehicle number, in a V2X (Vehicle to anything) method; matching the plurality of vehicles and the plurality of safety messages based on at least one of the vehicle number, locations and operations of the plurality of vehicles; extracting sensor data from the safety message of the second vehicle for which matching has been completed, and converting coordinates based on the vehicle; obtaining information about surroundings of the vehicle by fusing sensor data of the host vehicle and sensor data of the second vehicle based on a matching result of the second vehicle; and autonomous driving based on the acquired information. includes

The matching of the plurality of vehicles and the plurality of safety messages may include comparing the vehicle number of the second vehicle whose license plate is recognized with the vehicle number included in the plurality of safety messages; and matching a second safety message whose vehicle number matches the second vehicle with the second vehicle. more includes

The matching of the plurality of vehicles and the plurality of safety messages may include: sensing location, motion, and lane information of a third vehicle whose license plate is not recognized using a plurality of sensors provided in the sensing unit; analyzing at least one of location, operation, and lane information of the plurality of vehicles based on data included in the plurality of safety messages; setting at least one matching candidate estimated to be the safety message of the third vehicle; and comparing data of the third vehicle with the matching candidate, and matching a third safety message matching the data of the third vehicle with the third vehicle within an error range. more includes

generating and registering a new track with respect to the matched third vehicle; changing a matching candidate that has successfully matched for a certain period of time among at least one matching candidate included in pre-matched data to a confirmed match; converting a confirmed match that has not been matched during the predetermined period among at least one confirmed match from the pre-matched data into a matching candidate; and deleting a track of a matching candidate that has not been matched for the predetermined period of time among the at least one matching candidate. It is characterized in that it further includes, and manages matching results based on the accumulated data.

The converting of the coordinates may include: acquiring coordinates of a fourth vehicle sensed by the second vehicle from sensor data included in the safety message of the second vehicle; If a geomagnetic sensor is included, calculating a traveling direction angle of the vehicle and a traveling direction angle of the second vehicle based on north (N pole), detected by the geomagnetic sensor; Compensating for the rotation by rotating an angle of the direction of travel of each vehicle by a difference in a rotation axis with respect to the fourth vehicle; compensating for the transformation using a distance between the vehicle and the second vehicle, a distance between the second vehicle and the fourth vehicle, and a rotation angle with respect to the fourth vehicle; and converting coordinate values of the fourth vehicle sensed by the second vehicle using rotation compensation and change compensation. more includes

The converting of the coordinates may include: acquiring coordinates of a fourth vehicle sensed by the second vehicle from sensor data included in the safety message of the second vehicle; calculating a difference between a lane of a lane in which each vehicle is driving and a driving direction when the geomagnetic sensor is not included; calculating a change in the tangent slope of the lane; Calculating a rotation angle using a difference between the lane and the driving direction and a change in the slope of the tangential line of the lane; and converting the coordinate values of the fourth vehicle detected by the second vehicle in correspondence with the rotation angle. more includes

The communicating may include sending the safety message including sensor data including at least one of a lane detected by a plurality of sensors, another vehicle, an object, a traffic signal, and traffic information and a vehicle number in the option field to the plurality of sensors. Characterized in that it is received from the vehicle.

According to one aspect, the driving control system and method through vehicle-to-vehicle sensor fusion of the present invention have an effect of improving recognition performance for the surrounding environment and objects by fusion and using sensor data of different vehicles.

According to one aspect of the present invention, by converting the coordinates of the sensor data of the matched vehicle and fusing with the sensor data of the own vehicle, it is possible to easily recognize a vehicle or object located in the surroundings, improve the accuracy of recognition, and improve collision, etc. It has the effect of preventing accidents and improving vehicle stability.

According to the present invention, since the number of sensor data that can be fused increases according to the number of adjacent vehicles, the recognizable range is expanded, stable autonomous driving is possible, and recognition data having high reliability can be generated, so the level of autonomous driving can be raised.

1 is a block diagram briefly illustrating the configuration of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.
2 is a reference diagram for describing data used for inter-vehicle sensor fusion according to an embodiment of the present invention.
3 is an exemplary diagram referenced to describe a vehicle recognition method of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.
4 is a flowchart illustrating a sensor fusion method of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.
5 is a reference diagram for explaining vehicle matching using location information of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.
6 is a flowchart illustrating a vehicle matching method using location information of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.
7 is a flowchart illustrating a method of managing matched data of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.
8 is an exemplary view referenced to describe coordinate conversion of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.
9 is an exemplary view illustrating an example of vehicle matching of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.
10 is a flowchart illustrating a driving control method of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.

Hereinafter, the present invention will be described with reference to the accompanying drawings.

In this process, the thickness of lines or the size of components shown in the drawings may be exaggerated for clarity and convenience of explanation. In addition, terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of a user or operator. Therefore, definitions of these terms will have to be made based on the content throughout this specification.

1 is a block diagram briefly illustrating the configuration of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.

As shown in FIG. 1, the driving control system includes a camera unit 170, a sensing unit 160, a data unit 180, a communication unit 150, an image recognition unit 120, a data processing unit 130, and coordinates. It includes a conversion unit 140 and a control unit 110 that controls the overall operation.

The driving control system further includes a driving unit (not shown) for controlling a power source such as a motor (not shown), a battery management system (not shown) for managing a battery, a brake system (not shown), and a steering system (not shown). However, a description thereof will be omitted below.

The sensing unit 160 includes a plurality of sensors to detect an object adjacent to the vehicle, for example, another vehicle, and to detect the surrounding environment and road information of the vehicle.

The sensing unit 160 includes a position sensor 161 for detecting the position of the vehicle and an obstacle sensor 162 for detecting surrounding objects.

The position sensor 161 includes a GPS sensor, and is provided with a sensor capable of detecting the current position of the vehicle through communication with other devices. The position sensor 161 may include a sensor that detects a position by analyzing a signal transmitted and received using the UWB method.

The obstacle sensor 162 detects an object located within a predetermined distance from the vehicle. The obstacle sensor 162 includes at least one of an infrared sensor, an ultrasonic sensor, a laser sensor, RADAR, and LiDAR. The obstacle sensor 162 may measure a distance to a detected object.

In addition, the sensing unit 160 may detect the driving state of the vehicle and vehicle information by including a vehicle speed sensor, a temperature sensor, a pressure sensor, and the like.

The camera unit 170 includes a plurality of cameras 171 and 172 to capture images of the surroundings of the vehicle. The camera unit 170 includes a first camera 171 installed adjacent to one side of the front of the vehicle or adjacent to the windshield to capture the driving direction of the front, and a second camera 172 installed at the rear of the vehicle to capture rear images. includes

Also, the camera unit 170 may generate a surround view of the surroundings of the vehicle 10 using a plurality of cameras.

The camera unit 170 applies the captured image to the controller 110 and stores the image data in the data unit 180 . The camera unit 170 may further include an image processing unit (not shown) that converts an image signal of the camera into image data in a predetermined format.

The data unit 180 stores control data for vehicle control, data detected through the sensor 160, data transmitted and received through the communication unit 150, and data related to captured images.

The communication unit 150 includes a plurality of communication modules and performs CAN communication for transmitting and receiving data inside the vehicle, short-distance communication such as Bluetooth, and wireless communication for connection with an external server.

The communication unit 150 communicates with an external database (DB) (not shown) or a road information server that provides road information. The communication unit 150 receives current location information of the vehicle, road information on driving, traffic information, road congestion information, road surrounding information, weather information, and the like.

The communication unit 150 may be connected to a driver's portable terminal (not shown) through a short-range communication module.

The communication unit 150 communicates with other nearby cars.

The communication unit 150 transmits and receives a basic safety message (BSM) by communicating with other vehicles using a vehicle to anything (V2X) method.

The communication unit 150 transmits the sensor data by including the sensor data in the optional field of the safety message (BSM), thereby sharing sensor data between vehicles.

The image recognition unit 120 recognizes license plates of other vehicles located around the vehicle based on the image captured by the camera unit 170 .

The image recognition unit 120 may recognize a license plate of a vehicle located in the front through a front image, and recognize a license plate of a vehicle located in the rear through a rear image.

In addition, the image recognition unit 120 analyzes the movement of other vehicles based on the captured image. The image recognizing unit 120 analyzes movement of a lane, acceleration, or deceleration based on images of vehicles located in front, rear, left, and right directions.

The data processing unit 130 extracts information included in the option field of the safety message received through the communication unit, analyzes the extracted data, and identifies the vehicle by matching it with a license plate recognized through the image recognition unit 120.

The data processing unit 130 distinguishes vehicles that match license plates, and when license plate recognition is impossible, compares the operation of the vehicle analyzed based on the image with the data of the safety message to match the vehicle.

The coordinate conversion unit 140 converts data of other vehicles included in the safety message into coordinates based on the own vehicle. The coordinate conversion unit 140 may set coordinate conversion in consideration of the relative position of the matched vehicle.

Since each car records information about the location, etc. based on its own vehicle, the coordinate conversion unit 140 analyzes the data received from other vehicles and converts the coordinates set at the center of the corresponding vehicle into the own vehicle standard.

During autonomous driving, the control unit 110 sets a route to the destination and autonomously drives based on road information, surrounding environment, and information about other nearby vehicles acquired through the sensing unit 160 and the camera unit 170. Control. In addition, the controller 110 controls a steering angle, controls driving so that the vehicle accelerates or decelerates, and stops the vehicle through a brake system (not shown).

Since the autonomous driving method may be performed using various known methods, a detailed description thereof will be omitted.

The control unit 110 recognizes adjacent vehicles based on vehicle matching results through safety message and license plate recognition and vehicle matching results through safety message and vehicle operation, and tracks the operation of each vehicle.

In addition, the control unit 110 receives the data of another vehicle recognized by the matched vehicle through the safety message, and when the coordinate conversion of the sensor data of the corresponding vehicle is coordinated through the coordinate conversion unit 140, it is converted to the sensor data of the own vehicle 160 ) to control the driving.

The control unit 110 analyzes the operation of another vehicle based on the sensor data of the received safety message, and controls driving according to changes in the operation of the other vehicle.

Since the control unit 110 can use not only the data of the sensor 160 but also sensor data detected from other vehicles, more information can be obtained and information on blind spots can be obtained, so that the You can easily track motion and acquire information about your surroundings.

The controller 110 controls driving by decelerating or accelerating in response to a change in motion of a vehicle, such as when a vehicle that is not recognized as an image enters a driving lane or when one of the preceding vehicles decelerates.

The control unit 110 sends a notification about an event that occurs while the vehicle is driving, a change in recognized vehicle motion, a change during driving, etc. through an output unit (not shown) to at least one of a guide message, a warning sound, a sound effect, voice guidance, and a warning light. output as a combination of

Therefore, the driving control system of the vehicle 10 fuses the sensor data of the matched vehicle transmitted through the safety message (BSM) with the sensor information of the host vehicle through coordinate conversion, and uses the sensor information of other vehicles to recognize the driving environment. number of sensors can be increased.

2 is a reference diagram for describing data used for inter-vehicle sensor fusion according to an embodiment of the present invention.

The driving control system uses V2X to exchange basic safety messages (BSM) defined in the rules (SAE J2735) every specific period (ex: 100 ms) with a device in a predetermined communication range, for example, 500 m. .

As shown in FIG. 2, the driving control system transmits and receives a safety message including an optional field in addition to two data fields and a predetermined data field using the V2X communication method, but sensor data in addition to the optional field. and data on the recognized environment information can be transmitted and received.

The safety message includes a first data field 181, a second data field 182 and an option field 183.

The driving control system includes sensor data in the option field 183 and transmits a safety message to another vehicle, and detects and uses the sensor data of the option field included in the safety message of the other vehicle.

The first data field 181 includes basic vehicle information (Basic Vehicle State). Basic vehicle information includes Message Sequence Number, Temporary ID, Time, Position Information (Position Latitude, Longitude, Elevation, Accuracy), Vehicle Speed, Heading, and Steering Angle. Wheel Angle), acceleration information (Vehicle Accelerations, Yaw Rate), brake status, vehicle length, and vehicle width information.

The second data field 182 includes a Vehicle Events Object, a Warning History Object, a Vehicle Path Prediction Object, and a Vehicle Relative Positioning RTCM 1002 Data Object. include

The option field 183 includes sensor data and license plate. Sensor data may include lanes, other vehicles, traffic signs, traffic signals, and the like.

Since data is not specified in the option field 183, it may include data about sensed sensor data, recognized environment information, license plate, and the like.

The driving control system mutually shares sensor data using the option field 183 of the safety message, and obtains information about surroundings by fusing the data sensed by the response itself.

For example, when the driving control system assumes that there are two vehicles A and B in front, and that vehicles A and B drive at the same speed and acceleration, there is an error of several m in the GPS location information, It is difficult to distinguish between vehicle A and vehicle B using only GPS signals.

The driving control system fuses the sensor data included in the two safety messages transmitted from each vehicle and the sensor data measured by the sensing unit 160 of the own vehicle for two vehicles that are difficult to distinguish, thereby integrating the vehicle with the sensor data. Matching can be distinguished.

The driving control system may recognize and match license plates of vehicles, and may match vehicles by classifying vehicle operations based on sensor data of each vehicle.

3 is an exemplary diagram referenced to describe a vehicle recognition method of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.

As shown in (a) of FIG. 3 , the driving control system includes a plurality of cameras provided in the camera unit 170, in particular, a first camera 171 for capturing a front image and a first camera for capturing a rear image. (172) is used to recognize the license plate of another vehicle. The driving control system recognizes the license plate of the rear vehicle through the rear image.

The image recognition unit 120 of the driving control system recognizes vehicles from images, separates images of each vehicle, and then recognizes license plates for each vehicle.

For vehicle license plates, it is obligatory to write vehicle registration numbers (vehicle numbers) on rectangular metal plates for vehicle management. Since the vehicle number written on the license plate is a unique number assigned to each vehicle, the driving control system performs vehicle matching using the vehicle number.

The image recognition unit 120 recognizes the license plate and extracts the vehicle number, and the data processing unit compares the extracted vehicle number with the vehicle number included in the safety message to match the vehicle whose license plate is recognized with the safety message.

The driving control system includes the sensor data and vehicle number in the option field 183 of the safety message (BSM), thereby checking which vehicle the safety message is received from, as well as matching the vehicle number recognized through the video to the vehicle. can be distinguished, and the actual vehicle and safety message can be matched.

The driving control system acquires images from each of the front first camera 171 and the rear second camera 172, and recognizes the vehicle and license plate through the image recognition unit 120. The driving control system uses pre-stored learning data to extract a license plate number through deep learning-based recognition of letters or numbers from an image.

In some cases, the driving control system may be connected to an external server through the communication unit 150 and extract a license plate number from an image based on deep learning.

The driving control system uses an object recognition network to detect a bounding box surrounding a vehicle and a transversal character.

Since the license plate is mounted in the center of the vehicle bumper, the bounding box of the character class included in the vehicle bounding box to which the margin is added is extracted as a candidate group, and after changing the size of the area of the character bounding box to a certain size, , OCR (Optical Character Recognition) Recognizes a character by inputting an image of a character bounding box into the network.

The recognized character string is shown in (b) of FIG. 3 .

The driving control system determines whether the recognized character string matches the license plate format. Since the standards of license plates for each country may be different, one line string or two line string is determined according to the regulations.

For example, when there are a plurality of recognized character strings, the driving control system determines whether they match the license plate format among GENESIS, GV80, 4WD, and 129th 2912.

In addition, when a plurality of character strings are recognized in one vehicle, the driving control system also performs consistency determination on a result of combining two character strings.

For example, the drive control system may determine consistency by combining GENESIS, GV80, 4WD, and 129th 2912, and combining GENESISGV80, GV804WD, and the like.

The driving control system recognizes the 129th 2912 matching the license plate format as a license plate through combination and consistency analysis on the recognized character string.

4 is a flowchart illustrating a sensor fusion method of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.

As shown in FIG. 4 , the driving control system receives a safety message (BSM) from another vehicle through the communication unit 150 (S310). Safety messages can be mutually transmitted and received with other vehicles within a predetermined distance at regular intervals in a V2X manner.

During communication, in the option field of the safety message, sensor data including at least one of lanes, other vehicles, objects, traffic signals, and traffic information detected by a plurality of sensors and vehicle number are transmitted, and sensors from a plurality of vehicles are transmitted. Receives a safety message containing data.

By analyzing the images obtained from the plurality of cameras 171 and 172 provided in the camera unit 170, the image recognition unit 120 classifies the vehicle in the image and recognizes the license plate in the image area determined as the vehicle. Do (S320).

The driving control system acquires a license plate number by analyzing the image of the recognized license plate based on deep learning (S340). The driving control system compares the vehicle number extracted from the option field of the safety message (BSM) with the vehicle number recognized from the license plate through the data processing unit 130 to determine whether they match (S340).

When the vehicle number extracted from the option field of the safety message and the vehicle number recognized from the image match, the data processing unit 130 matches the sensor data to the corresponding vehicle number (S350).

On the other hand, the data processing unit 130 determines if the vehicle number extracted from the option field of the safety message and the vehicle number recognized from the image do not match, or if the matching vehicle number does not exist, it is difficult to recognize the license plate through the image. It is classified as a vehicle located in the detection area (S370). With respect to the vehicle in the undetected area, the data processing unit 130 may perform vehicle matching based on a vehicle operation or vehicle location, which will be described later.

The data processing unit 130 stores data on the matched vehicle in the data unit 180 and tracks the operation of the corresponding vehicle (S360).

The coordinate conversion unit 140 coordinates the sensor data included in the matched safety message of the vehicle based on the own vehicle.

The control unit 110 fuses the data of the sensing unit 160 and the sensor data of the matched vehicle coordinate conversion to obtain information about the surroundings of the vehicle and controls autonomous driving based on this.

For example, when a vehicle located in front is matched, a collision can be prevented by controlling the driving speed in response to a change in speed of the vehicle in front, and driving can be controlled in response to the vehicle on the right in front moving the lane.

5 is a reference diagram for explaining vehicle matching using location information of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.

When the car 10 recognizes a license plate using a camera image and matches it with a safety message, the accuracy of vehicle matching is improved, but there is a condition that the distance between vehicles must be close enough to distinguish a string.

In addition, when the vehicle 10 recognizes a license plate based on an image captured through a camera, it is possible to recognize license plates of vehicles in the front, rear, left front and rear, and right front and rear vehicles. However, since it is difficult to recognize a number plate through an image of a number of vehicles, such as being located on the left or right side or including only a part of the camera's angle of view, it is impossible to match the vehicle using the image.

Accordingly, the driving control system matches the vehicle based on the location or operation of the vehicle when the license plate cannot be recognized from the image through the data processing unit 130 .

The driving control system may mismatch the vehicle when matching only with the absolute coordinates, speed, and direction information at a specific point in time due to the possibility of many-to-many correspondence between the vehicle and the message detected by the GPS error and the sensor. Therefore, the matched vehicle is tracked to check whether the information has consistency for a number of samples, and then vehicle matching is finally performed. The driving control system may increase accuracy by reconfirming whether vehicle matching is normally completed through vehicle tracking.

As shown in (a) of FIG. 5 , the driving control system may receive GPS values and sensor values of each vehicle when the vehicle is located adjacent to the side lane.

The data processing unit 130 maps the vehicle values measured by the sensor within the GPS error range of the own vehicle to configure the actual vehicle candidate area, applies the GPS error range to the received safety message (BSM), and transmits the safety message to the vehicle. A candidate area for is generated, and a matching candidate is set by obtaining an intersection with a candidate area for the measured vehicle.

The data processing unit 130 has a many-to-many intersection of sensor values and GPS values, and determines that matching has failed if it cannot be distinguished even after erasing the matching-confirmed sensor and GPS information.

The data processing unit 130 matches the GPS signal of the first vehicle V1 with the sensor value received from the first vehicle V1, and matches the GPS signal of the second vehicle V2 with the second vehicle. The sensor values of (V2) are mutually matched.

However, the data processing unit 130 determines that the matching fails because the candidate regions for the first vehicle and the second vehicle overlap.

In addition, the data processing unit 130 determines that the matching fails with respect to the vehicle when data matching the GPS value of the third vehicle e does not exist.

As shown in (b) of FIG. 5, when the longitudinal position of the vehicle can be distinguished, that is, if the distance difference is greater than the GPS error or can be distinguished by the speed difference, the driving control system determines the GPS value and the sensor value. Vehicles are matched through location intersection, and after matching, tracks are created and tracked.

The data processing unit 130 matches the GPS signal of the first vehicle V1 with the sensor value received from the first vehicle V1, and matches the GPS signal of the second vehicle V2 with the second vehicle. The sensor values of (V2) are mutually matched, but since the areas of the first vehicle and the second vehicle do not overlap, each is matched to the vehicle, and a track is created and tracked.

The driving control system finally performs vehicle matching through the tracking result.

6 is a flowchart illustrating a vehicle matching method using location information of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.

As shown in FIG. 6 , the driving control system manages the matched vehicle by tracking the motion of the matched vehicle.

The data processing unit 130 mutually matches the location of the vehicle measured by itself and the location information included in the safety message to match the actual vehicle and the safety message (S410).

Based on the ID (temporary ID) included in the safety message, the data processing unit 130 checks whether the match has already been confirmed (S420), and if the match is confirmed, vehicle matching is performed (S460).

If it is not the confirmed match, the data processing unit 130 determines whether there is one expected candidate to be matched (S430).

If there is only one expected candidate, the data processing unit 130 matches the candidate with the vehicle (S460).

Meanwhile, when there are a plurality of predicted candidates, the data processing unit 130 determines whether matching is possible using vehicle speed information (S440). The data processor 130 calculates the speed of the vehicle, compares it with the speed of the vehicle included in the safety message, and matches the vehicle (S460).

The data processing unit 130 determines whether matching is possible through lane information when it is impossible to match by speed (S450). If matching is possible with the lane, the data processing unit 130 performs vehicle matching based on the lane and the direction of the vehicle (S460).

The data processing unit 130 extracts sensor data from the option field of the safety message (BSM) to determine if it is the same as the sensor information when the host vehicle and candidate vehicle are located on the edge lane or can recognize the driving lane based on a guard rail or the like. The vehicle can be matched by judging.

If it is possible to match by speed, the data processing unit 130 matches the vehicle based on the vehicle speed of the safety message (S460).

On the other hand, the data processing unit 130 determines that matching has failed if matching has not been confirmed and matching is impossible with speed and lane information even though there are a plurality of predicted candidates.

7 is a flowchart illustrating a method of managing matched data of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.

As shown in FIG. 7 , the driving control system checks whether the vehicle matching is normally performed by tracking the operation of the vehicle when matching information on the location or operation of the vehicle in addition to license plate recognition through the image.

The data processing unit 130 merges the existing matching result with the new matching result based on the ID of the safety message (S470).

The data processing unit 130 registers a new track for a matching candidate (S490) for new matching (S480). The driving control system tracks and records the motion of the vehicle for which the information is matched.

The data processing unit 130 determines whether there exists a track that has succeeded in long-term matching among matching candidates with respect to existing matching (S500). If there is a track that has been successfully matched for a long time, the matching candidate is changed to a confirmed match (S510).

The data processing unit 130 determines whether there is a candidate that has not been matched for a long time with respect to the confirmed matching (S520). Among the confirmed matches, for a candidate that has not been matched for a long time, the corresponding confirmed match is changed to a matching candidate (S530).

In addition, the data processing unit 130 determines whether there is a candidate that has not been matched for a long time among the matching candidates (S540), and deletes the matching candidate track of the candidate that has not been matched for a long time (S550).

The data processing unit 130 matches a vehicle based on the location or operation of the vehicle and then registers and manages the track of the matched vehicle, thereby changing the candidate to a confirmed match based on the matching result for a certain period of time, or The confirmed matching may be changed to a candidate, and information on vehicle matching may be managed according to the vehicle tracking record by deleting tracks for candidates that do not match within a period.

8 is an exemplary view referenced to describe coordinate conversion of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.

The driving control system integrates sensor information by transmitting and receiving sensor data in the option field of the safety message (BSM) of V2X. The driving control system moves the sensor data of the matched vehicle in parallel based on the relative coordinate information measured through the own vehicle sensor, and rotates using the rotation angle calculated based on the geomagnetic sensor, the lane and the direction of travel, convert to coordinate system

As shown in (a) of FIG. 8 , the coordinate conversion unit 140 checks the relative coordinates between the vehicles based on the sensor data when the matching between the safety message (BSM) of the other vehicle and the sensor data is completed.

The coordinate conversion unit 140 fuses the sensor data measured in each vehicle by adding the geomagnetic sensor or lane reference direction information, and uses the option field of the safety message (BSM) to determine the difference in direction of the vehicle. It transmits sensor information or the driving angle based on the lane.

The driving control system of the vehicle 10 matches vehicles and detects a target vehicle using sensor data of other vehicles.

For example, the first vehicle 11 drives in an eleventh direction D11 forming a first angle θe with respect to north (N), and the second vehicle 12 is driven with respect to north (N). and travels in the twenty-first direction D21 forming the second angle θv.

The first vehicle 11 may match the second vehicle 12 and check the third vehicle 13 based on data included in the safety message of the second vehicle 12 .

The second vehicle 12 detects the third vehicle 13 with coordinates (Rt, Θt), and the first vehicle 11 transmits the safety message BSM of the second vehicle 12 to the third vehicle ( Convert the coordinates of 13) to the coordinates of the own vehicle.

When converting coordinates, the first vehicle 11 compensates for rotation and translation of the coordinate system.

The first vehicle 11 uses a geomagnetic sensor to share the traveling direction angles (Θe, Θv) of the vehicle based on the North (N) and N poles, thereby rotating and converting the rotation axis difference by Θe - Θv to rotate compensate for In addition, the first vehicle 11 compensates translation by using the distance Rs to the second vehicle measured by the host vehicle sensor and the rotation angle Θs with respect to the third vehicle.

)를 산출한다. The first vehicle 11 converts the position of the third vehicle 13, which is the target vehicle detected by the other vehicle, into the coordinate system of the own vehicle and coordinates R _f ,

) is calculated.

Coordinate conversion of the first vehicle 11 to the third vehicle 13 can be calculated as in Equation 1 below.

)을 이용하여 좌표를 변환한다. The traveling direction angle of the first vehicle 11 and the second vehicle 12, the straight-line distance Rs between the first vehicle 11 and the second vehicle 12, and the second vehicle 12 and the third vehicle 13 ) of the straight line distance (Rt), the rotation angle for the third vehicle (

) to transform the coordinates.

As shown in (b) of FIG. 8 , when there is no geomagnetic sensor or the error is large, the driving control system of the vehicle 10 converts coordinates using lanes and the like.

The fourth vehicle 14 calculates a direction angle difference by sharing the heading angle based on the lane using the option field of the safety message BSM.

When the fourth vehicle 14 travels in the forty-first direction D41, the bearing angle Θ4 with respect to the forty-first direction D41 is calculated based on the tangential direction D42 to the lane.

When the fifth vehicle 15 travels in the fifty-first direction D51, the direction angle Θ5 with respect to the fifty-first direction D51 based on the tangential directions D52 and D53 to the lane is set. yield

(f'(x)) -

(f'(0))가 된다. Assuming that the equation of the lane of the fourth vehicle 14 is f(o) and the equation of the lane of the identified matching vehicle, the fifth vehicle 15, is g(o), the change in the slope of the tangential line of the lane is

(f'(x)) -

(f'(0)).

와

를 바탕으로, 장애물 위치를 회전 변환하여 자차 좌표계로 옮기기 위한 회전 각은The change in the tangential slope of the lane and the difference between the lane and the direction of travel obtained from each vehicle

and

Based on , the rotation angle for rotating the obstacle position and moving it to the own vehicle coordinate system is

Therefore, the vehicle 10 can obtain a position in the host vehicle coordinate system by translating the obstacle detected by the identified matching vehicle by the host vehicle sensor information (Rs, Θs).

9 is an exemplary view illustrating another example of vehicle matching of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.

As shown in FIG. 9 , the driving control system fuses and uses sensor data of another vehicle when matching and coordinate conversion are completed with respect to another vehicle.

The control unit 110 obtains information on the surrounding environment by converging sensor data of other vehicles with data acquired through the sensing unit 160 and controls autonomous driving by detecting the approach of another vehicle.

The sixth vehicle 21 matches the vehicle to the eighth vehicle 23 by recognizing the license plate, and matches the vehicle to the seventh vehicle 22 based on location.

The sixth vehicle 21 receives safety messages from the seventh vehicle 22 and the eighth vehicle 23 and uses sensor data, thereby increasing the number of usable sensors.

The sixth vehicle 21 may also obtain information about a vehicle ahead of the third vehicle 23 by using sensor data of the matched vehicle.

Accordingly, the driving control system can use more sensors than the number of sensors provided, can generate data with high reliability, and can precisely control driving according to the movement (action) of an adjacent vehicle.

10 is a flowchart illustrating a driving control method of a driving control system through inter-vehicle sensor fusion according to an embodiment of the present invention.

As shown in FIG. 10 , the driving control system of the vehicle recognizes surrounding vehicles using an image captured through a camera (S570).

The driving control system transmits and receives a safety message (BSM) with a vehicle within a certain distance at a predetermined period (S580).

The driving control system detects a license plate from the image captured by the camera unit 170 through the image recognition unit 120 and recognizes the vehicle number from the license plate.

If the license plate recognition is possible, the driving control system matches the vehicle based on the license plate (S600), and if the license plate recognition is impossible, the driving control system matches the vehicle based on the location or operation of the vehicle (S610).

As described above, the driving control system may match the vehicle number with the vehicle number included in the safety message, and may match the vehicle by tracking based on the location or operation of the vehicle.

When vehicle matching is completed, the driving control system extracts sensor data included in the option field of the safety message of the matched vehicle (S620).

Since the driving control system knows relative position information between vehicles after matching is completed, obstacle recognition performance can be improved by transmitting data acquired from each sensor.

Since the location information of the sensor data is based on the location of each vehicle, the driving control system converts the received data of other vehicles into coordinates based on the vehicle.

Since the driving control system requires rotation information for coordinate conversion, it converts the coordinates by calculating conversion angle information based on a geomagnetic sensor or a lane.

When the geomagnetic sensor is provided (S630), as shown in (a) of FIG. 8 described above, the coordinates are converted based on the measured value of the geomagnetic sensor (S640). As in b), the coordinates are converted based on the lane and direction (S650).

The driving control system combines the converted coordinates of another vehicle with the coordinate values of its own sensor (S660), and controls driving in response to the recognized vehicle motion and data (S670).

Although the present invention has been described with reference to the embodiments shown in the drawings, it should be noted that this is only exemplary and various modifications and equivalent other embodiments are possible from those skilled in the art in the art to which the technology pertains. will understand Therefore, the true technical protection scope of the present invention should be defined by the claims below.

10, 11 to 15, 21 to 23: car
110: control unit 120: image recognition unit
130: data processing unit 140: coordinate conversion unit
150: communication unit 160: detection unit
161: position sensor 162: obstacle sensor
170: camera unit 180: data unit

Claims (20)

A camera unit including a plurality of cameras to capture images around the vehicle;
a sensing unit that measures the position of the vehicle and detects objects around the vehicle;
an image recognition unit for analyzing an image captured by the camera unit, recognizing a license plate of a second vehicle located nearby, and extracting a license plate number;
A communication unit that communicates with a plurality of vehicles located within a certain distance, including the sensor data and vehicle number obtained from the sensor data and the V2X (Vehicle to anything) method in an option field of a safety message (BSM, Basic Safety Message);
a data processing unit matching the plurality of vehicles and the plurality of safety messages based on at least one of the vehicle number and the location and operation of the plurality of vehicles;
a coordinate conversion unit extracting sensor data from the safety message of the second vehicle for which matching has been completed and converting coordinates based on the vehicle; and
a control unit controlling driving based on information about surroundings of the vehicle obtained by fusing data of the sensing unit and sensor data of the second vehicle based on a matching result of the second vehicle; Driving control system comprising a. According to claim 1,
The data processing unit compares the vehicle number of the second vehicle with the vehicle numbers included in the plurality of safety messages, and matches a safety message having the same vehicle number with the second vehicle;
For a third vehicle whose license plate is not recognizable among the plurality of vehicles, based on the information included in the plurality of safety messages, the location and operation of the plurality of vehicles are analyzed, and the third vehicle sensed through the detector. Comparing with vehicle data, matching the third vehicle. According to claim 2,
The data processing unit compares the location of the third vehicle detected through the sensor with location information included in the plurality of safety messages, and matches a safety message that matches within an error range with the third vehicle. driving control system. According to claim 2,
The data processing unit compares the operation information of the third vehicle detected through the sensor with operation information included in the plurality of safety messages, sets at least one matching candidate, accumulates the operation information, and performs the matching operation. The driving control system, characterized in that for matching a safety message that matches within an error range among candidates with the third vehicle. According to claim 2,
The data processing unit detects the position of the driving lane and an object such as a guardrail located in the driving lane through the sensing unit, and compares object detection information included in the plurality of safety messages to determine the safety message and the third vehicle. A driving control system characterized in that matching. According to claim 1,
The data processing unit creates and registers a new track for the matched vehicle;
Among at least one matching candidate included in the pre-matched data, a matching candidate that has successfully matched for a certain period of time is changed to a confirmed match, and among the at least one confirmed match, a confirmed match that has not been matched during the certain period is changed. and managing matching results for the plurality of vehicles by switching to a matching candidate and deleting a track for a matching candidate that has not been matched for the predetermined period among the at least one matching candidate. According to claim 1,
The coordinate conversion unit acquires the coordinates of the fourth vehicle detected by the second vehicle from sensor data included in the safety message of the second vehicle, and converts the coordinates of the fourth vehicle based on the own vehicle. Featured driving control system. According to claim 7,
When the coordinate conversion unit includes a geomagnetic sensor, based on the traveling direction angle of the vehicle based on the north (N pole) detected by the geomagnetic sensor and the traveling direction angle of the second vehicle, in the second vehicle The driving control system, characterized in that for coordinate conversion of the detected coordinate value of the fourth vehicle. According to claim 8,
The coordinate conversion unit compensates for the rotation by rotating the angle of the traveling direction of each vehicle based on the N pole by the difference in the rotation axis for the fourth vehicle,
Characterized in that the coordinates of the fourth vehicle are converted to coordinates by compensating the transformation using the distance between the vehicle and the second vehicle, the distance between the second vehicle and the fourth vehicle, and the rotation angle of the fourth vehicle. driving control system. According to claim 7,
The driving control system of claim 1 , wherein the coordinate conversion unit converts the coordinates of the fourth vehicle based on a difference between a lane of a lane in which each vehicle is driving and a traveling direction when the geomagnetic sensor is not included. According to claim 10,
The driving control system of claim 1 , wherein the coordinate conversion unit calculates a change in the tangential slope of the lane and converts the coordinates of the fourth vehicle using a rotation angle calculated using a difference between the lane and the traveling direction. Taking images around the vehicle through a plurality of cameras, and detecting the location of the vehicle and objects around the vehicle;
Recognizing the license plate of the second vehicle from the image and extracting the vehicle number;
In the optional field of the safety message (BSM, Basic Safety Message), communicating with a plurality of vehicles within a certain distance, including sensor data and vehicle number, in a V2X (Vehicle to anything) method;
matching the plurality of vehicles and the plurality of safety messages based on at least one of the vehicle number, locations and operations of the plurality of vehicles;
extracting sensor data from the safety message of the second vehicle for which matching has been completed, and converting coordinates based on the vehicle;
obtaining information about surroundings of the vehicle by fusing sensor data of the host vehicle and sensor data of the second vehicle based on a matching result of the second vehicle; and
autonomous driving based on the acquired information; A method of operating a driving control system comprising a. According to claim 12,
The step of matching the plurality of vehicles and the plurality of safety messages,
comparing the vehicle number of the second vehicle whose license plate is recognized with the vehicle number included in the plurality of safety messages; and
matching a second safety message whose vehicle number matches the second vehicle with the second vehicle; A method of operating a driving control system further comprising a. According to claim 12,
The step of matching the plurality of vehicles and the plurality of safety messages,
sensing a location, operation, and lane information of a third vehicle whose license plate is not recognized by using a plurality of sensors provided in the sensing unit;
analyzing at least one of location, operation, and lane information of the plurality of vehicles based on data included in the plurality of safety messages;
setting at least one matching candidate estimated to be the safety message of the third vehicle; and
comparing data of the third vehicle with the matching candidate and matching a third safety message matching the data of the third vehicle with the third vehicle within an error range; A method of operating a driving control system further comprising a. 15. The method of claim 14,
The step of matching the plurality of vehicles and the plurality of safety messages,
Operation of the driving control system, characterized in that by detecting a lane of the driving lane and an object adjacent to the driving lane, comparing the lane and object detection information from the data included in the plurality of safety messages, and performing vehicle matching according to whether or not they match. method. 15. The method of claim 14,
The step of matching the plurality of vehicles and the plurality of safety messages,
The operation method of the driving control system, characterized in that for the matching candidate, matching the third vehicle and the third safety message based on data accumulated for a predetermined time. 15. The method of claim 14,
generating and registering a new track with respect to the matched third vehicle;
changing a matching candidate that has successfully matched for a certain period of time among at least one matching candidate included in pre-matched data to a confirmed match;
converting a confirmed match that has not been matched during the predetermined period among at least one confirmed match from the pre-matched data into a matching candidate; and
deleting a track of a matching candidate that has not been matched for the predetermined period of time among the at least one matching candidate; Including more,
A method of operating a driving control system, characterized in that for managing a matching result based on the accumulated data. According to claim 12,
The step of transforming the coordinates,
obtaining coordinates of a fourth vehicle sensed by the second vehicle from sensor data included in the safety message of the second vehicle;
If a geomagnetic sensor is included, calculating a traveling direction angle of the vehicle and a traveling direction angle of the second vehicle based on north (N pole), detected by the geomagnetic sensor;
Compensating for the rotation by rotating an angle of the direction of travel of each vehicle by a difference in a rotation axis with respect to the fourth vehicle;
compensating for the transformation using a distance between the vehicle and the second vehicle, a distance between the second vehicle and the fourth vehicle, and a rotation angle with respect to the fourth vehicle; and
converting coordinate values of the fourth vehicle sensed by the second vehicle using rotation compensation and change compensation; A method of operating a driving control system further comprising a. According to claim 12,
The step of transforming the coordinates,
obtaining coordinates of a fourth vehicle sensed by the second vehicle from sensor data included in the safety message of the second vehicle;
calculating a difference between a lane of a lane in which each vehicle is driving and a driving direction when the geomagnetic sensor is not included;
calculating a change in the tangent slope of the lane;
Calculating a rotation angle using a difference between the lane and the driving direction and a change in the slope of the tangential line of the lane; and
converting the coordinate values of the fourth vehicle detected by the second vehicle in correspondence with the rotation angle; A method of operating a driving control system further comprising a. According to claim 12,
The communication step is
In the option field, the safety message including the vehicle number and sensor data including at least one of lanes, other vehicles, objects, traffic signals, and traffic information detected by a plurality of sensors is received from the plurality of vehicles. Operation method of the driving control system to be.

Images