KR101848312B1 - Sensor fusion system for autonomous emergency braking system in car - Google Patents

Abstract

The present invention provides a sensor fusion system for an autonomous emergency braking (AEB) system of a vehicle, comprising: a camera module to photograph surroundings of a vehicle; a light detection and ranging (LiDAR) module scanning the surroundings of the vehicle to recognize an object and detect a distance from the recognized object; a camera module fault monitoring unit to monitor whether or not the camera module is normally operated; a LiDAR module fault monitoring unit to monitor whether or not the LiDAR module is normally operated; and a fault monitoring unit receiving a signal from the camera module fault monitoring unit and the LiDAR module fault monitoring unit to monitor whether or not the overall operation of a sensor fusion system is failed, and outputting result data thereof to the outside including an AEB system. The present invention can enhance driving safety of an autonomous driving vehicle including the AEB system, and improve a recognition range and accuracy of a sensor.

Description

[0001] The present invention relates to a sensor fusion system for an automotive emergency braking system,

The present invention relates to a sensor fusion system for an automotive emergency braking system, and more particularly to a camera sensor module and a LiDAR sensor fusion based recognition system mounted on a vehicle.

A self-propelled vehicle refers to a smart car that incorporates self-propelled technology applied to aircraft and ships, such as a car that travels to its destination without requiring the driver to operate the steering wheel, accelerator pedal, or brake.

In order to realize the autonomous driving of the vehicle, it is necessary to use a highway driving assist (HDA) technology, a lane departure warning system (LDWS), a lane keeping assistance system (LKAS) Vehicle technologies such as Blind Spot Detection (BSD), Advanced Smart Cruise Control (ASCC), and Autonomous Emergency Braking (AEB) are required. In addition, Satellite communication technology, and so on.

On the other hand, in the autonomous driving situation, the driver does not operate the pedal (accelerator pedal or brake pedal) of the vehicle. If the driver operates the pedal during autonomous driving, the vehicle controller ends the autonomous driving of the driver, It is determined that the operation is desired and the control for the autonomous running is terminated.

The Advanced Emergency Braking (AEB) system technology is an automatic emergency braking system for vehicles. It attaches a sensor to the front or rear of the vehicle, detects other vehicles or obstacles around the vehicle, and uses this to detect the possibility Is a technique that automatically brakes the vehicle.

The recent AEB technology uses a combination of a video camera, a GPS, and the like beyond a step of recognizing an obstacle with a simple proximity sensor, thereby more accurately detecting other vehicles or obstacles around the vehicle, accurately determining the possibility of collision, To effectively braking the engine.

In this way, AEB technology is a technology to automatically braking a vehicle while recognizing an object ahead through a sensor. It is an active safety device that operates when the preceding vehicle is speeding down or stopping, or when obstacles such as pedestrians suddenly appear. These AEBs are reported to reduce collision accidents by about 27%, and they are classified into AEB for urban area, AEB for street, and AEB for pedestrian depending on the speed and detection distance.

In this regard, the Euro NCAP (New Car Assessment Program) has decided to include the AEB function in the category of automobile safety grade in 2014, and subsequently regulate the AEB's mandatory installation, We are starting production of vehicles equipped with AEB function. Thus, in recent years, there is a need for improving the safety of a self-driving vehicle when driving the vehicle.

Korean Patent Publication No. 10-2016-0134656

SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an automatic emergency braking system for an autonomous vehicle that can improve the stability of a vehicle by detecting a failure of the camera sensor module and the Raider sensor module, And to provide a sensor fusion system capable of improving accuracy.

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

In order to accomplish the above object, there is provided a sensor fusion system for an automatic emergency braking system for a vehicle, comprising: a camera module for photographing a vicinity of a vehicle; an object recognition unit for scanning the vicinity of the vehicle; A camera module fault monitoring unit for monitoring whether the camera module operates normally, a ladder module fault monitoring unit for monitoring whether the Ladia module is normally operating, And a fault monitoring unit for monitoring the failure of the overall operation of the sensor fusion system and outputting the resultant data to the outside including the automatic emergency braking system, including receiving a signal from the LIDAR module fault monitoring unit .

The sensor fusion system includes a camera module battery monitoring unit for monitoring whether power is normally supplied to the camera module, a Lada module battery monitoring unit for monitoring whether power is normally supplied to the Lada module, A battery monitoring unit for monitoring whether or not main power is normally supplied to the camera module, a hardware watchdog for monitoring whether the sensor fusion system normally operates using a system watchdog, And a time clock unit for confirming a transmission speed of data transmitted from the Lidar module and performing a synchronization operation between them.

The camera module fault monitoring unit monitors whether the image data photographed by the camera module has a predetermined resolution, whether the camera module is powered off or not, and whether the operation range for the horizontal FOV (field of view) And when it detects a fault, it can notify the fault monitoring unit.

The LIDAR module fault monitoring unit monitors whether or not the signal received by the Lidar module is reflected after being emitted from the Lidar module, whether the rotational angle of the Lidar motor is obstructed, whether the horizontal FOV and the vertical FOV are obstructed And if it detects a fault, it can notify the fault monitoring unit.

Wherein the time clock unit monitors whether the transmission speed of the camera data transmitted from the camera module corresponds to a predetermined speed and whether the transmission speed of the Lada data transmitted from the Lida module corresponds to a predetermined speed, If a fault is detected, it can be notified to the fault monitoring unit.

According to the present invention, it is possible to improve the operational safety of an autonomous vehicle including an automatic emergency braking system and improve the recognition range and accuracy of the sensor by monitoring various failures associated with the camera sensor module and the Lidar sensor module mounted on the vehicle There is an effect that can be made.

1 is a diagram showing an outline of a sensor fusion system of a vehicle.
2 is an overall configuration diagram of a sensor fusion system of a vehicle.
3 is a diagram illustrating an architecture of a sensor fusion system according to an embodiment of the present invention.
4 is a state transition diagram for explaining an operation in a camera module fault monitoring unit according to an embodiment of the present invention.
5 is a state transition diagram for explaining an operation in a lidar module fault monitoring unit according to an embodiment of the present invention.
6 is a state transition diagram for explaining an operation in a camera module battery monitoring unit according to an embodiment of the present invention.
7 is a state transition diagram for explaining an operation in the lithium ion battery monitoring unit according to an embodiment of the present invention.
8 is a state transition diagram for explaining an operation in the battery monitoring unit according to an embodiment of the present invention.
9 is a state transition diagram for explaining operations in a hardware watchdog according to an embodiment of the present invention.
10 is a state transition diagram for explaining an operation in a time clock unit according to an embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted in an ideal or overly formal sense unless expressly defined in the present application Do not.

In the following description of the present invention with reference to the accompanying drawings, the same components are denoted by the same reference numerals regardless of the reference numerals, and redundant explanations thereof will be omitted. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail.

The present invention relates to a sensor fusion system (SFS) for an automotive emergency braking (AEB) system.

1 is a diagram showing an outline of a sensor fusion system of a vehicle.

Referring to FIG. 1, the front camera and LiDAR sensor standards-based recognition system of the present invention recognizes a lane ahead, a vehicle, a pedestrian, and the like, In ESC, ACC, EMS, Cluster, etc., information is provided for urgent braking of the vehicle in dangerous condition, and the direction of the vehicle is recognized through SAS.

2 is an overall configuration diagram of a sensor fusion system of a vehicle.

Referring to FIG. 2, the camera-Rada integrated sensor module of the present invention includes a Lada optical system, a PLD (Pulse Laser Diode), an APD (Avalanche Photo Diode), a camera sensor, and a signal processing board. Here, the PLD is a laser that outputs light using a diode, which oscillates and stops with time, and APD performs a function of converting an optical signal into an electric signal by using an avalanche phenomenon.

The multi-object estimation / analysis and information display module analyzes and displays the sensed data in the camera-ray integrated sensor module, performs 3D point cloud-based information analysis, and performs optimal AED signals such as braking strength and avoidance direction And transmits it to the AEB ECU.

3 is a diagram illustrating an architecture of a sensor fusion system according to an embodiment of the present invention.

3, the sensor fusion system 10 according to an exemplary embodiment of the present invention includes a camera module 210, a LiDAR module 300, a camera module fault monitoring unit 110 A battery module monitoring unit 120, a camera module battery monitoring unit 130, a lidar module battery monitoring unit 140, a battery monitoring unit 150, a hardware watchdog 160, A clock (Time Clock) unit 170, and a fault monitoring unit 180.

The camera module unit 200 includes a camera module 210 for photographing the surroundings of the vehicle and a camera module 210 for converting the image data of the AHD standard photographed by the camera module 210 into a high definition multimedia interface And an AHD TO HDMI converter 220.

The Lidar module 300 is a laser module using an optical signal. The Lidar module 300 scans the periphery of the vehicle to recognize the object and detects the distance to the recognized object.

The camera module fault monitoring unit 110 monitors whether the camera module 210 operates normally.

The LIDAR module fault monitoring unit 120 monitors whether the LIDAR module 300 is operating normally.

The camera module battery monitoring unit 130 monitors whether the camera module 210 is normally supplied with power.

The Lida module battery monitoring unit 140 monitors whether or not power is normally supplied to the Lida module 300. [

The battery monitoring unit 150 monitors whether the main power is normally supplied to the sensor fusion system 10.

The hardware watchdog 160 monitors whether the sensor fusion system 10 is operating normally using a system watchdog.

The time clock unit 170 checks the transmission speed of the data transmitted from the camera module 210 and the data transmitted from the Lydia module 300, and performs a synchronization operation between them.

The fault monitoring unit 180 includes a camera module fault monitoring unit 110, a lidar module fault monitoring unit 120, a camera module battery monitoring unit 130, a lidar module battery monitoring unit 140, a battery monitoring unit 150 The hardware watchdog 160, the time clock unit 170, and the like, monitors the failure of the overall operation of the sensor fusion system 10, and outputs the resultant data to the automatic emergency braking system (AEB) And outputs it to the outside.

4 is a state transition diagram for explaining an operation in a camera module fault monitoring unit according to an embodiment of the present invention.

Referring to FIG. 4, the camera module fault monitoring unit 110 determines whether the image data photographed by the camera module 210 has a predetermined resolution, whether the camera module 210 is turned off, Horizontal FOV (Field of View), and notifies the fault monitoring unit 180 of a fault.

For example, the camera module fault monitoring unit 110 can monitor whether the image data has a resolution of FHD (Full High-Definition) (1080p), whether the horizontal FOV has an operation range of 140 °, and the like .

Upon receiving a failure from the camera module fault monitoring unit 110, the fault monitoring unit 180 resets the sensor fusion system and transmits a fault message to an ADAS (Advanced Driver Assistance System) including AEB (Autonomous Emergency Braking) message. Then, only the Lada module 300 is temporarily activated to operate the system.

5 is a state transition diagram for explaining an operation in a lidar module fault monitoring unit according to an embodiment of the present invention.

Referring to FIG. 5, the RID module fault monitoring unit 120 monitors whether or not the signal received after being reflected from the Lid module 300 is received, ) FOV and the vertical FOV, and informs the fault monitoring unit 180 when a failure is detected.

For example, when the signal emitted from the PLD does not receive one thread data through the APD, the LIDAR module fault monitoring unit 120 generates a failure (LiDAR) BLDC motor driving failure, a horizontal FOV It is possible to detect the occurrence of a fault such that the viewing angle is 140 ° and the vertical FOV viewing angle is 5 °.

The fault monitoring unit 180 resets the sensor fusion system and receives a fault message (ADR) from the Advanced Driver Assistance System (ADAS) including the AEB (Autonomous Emergency Braking) fault message. Then, only the camera module 210 is temporarily activated to operate the system.

6 is a state transition diagram for explaining an operation in a camera module battery monitoring unit according to an embodiment of the present invention.

Referring to FIG. 6, the camera module battery monitoring unit 130 periodically monitors whether the power is normally supplied to the camera module 210. When a fault occurs, the camera module battery monitoring unit 130 monitors a power management integrated circuit (PMIC) The fault monitoring unit 180 notifies the fault monitoring unit 180 of the failure.

For example, the camera module battery monitoring unit 130 monitors whether or not 12 V power is normally supplied to the camera module 210 in a period of 100 ms. If the 12 V power is not supplied, the PMIC 410 monitors power , And notifies the fault monitoring unit 180 of the fault.

7 is a state transition diagram for explaining an operation in the lithium ion battery monitoring unit according to an embodiment of the present invention.

Referring to FIG. 7, the Raider module battery monitoring unit 140 periodically monitors whether the power is normally supplied to the Raider module 300, and if a failure occurs, And notifies the monitoring unit 180 of the failure.

For example, the Lidar module battery monitoring unit 140 monitors whether the 12-V power is normally supplied to the Lidar module 300 in a period of 100 ms, and if the 12-V power is not supplied, After the management, the fault monitoring unit 180 is notified of the failure.

8 is a state transition diagram for explaining an operation in the battery monitoring unit according to an embodiment of the present invention.

Referring to FIG. 8, the battery monitoring unit 150 periodically monitors whether the PMIC 410 is normally operating, sends an alarm to a driver when a failure occurs, and operates a recovery plan.

For example, the battery monitoring unit 150 monitors whether or not the ignition (IG) is on and whether the PMIC 410 operates normally at a period of 100 ms, warns the driver when a fault occurs, Is reset once.

9 is a state transition diagram for explaining operations in a hardware watchdog according to an embodiment of the present invention.

Referring to FIG. 9, the hardware watchdog 160 periodically checks whether the main system processor 510 operates normally, and notifies the fault monitoring unit 180 of a failure. For example, the hardware watchdog 160 may monitor whether the main system processor 510 is operating normally at a period of 100 ms.

The fault monitoring unit 180 resets the sensor fusion system when a fault is detected from the hardware watchdog 160 and transmits a fault message to an Advanced Driver Assistance System (ADAS) including Autonomous Emergency Braking (AEB) ). Then, when the normal operation return error occurs after reset, it is operated in the manual mode.

10 is a state transition diagram for explaining an operation in a time clock unit according to an embodiment of the present invention.

10, the time clock unit 170 determines whether the transmission speed of the camera data transmitted from the camera module 210 corresponds to a predetermined speed, and whether the transmission speed of the camera data transmitted from the LIDAR module 300 Monitors whether or not the speed corresponds to a predetermined speed, and notifies the fault monitoring unit 180 when a fault is detected. That is, the time clock unit 170 monitors the processing speed of the camera data transmitted from the camera module 210 and the processing speed of the Larda data transmitted from the Lydia module 300, and if not the predetermined processing speed, And informs the fault monitoring unit 180 of the fact of occurrence.

For example, the time clock unit 170 monitors whether the processing speed of the camera data transmitted from the camera module 210 is 30 fps (30 frames per second), and if not, 180). The time clock unit 170 monitors whether the processing speed of the RLayer data transmitted from the RLayer module 300 is 15 fps (15 frames per second), and if not, 180).

Upon receiving a fault from the time clock unit 170, the fault monitoring unit 180 transmits a fault message to the ADAS (Advanced Driver Assistance System) including the AEB (Autonomous Emergency Braking) After the resetting of the module 210 and the Lidar module 300, if the same failure is detected, the user is informed and switches to the manual mode.

While the present invention has been described with reference to several preferred embodiments, these embodiments are illustrative and not restrictive. It will be understood by those skilled in the art that various changes and modifications may be made therein without departing from the spirit of the invention and the scope of the appended claims.

10 Sensor fusion system 210 Camera module
300 Raidas module
110 Camera Module Fault Monitoring
120 Lider module fault monitoring part
130 Camera module Battery monitoring section
140 Raidas module Battery monitoring part
150 Battery monitoring section
160 Hardware Watchdog
170 time clock section
180 Fault monitoring part

Claims (8)

In a sensor fusion system for an automotive emergency braking system,
A camera module for photographing the surroundings of the vehicle;
A lidar module for recognizing an object by scanning the periphery of the vehicle and detecting a distance to the recognized object;
A camera module fault monitoring unit for monitoring whether the camera module operates normally;
A lidar module fault monitoring unit for monitoring whether the ladder module operates normally;
A camera module battery monitoring unit for monitoring whether power is normally supplied to the camera module;
A lidar module battery monitoring unit for monitoring whether power is normally supplied to the ladda module;
A battery monitoring unit for monitoring whether the main power is normally supplied to the sensor fusion system;
A hardware watchdog for monitoring whether the sensor fusion system is operating normally using a system watchdog;
A time clock unit for confirming a transmission speed of data transmitted from the camera module and data transmitted from the LIDAR module and performing a synchronization operation between them; And
And receiving a signal from the camera module fault monitoring unit, the Rider module fault monitoring unit, the camera module battery monitoring unit, the Rider module battery monitoring unit, the battery monitoring unit, the hardware watch dog, and the time clock unit And a fault monitoring unit for monitoring a failure of the overall operation of the sensor fusion system and outputting the resultant data to the outside including the automatic emergency braking system,
The camera module fault monitoring unit monitors whether the image data photographed by the camera module has a predetermined resolution, whether the camera module is powered off or not, and whether the operation range for the horizontal FOV (field of view) Monitors the fault monitoring unit, and when a fault is detected, notifies the fault monitoring unit,
The LIDAR module fault monitoring unit monitors whether or not the signal received by the Lidar module is reflected after being emitted from the Lidar module, whether the rotational angle of the Lidar motor is obstructed, whether the horizontal FOV and the vertical FOV are obstructed And when it detects a fault, notifies the fault monitoring unit of the fault,
Wherein the time clock unit monitors whether or not the transmission speed of the camera data transmitted from the camera module corresponds to a predetermined speed and whether the transmission speed of the Lada data transmitted from the Lida module corresponds to a predetermined speed, When a fault is detected, the fault monitoring unit is notified of the fault,
The fault monitoring unit resets the sensor fusion system and transmits a fault message to an Advanced Driver Assistance System (ADAS) including the automatic emergency braking system upon receiving a fault from the camera module fault monitoring unit, Only the Lada module is temporarily driven to drive the sensor fusion system,
The fault monitoring unit resets the sensor fusion system, transmits a fault message to the ADAS including the automatic emergency braking system, temporarily drives only the camera module, Driving the fusion system,
The camera module battery monitoring unit periodically monitors whether or not power is normally supplied to the camera module. When a fault occurs, the camera module battery monitoring unit manages power by a PMIC (Power Management Integrated Circuit) To inform the fact of the failure,
The Raider module battery monitoring unit periodically monitors whether or not power is normally supplied to the Raider module. When a failure occurs, the Raider module battery monitoring unit manages the power by the PMIC, informs the fault monitoring unit of the failure,
The battery monitoring unit periodically monitors whether the PMIC is normally operating, outputs an alarm when a failure occurs, resets the sensor fusion system,
Upon receiving the fault detection signal from the hardware watchdog, the fault monitoring unit resets the sensor fusion system, transmits a fault message to the ADAS including the automatic emergency braking system, And,
Wherein the fault monitoring unit receives a fault from the time clock unit, transmits a fault message to the ADAS including the automatic emergency braking system, resets the camera module and the Lida module, And outputs an alarm and switches to a manual mode.
delete delete delete delete delete delete delete

Images