CN113734166A - Automobile automatic driving control system and method based on perception fusion SWC - Google Patents
Automobile automatic driving control system and method based on perception fusion SWC Download PDFInfo
- Publication number
- CN113734166A CN113734166A CN202111155811.5A CN202111155811A CN113734166A CN 113734166 A CN113734166 A CN 113734166A CN 202111155811 A CN202111155811 A CN 202111155811A CN 113734166 A CN113734166 A CN 113734166A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- rte
- swc
- signal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 82
- 230000008447 perception Effects 0.000 title claims abstract description 52
- 238000000034 method Methods 0.000 title claims description 14
- 230000006870 function Effects 0.000 claims abstract description 83
- 238000012545 processing Methods 0.000 claims abstract description 36
- 230000004888 barrier function Effects 0.000 claims abstract description 5
- 238000004891 communication Methods 0.000 claims description 34
- 230000003993 interaction Effects 0.000 claims description 15
- 230000005540 biological transmission Effects 0.000 claims description 9
- 230000002093 peripheral effect Effects 0.000 claims description 9
- 230000002159 abnormal effect Effects 0.000 claims description 7
- 238000012216 screening Methods 0.000 claims description 7
- 238000007405 data analysis Methods 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 5
- 230000001681 protective effect Effects 0.000 claims description 4
- 238000004806 packaging method and process Methods 0.000 claims description 3
- 230000010354 integration Effects 0.000 claims description 2
- 230000003044 adaptive effect Effects 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 6
- 238000013461 design Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 238000007781 pre-processing Methods 0.000 description 4
- 238000007726 management method Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005538 encapsulation Methods 0.000 description 2
- 238000002955 isolation Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000000889 atomisation Methods 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
- B60W30/14—Adaptive cruise control
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R16/00—Electric or fluid circuits specially adapted for vehicles and not otherwise provided for; Arrangement of elements of electric or fluid circuits specially adapted for vehicles and not otherwise provided for
- B60R16/02—Electric or fluid circuits specially adapted for vehicles and not otherwise provided for; Arrangement of elements of electric or fluid circuits specially adapted for vehicles and not otherwise provided for electric constitutive elements
- B60R16/023—Electric or fluid circuits specially adapted for vehicles and not otherwise provided for; Arrangement of elements of electric or fluid circuits specially adapted for vehicles and not otherwise provided for electric constitutive elements for transmission of signals between vehicle parts or subsystems
- B60R16/0231—Circuits relating to the driving or the functioning of the vehicle
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
- B60W30/08—Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
- B60W30/09—Taking automatic action to avoid collision, e.g. braking and steering
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2552/00—Input parameters relating to infrastructure
- B60W2552/50—Barriers
Landscapes
- Engineering & Computer Science (AREA)
- Automation & Control Theory (AREA)
- Mechanical Engineering (AREA)
- Transportation (AREA)
- Traffic Control Systems (AREA)
Abstract
The invention discloses an automobile automatic driving control system based on perception fusion, which relates to an automobile automatic driving technology, integrates three-level automatic driving perception codes and AEB, ACC and other simulink-based SWC function modules in an open source compiling environment, constructs an automobile automatic driving framework with complete data transceiving, perception processing, target identification and decision planning through a middleware interface component, realizes the autonomous realization of low-cost L1L2 automatic driving functions, and can realize function expansion and iteration. Collecting and identifying the output road target and barrier data, integrating all external SWC module signals, verifying safety functions, giving signal states, completing signal distribution, fusing multi-sensor data by adopting a single-thread data processing mode, and outputting target-level data; and controlling the vehicle to avoid the obstacle in the emergency of the vehicle to the front obstacle, and realizing the intelligent self-adaptive cruise of the vehicle by the ACC function SWC.
Description
Technical Field
The invention relates to an automobile control system, in particular to emergency obstacle avoidance and intelligent adaptive cruise in an unmanned automobile.
Background
The sensing fusion and control function codes which can be operated on the low-cost system on chip are a commercialized integrated software whole body represented by Bosch and a corresponding hardware system. The first scheme is as follows: and respectively and independently realizing the perception classification of the target environment, the planning and the control of the vehicle track based on the perception code and the control code of various open source development platforms. The cross-platform functions are dispersed, and effective overall development is difficult. Scheme II: the full-function architecture based on L1L2 of the whole vehicle-level controller comprises automatic driving first-level and second-level functions such as AEB, IACC and the like. The function code can not be opened and is difficult to modify, and the cost is high.
Publication No.: chinese invention patent CN112429012A, "car electric control system, automatic driving control method and car", discloses a car electric control system, automatic driving control method and car, the system includes: the intelligent cockpit area processing unit, the automatic driving area processing unit, the vehicle control area processing unit, the central control area processing unit and the sensor system are respectively connected with the Ethernet bus; an automatic driving limp algorithm program is backed up in the intelligent cabin area processing unit; when the vehicle control domain processing unit monitors that the running state of the automatic driving domain processing unit is abnormal and the running state of the intelligent cabin domain processing unit is normal, the vehicle control domain processing unit controls the intelligent cabin domain processing unit to run an automatic driving limp algorithm program and receive sensing data in the Ethernet bus to perform fusion operation and decision control so as to control the vehicle control domain processing unit to perform vehicle limp safety control. The problem of failure of the automatic driving area controller is solved, safety of vehicles and passengers is guaranteed, and safety backup of functions of the automatic driving area controller is achieved. The intelligent cockpit area processing unit, the automatic driving area processing unit, the vehicle control area processing unit, the central control area processing unit and the sensor system are respectively connected with the Ethernet bus; an automatic driving limp algorithm program is backed up in the intelligent cabin area processing unit; the automatic driving domain processing unit is used for receiving sensing data loaded by the sensor system in the Ethernet bus and outputting a corresponding automatic driving control instruction to the vehicle control domain processing unit; the vehicle control domain processing unit is used for receiving the automatic driving control instruction and controlling vehicle motion based on the automatic driving control instruction; the vehicle control domain processing unit is used for monitoring the running states of the automatic driving domain processing unit, the intelligent cabin domain processing unit and the vehicle control domain in real time, and when the running state of the automatic driving domain processing unit is monitored to be abnormal and the running state of the intelligent cabin domain processing unit is monitored to be normal, the intelligent cabin domain processing unit is controlled to run the automatic driving limp algorithm program and receive sensing data in the Ethernet bus to perform fusion operation and decision control so as to control the vehicle control domain processing unit to perform vehicle limp safety control.
The automatic driving system with complete functions based on the domain controller in the prior art disclosed by the patent has the advantages of large system volume, high cost and complex system, is directly used for an auxiliary safety automatic driving function mainly based on automatic emergency braking and intelligent self-adaptive cruise, has obvious waste of software and hardware resources, does not particularly aim at various formats acquired by various sensors of an automobile to contain relevant perception data related to various information for fusion, is not beneficial to different automobile types, and is beneficial to sharing and interactive utilization of different system data; this patent uses automatic emergency braking, intelligent self-adaptation to cruise as the functional requirement, has developed lightweight, low-cost, high performance autopilot system based on fuse perception data information to its characteristic specially.
Disclosure of Invention
The invention aims at the characteristics of cross-platform function dispersion, difficult effective overall development, incapability of opening sources and difficult modification of function codes and high cost. Integrating three-level automatic driving perception codes, automatic emergency braking AEB, intelligent self-adaptive cruise ACC and other functional modules based on simulink of a matlab toolkit in an open source compiling environment HighTec, and constructing an automobile automatic driving framework with complete data transceiving, perception processing, target identification and decision planning through a middleware interface assembly RTE, so that the autonomous realization of low-cost auxiliary or partial auxiliary automatic driving functions is realized, and the function expansion and iteration can be realized.
The technical scheme for solving the technical problems is that the automobile automatic driving control system based on perception fusion comprises: the system comprises a data transceiver module, a signal middleware interface RTE module, a perception fusion SWC module, an emergency braking AEB function SWC module and an ACC function SWC module, wherein the data transceiver module acquires data of a road target and an obstacle in front of the driving of an automobile in real time and transmits the data to the signal middleware interface RTE module according to a specified signal format of a preset signal list; the signal middleware interface RTE module is responsible for integrating all SWC module signals, verifying safety functions, determining and giving signal states (such as normal signals, abnormal signal exceeding thresholds and the like) and finishing signal distribution; the sensing fusion module SWC fuses multi-sensor data in a single-thread data processing mode, and screens and outputs a certain amount of target-level data which have close influence on driving safety and driving performance according to a vehicle driving area; according to the obtained target level data, the AEB function SWC module controls the vehicle to avoid the obstacle in the front in an emergency mode, and the ACC function SWC controls the vehicle to achieve intelligent self-adaptive cruise.
The data transceiver module further includes: the front-view camera can acquire the speed, the course and other information of the vehicle through an intermediate interface RTE module, and finally sends the target information identified by the image to a rear-end perception fusion module through the intermediate interface for use. The front millimeter wave radar acquires signals such as required vehicle speed and course information through the middle data interface, and sends the targets identified by clustering to the rear-end perception fusion module through the middle interface for use.
Furthermore, a data transceiver module of the front-view camera is integrated on a master chip, a horizon line target identification module is integrated on a slave chip, the master chip and the slave chip adopt a serial peripheral interface to interact data through an RTE module, and road target and barrier data are packaged by using an SPI communication protocol to form each signal API (application program interface) function with a prefix of 'Rte _ fc _ signal name'; the front millimeter wave radar data transceiver module is integrated on a main chip, performs data interaction with a vehicle body system through a vehicle body CAN (controller area network) network interface, performs interaction between the RTE module and the FR module by adopting a CAN communication protocol, and performs data analysis and packaging by using factors, offsets, max and min parameters.
After receiving the FR data of the forward millimeter wave radar, the RTE module encapsulates the FR data by using an SPI communication protocol to form various signal API functions with prefixes of 'Rte _ FR _ signal names', encapsulates the data to form a standard external interface function, wherein signals related to the front camera FC are encapsulated in Rte _ spi.h, signals related to the millimeter wave radar FR and vehicle body data are encapsulated in Rte _ Com.h, and signals related to the sensing fusion module SDF are encapsulated in Rte _ sdf.h.
The RTE module and all SWC modules perform data interaction through corresponding interface functions, data from the RTE module are distinguished by prefix RTE, redefinition is performed on the same data to obtain signals meeting the requirements of the SWC module, a rollingConter (CHECK flag bit) is used for validity CHECK, data rationality judgment is performed according to dbc physical signal definition, and data which passes the validity and rationality judgment are output to two states of RTE _ E _ OK and RTE _ E _ CHECK.
The input end of the perception fusion module SWC is provided with a sensor module for receiving data which are collected by all sensors of an abstract sensor receiver AbstractSensorRecevier and contain pure virtual functions, the data are respectively classified and provided with three receiving sub-class objects of Ethernet data, CAN data and CANFd data according to Ethernet, CAN network or CANFd network, and the three sub-class objects respectively contain pure virtual functions.
The perception fusion module SWC sets an abstract _ parser parent class, and converts the original data collected by the sensor into a readable and usable data form according to data parsing realized by a data transmission mode, a manufacturer, a model and a CAN dbc or signal list provided by the sensor.
On the basis of data receiving and analyzing, a perception fusion module SWC is internally provided with a perception preprocessing duration module, unreasonable data are removed according to the data characteristics of each sensor, then target data fusion and lane line data fusion are carried out, target screening is carried out according to the driving area of the vehicle, and targets which have close influence on driving safety and driving performance are output.
The system comprises a vehicle body, a time synchronization module, a space synchronization module, a sequence container, a time synchronization module and a data acquisition module, wherein the time synchronization module is arranged on the vehicle body and is used for acquiring the time interval of the vehicle body; the space synchronization module takes the vehicle front protective center point as an origin, and takes a vehicle coordinate system with left negative, right positive, front positive and back negative as a reference coordinate system of all the sensors.
The invention also provides an automobile automatic driving control method based on perception fusion, which comprises the steps that multiple sensors of a forward-looking camera, a horizon target recognition module and a front millimeter wave radar data transceiver module acquire and recognize road targets and barrier data in front of the running of the automobile in real time, integration and safety function verification are carried out, the signal state is confirmed and signal distribution is completed, single-thread data processing mode is adopted to fuse the data of the multiple sensors, and a certain amount of target-level data which closely influences the running safety and the running performance are screened and output according to the running area of the automobile; according to the obtained target level data, the AEB function SWC module controls the vehicle to avoid the obstacle in the front in an emergency mode, and the ACC function SWC controls the vehicle to achieve intelligent self-adaptive cruise.
The front-view camera data transceiver module and the horizon line target identification module are in data communication through an SPI (serial peripheral interface), road target and obstacle data are packaged through an SPI (serial peripheral interface) communication protocol to form various signal API (application program interface) functions with prefixes of 'Rte _ fc _ signal names', the front millimeter wave radar data transceiver module is in data interaction with a vehicle body system through a vehicle body CAN (controller area network) interface, and the RTE module and the FR module are in data interaction through a CAN communication protocol.
After receiving FR data, the RTE module encapsulates the FR data by using an SPI communication protocol to form each signal API function with a prefix of 'RTE _ FR _ signal name', encapsulates all data by the RTE module to form a standard external interface function, wherein signals related to FC are encapsulated in RTE _ spi.h, signals related to FR and vehicle body data are encapsulated in RTE _ Com.h, and signals related to SDF are encapsulated in RTE _ sdf.h.
The RTE module and all SWC modules perform data interaction through corresponding interface functions, data from the RTE are distinguished by prefix RTE, the same data are redefined by using signal definition to obtain signals meeting the requirements of the SWC module, effective bit verification is performed by using signal zone bits, data rationality judgment is performed according to dbc physical signal definition, and data judged by validity and rationality are output to two states of RTE _ E _ OK and RTE _ E _ CHECK.
The input end of the sensing fusion module SWC is provided with a sensor module which receives data which are acquired by an AbstractSensorRecevier parent object and are responsible for managing all sensors of the automobile and contain a pure virtual function recevie, and three receiving sub-class objects of Ethernet data, Can data and Canfd data are respectively arranged according to the classification of Ethernet, CAN network or CANFd network, wherein the three sub-class objects respectively contain pure virtual functions. The perception fusion module SWC sets an abstract _ parser parent class, realizes data parsing according to a data transmission mode, a manufacturer, a model and a CAN dbc or signal list provided by the sensor, and converts original data acquired by the sensor into a readable and usable data form; the perception fusion module SWC is internally provided with a perception module, unreasonable data are removed according to the data characteristics of each sensor, then target data fusion and lane line data fusion are carried out, and target screening is carried out according to the driving area of the vehicle.
The time synchronization module adopts system absolute time as a time axis, selects the output time of any sensor as reference time, utilizes a vector container and a map container to sequentially query, compares the difference value of each frame of data of the other sensors with the reference time in sequence, and searches a frame with the minimum time interval as synchronized time; the space synchronization module takes the vehicle front protective center point as the origin, and takes the vehicle coordinate system with left negative, right positive, front positive and back negative as the reference coordinate system of all the sensors
And an abstract sensor module receiving function is arranged at the input end of the perception fusion module SWC and is responsible for managing the acquisition and reception of all sensors of the automobile. Meanwhile, the system CAN be used for various working conditions, Ethernet, CAN and CANFd communication protocols are developed, and the multi-state concept in C + + is utilized for realization. The perception fusion module SWC sets an abstact _ parser parent class, and converts the original data collected by the sensor into a readable and usable data form according to the data analysis realized by the data transmission mode, manufacturer, model and the CAN message or signal list provided by the sensor. The conversion of the external sensor or the communication protocol thereof only needs to change and analyze partial codes, thereby realizing the isolation and the independence of data transmission and having better granularity. On the basis of data receiving and analyzing, a perception fusion module SWC is internally provided with a perception module, unreasonable data are removed according to data characteristics of each sensor, a target data fusion module and lane line data fusion are carried out, and a target screening module is carried out according to a vehicle driving area. And a time synchronization module and a space synchronization module are also arranged to solve the time synchronization problem and the space synchronization problem of the sensors such as cameras and radars which are arranged at different positions of the vehicle body.
The invention provides a complete system based on L1L2 (auxiliary automatic driving), which realizes the functions of automatic emergency braking, intelligent self-adaptive cruise and the like. All signals of the system are managed in a unified mode through the data intermediate interface, different targets and data collected by the sensors are subjected to format unification, classification identification and effective fusion, data requirements in aspects of driving, control, communication and the like of the intelligent automobile are met, an automatic driving new function interface is reserved, specifications and possibilities are provided for different data utilization of sensing equipment such as different types of sensors, and system support is provided for future function upgrading. The system can be deployed on low-power-consumption and low-cost control chips such as a system on a chip due to a simplified framework, and meanwhile, the system adopts a modular design, so that the system functions are fully decoupled, and the design idea of the automatic driving function atomization service is met.
Drawings
FIG. 1 is a block diagram of an automatic driving control system of an automobile according to the present invention.
Detailed Description
The invention is further illustrated and described with reference to the accompanying drawings and specific examples. The embodiment is only used for clearly illustrating and explaining the implementation process of the invention, and the protection scope of the invention is not limited to the embodiment, and any technical solution obtained based on simple reasoning and logic combination of those skilled in the art on the basis of the embodiment shall also belong to the protection scope of the invention.
Fig. 1 shows a schematic block diagram of the system structure of the present invention. The system comprises a forward-looking camera data transceiving module, a front millimeter wave radar data transceiving module, a signal middleware interface RTE module, a perception fusion SWC module, an automatic emergency braking AEB function perception fusion SWC and an adaptive cruise ACC function perception fusion SWC. The forward-looking camera data transceiver module acquires road target and pedestrian data which are identified and output by the monocular camera in real time and transmits the road target and pedestrian data according to a predetermined signal list signal format; the module is externally connected with a front-view camera composed of a horizon dual-core chip, the camera integrates a monocular camera eye identification kernel, and a data interface is provided. And the front millimeter wave radar data transceiver module acquires road target data sent by the front millimeter wave radar in real time and transmits the road target data according to a predefined signal format. The data intermediate interface RTE module is a central control of the whole system architecture and is responsible for integrating and uniformly managing signals of all external SWC modules and all functional modules, and signal distribution and management of all module signals need to be carried out through the RTE.
The sensing fusion module SWC provides sensing data for realizing automatic driving for multi-sensor data fusion, and then outputs target-level data with relatively high reliability, meanwhile, according to the compiling environment and hardware characteristics, the sensing fusion adopts a single-thread data processing mode, target information detected by a camera and a millimeter wave radar, vehicle speed and course information of a vehicle are subjected to data association, data fusion and target evaluation steps, and a fusion target result is output, wherein the fusion target result comprises the following steps: and the transverse and longitudinal position, the transverse and longitudinal speed, the transverse and longitudinal acceleration, the course angle, the tracking serial number, the target type, the target motion state and other comprehensive information.
The SDF perception fusion module SWC defines an abstract receiving type pointer according to an actual external hardware sensor and a communication mode, and the realization of a specific receiving cancel function can be realized by pointing to the communication mode by using the pointer; defining an abstract data analysis pointer, realizing the function of analyzing a Parser (data analysis) function of concrete data by pointing the pointer to a concrete sensor, analyzing each sensor according to a signal list predefined by a manufacturer, and finally analyzing original sensor data obtained by receiving data into target data (such as target position, speed, number, motion attribute and the like) required by the interior of a sensing module; and on the basis, performing Perception persistence (Perception preprocessing) on the target, and performing data filtering on the target data obtained by analysis to eliminate obviously abnormal and unreliable data. Then an obj fusion module associates data of the same target with data of multiple sensors, aggregates the data of the multiple sensors into the same target by using a mahalanobis distance and probability ellipse equidistant matching mode, and then performs data fusion by using the output characteristics of each sensor: the FC camera has good target classification performance and accurate identification, but the output target has large transverse position fluctuation, is easily influenced by weather and has insufficient speed measurement capability on a remote object; the millimeter wave radar FR has good capability of testing an object moving at a high speed by using the doppler effect, but is easily interfered by the environment, and the output target position information is unstable. The fusion strategy fully considers the dominant working conditions of each sensor, fuses the associated target attributes into the optimal attributes in the current environment, and outputs target attribute information (target position, speed, number, motion attributes and the like) with higher redundancy and higher precision; the Lane line fusion module performs rationality judgment by using the ground lane line cubic curve parameters acquired by the FC camera, filters input accompanying interference according to the principle that a lane line cannot be suddenly distorted and needs to be continuously smooth, and outputs stable and reliable lane line parameters (cubic curve parameters); target _ selector (fusion target screening) filters out targets and pedestrians outside the lane and the left and right adjacent lanes by using results output by obj fusion and Lanefusion, and numbers and outputs surrounding vehicles according to front, back, left, right, side front and side rear positions by taking the vehicle as the center in combination with the requirement of the auxiliary automatic driving function.
The automatic emergency braking AEB function SWC module realizes emergency obstacle avoidance of the vehicle on the structured road to a front obstacle, the adaptive cruise ACC function SWC module realizes intelligent adaptive cruise of the vehicle on the structured road, respective function logics can be developed by adopting a simulink tool in matlab, the AEB and the ACC utilize target attributes output by the SDF to carry out internal logic operation according to target positions and motion states, vehicle control signals such as an electronic steering wheel steering angle, an accelerator pedal control quantity, a brake pedal control quantity and the like are finally generated, and a rear-end execution mechanism controls the vehicle. Other development tools and platform implementations may also be employed.
In the embodiment, an AC chip is combined with a horizon J2 vision processing chip to output targets, three-level automatic driving C + + sensing codes, automatic emergency braking AEB, adaptive cruise ACC and other functional modules developed based on matlab are integrated in an open source compiling environment Hightec, and a complete automobile automatic driving framework with data transceiving, sensing processing, target identification and decision planning is constructed through a data intermediate interface RTE, so that the autonomous implementation of a low-cost auxiliary automatic driving function is realized, and the function expansion and iteration can be realized.
And the high htec cross-platform compiling environment is utilized, data interaction and verification of all modules are realized through the middleware RTE, and the modularization and the standardized management of the system architecture are realized under the condition of ensuring smooth data. The method comprises the steps that a millimeter wave radar, a front-view camera and vehicle body information which are used as external equipment are connected to an RTE (remote time center) as external input, a sensing fusion SWC module uses an FR (forward millimeter wave radar), an FC (front camera) and the vehicle body information to conduct target fusion, a fusion target result is output and sent back to the RTE, an automatic emergency braking and self-adaptive cruise module mounted on the RTE uses the fusion target as input, operation is conducted in a module of the module, an actual vehicle control variable is finally obtained and sent to the RTE, the actual vehicle control variable is finally received by a vehicle, ESP vehicle body posture control is conducted, and automatic driving is achieved.
The front-view camera data transceiver module can be integrated on a main chip (for example, an English-flying SAK-TC233LP-32F200N AC chip can be adopted), the data sending mode is a serial peripheral interface, and the horizon target identification chip is a slave chip. The data communication of the master chip and the slave chip is interacted through a data intermediate interface RTE, the bottom layer utilizes a serial peripheral interface communication protocol to realize the encapsulation of the output target of the camera, and finally, signal application program interface functions with prefixes of ' Rte _ fc ' (data intermediate interface _ front camera target information) signal names ' are formed, so that the RTE can obtain the target data of the camera from the signal application program interface functions; the front millimeter wave radar data transceiver module is integrated on the main chip, is different from the front-view camera, and performs CAN communication through a CAN interface to perform data interaction with an automobile hardware system. Therefore, CAN communication is adopted between the intermediate data interface RTE and the front millimeter wave radar FR, and signals are received and transmitted according to the definition of a CAN message signal list.
The data communication of the foresight camera data transceiver module and the horizon line target identification module exchanges data through an SPI (serial peripheral interface), the data of a road target and an obstacle are packaged by using an SPI (serial peripheral interface) communication protocol to form various signal API (application program interface) functions with prefixes of 'Rte _ fc _ signal names', the foresight millimeter wave radar data transceiver module exchanges data with a vehicle body system through a vehicle body CAN (controller area network) interface, the RTE module and the FR module exchange through a CAN communication protocol, and the parameters are analyzed and packaged by using offset and proportion.
After receiving the FR data, the RTE module encapsulates the FR data by using an SPI communication protocol to form each signal API function with a prefix of "RTE _ FR _ signal name (millimeter wave radar intermediate interface data)", and encapsulates all data to make it a standard external interface function, wherein FC-related signals are encapsulated in RTE _ spi.h (front camera intermediate interface data), FR and body data signals are encapsulated in RTE _ com.h (body intermediate interface data), and SDF-related signals are encapsulated in RTE _ sdf.h (fused data intermediate interface data).
The RTE module and all the SWC modules perform data interaction through corresponding interface functions, data prefixes from the RTE are distinguished by Rte, and the same data is redefined by using variable name definition to obtain signals meeting the requirements of the SWC modules. And verifying the valid bit by using the mark CHECK bit, judging the data rationality according to the definition of the physical signal of the signal list, and outputting the data which is judged to be valid and rational to two states of RTE _ E _ OK (signal normal) and RTE _ E _ CHECK (signal abnormal).
The RTE may specifically perform signal distribution by performing data interaction with other components through the SPI after receiving the FR data, and the bottom layer of the RTE implements encapsulation of each signal by using the SPI communication protocol, and finally forms each signal API function with a prefix "RTE _ FR _ (data intermediate interface _ millimeter wave radar target information) signal name" so that the RTE can obtain millimeter wave target data from the signal API function.
And the RTE middleware designs corresponding SWC components according to functional requirements, limits the receiving and transmitting signals for all the components and realizes unified management of interfaces.
In RTE, all signals in the requirements are packaged by using open source compiling software HighTec, so that all signals become standard external interface functions. The signals related to the front camera FC are packaged in a Rte _ spi.h head (front camera middle interface data) file, the output target of the millimeter wave radar FR and the vehicle body data signals are packaged in a Rte _ Com.h (vehicle body middle interface data) head file, the signals related to the sensing fusion output target are packaged in a Rte _ sdf.h (fusion data middle interface data) head file, and the control function terminal module calls the signals from the packaging library for use.
In order to ensure the unification of the signal interface and the design mode, the data interaction of the data intermediate interface module RTE and all the functional module signals adopts corresponding intermediate interface data: each target signal output by the front camera FC has an external interface function, and the signal can be acquired by the rear end. For example, if the sensing fusion module SDF needs to use the target longitudinal position output by the front camera, the SDF directly calls a signal acquisition function which is already encapsulated as Rte _ spi.h, and the return value of the function is the target longitudinal position, and the other signals are the same; meanwhile, the data interface module can also judge the rationality when transmitting each signal: each signal contains, in addition to the physical definition itself, a set of signal plausibility flags RTE _ E _ OK (signal normal) and RTE _ E _ CHECK (signal abnormal) within its function. In the data intermediate module RTE, corresponding rationality judgment logic is programmed for each signal, if a signal receiving signal exceeds a threshold value, RTE _ E _ CHECK is set to be 1, RTE _ E _ OK is set to be 0, otherwise, RTE _ E _ CHECK is set to be 0, and RTE _ E _ OK is set to be 1.
And the perception fusion module SWC integrates related functions of three-level automatic driving. In order to realize the communication protocol required by the same sensing code to adapt to various types of sensors, the fused input end utilizes a C + + polymorphic mode to establish a parent abstract layer receiving class Receive (receiving abstract class), and according to different sensors which are actually accessed, specific receiving modes and signal analysis are respectively realized in subclasses. Finally, the obtained target signals output by the sensors are used for the back end to perceive the fusion codes for data association and fusion; the sensor communication protocol is mainly divided into Ethernet, CAN (area network controller communication protocol) network or CANFd (frame message data segment baud rate variable communication protocol), the three subclasses are all inherited from a parent class abstract layer receiving class, and in the concrete receiving implementation, the various subclasses implement the functions of the respective required communication protocols in a polymorphic mode.
The input end of the perception fusion module SWC is provided with a Sensor module which receives data which are acquired by Abstract Sensor receivers and contain pure virtual functions and collected by all sensors of an automobile, and the data are respectively classified and provided with three receiving sub-class objects of Socket _ Sensor _ Receiver (Ethernet data receiving), Can _ Sensor _ Receiver (Can data receiving) and Canfd _ Sensor _ Receiver (Canfd data receiving) according to Ethernet, CAN or CANFd networks, wherein the three receiving sub-class objects respectively contain the virtual functions and the pure virtual functions and the data are collected by all sensors of the automobile.
The perception fusion module SWC sets an abstact _ parser parent class, and converts the original data collected by the sensor into a readable and usable data form according to the data transmission mode, manufacturer, model and CAN signal list or data parsing realized by the signal list provided by the sensor; on the basis of data receiving and analyzing, a perception fusion module SWC is internally provided with a perception preprocessing module, unreasonable data is removed according to the data characteristics of each sensor, an obj fusion module and a Lane line data fusion module are carried out, and target _ selector (fusion target screening) is carried out according to the driving area of the vehicle.
On the basis of realizing normal receiving of the original data of the sensor, the original data are all stored in a raw data RawSensorMsg (raw data of the sensor) structure body of the sensor for rear-end analysis; by setting an abstract analysis parent class Parser (abstract analysis parent class), the analysis work of all original data is uniformly managed. According to data analysis realized by each data transmission mode, manufacturer, model and a signal list provided by the sensor, original data collected by the sensor is converted into a readable and usable data form, and finally the data is uniformly stored in a preset sensor internal data SensorMsg (sensor internal data) structure body, and the conversion of an external sensor or a communication protocol thereof only needs to change and analyze partial codes, so that the isolation and the independence of data transmission are realized, and the granularity is better.
On the basis of obtaining sensor data, a perception algorithm fusion module SDF realizes the association and fusion of multi-sensor data, a perception module is utilized to carry out data preprocessing according to the data characteristics of each sensor, unreasonable data are eliminated, then an obj fusion module and a Lane line data fusion module are utilized to realize the 'strong and short' of the output characteristics of the sensors, finally, a vehicle area is screened for target _ selector according to the driving area of the vehicle, and finally, a certain number (such as 0-18) of targets which have close influence on the driving safety and the driving performance are output as a final output result.
The result is target-level data with relatively high reliability, and the output accuracy is higher than that of a single sensor. The time synchronization problem of different sensors is considered at the same time: the time synchronization module adopts system absolute time as a time axis, selects the output time of one sensor as reference time, utilizes a data container to sequentially inquire, compares the difference value of each frame of data of the other sensors with the reference time in sequence, and searches a frame with the minimum time interval as synchronized time; the outputs of the sensors installed at different positions of the vehicle body need to be spatially synchronized: the space synchronization module takes the vehicle front protective center point as an origin, and takes a vehicle coordinate system with left negative, right positive, front positive and back negative as a reference coordinate system of all the sensors.
And the automatic emergency braking AEB, the adaptive cruise ACC and other functional software modules SWC use the fusion target result output by the perception fusion module SDF through the intermediate interface RTE, obtain the actual vehicle control quantity after the internal operation of the AEB and ACC modules, and send the vehicle body control signal to the vehicle end through the RTE to realize automatic driving.
Claims (15)
1. An automobile automatic driving control system based on perception fusion is characterized by comprising: the system comprises a data transceiver module, a signal middleware interface RTE module, a perception fusion SWC module, an AEB function SWC module and an ACC function SWC module, wherein the data transceiver module acquires data of a road target and an obstacle in front of the driving of an automobile in real time and transmits the data to the signal middleware interface RTE module according to a preset dbc signal format; the signal middleware interface RTE module integrates and verifies the safety function of all received SWC module signals, determines the signal state and completes signal distribution; the sensing fusion module SWC fuses multi-sensor data in a single-thread data processing mode, and selects and outputs target-level data which have close influence on driving safety and driving performance according to a vehicle driving area; the AEB function SWC module controls the vehicle to avoid the obstacle to the front obstacle urgently according to the target level data, and the ACC function SWC controls the vehicle to intelligently and adaptively cruise.
2. The system of claim 1, wherein the data transceiver module comprises: the system comprises a foresight camera data transceiver module, a front millimeter wave radar data transceiver module and a horizon line target identification module, wherein the foresight camera data transceiver module is integrated on a master chip, the horizon line target identification module is integrated on a slave chip, the master chip and the slave chip adopt a serial peripheral interface to exchange data through an RTE module, and the road target and barrier data are encapsulated by using an SPI communication protocol to form each signal API function with a prefix of 'Rte _ fc _ signal name'; the front millimeter wave radar data transceiver module is integrated on the main chip, performs data interaction with a vehicle body system through a vehicle body CAN network interface, performs interaction between the RTE module and the FR module by adopting a CAN communication protocol, and performs data analysis and packaging by using factors, offsets, max and min parameters.
3. The system of claim 1, wherein the FR data received by the RTE module is encapsulated using SPI communication protocol to form each signal API function prefixed by "RTE _ FR _ signal name" to be a standard external interface function, FC related signals are encapsulated in RTE _ spi.h, FR and body data signals are encapsulated in RTE _ com.h, and SDF related signals are encapsulated in RTE _ sdf.h.
4. The system according to any one of claims 1-3, characterized in that the RTE module and all SWC modules perform data interaction through corresponding interface functions, the data of the RTE module is distinguished by prefix Rte, the same data is redefined by typedefend to obtain a signal meeting the requirements of the SWC module, the rollingConter is used for validity check, data rationality judgment is performed according to dbc signals, and the data which passes the validity and rationality judgment is output in two states of signal normal and signal abnormal.
5. The system according to any one of claims 1-3, wherein the sensor module is arranged at the SWC input end of the perception fusion module to receive data collected by an AbstractSensorRecevier parent object and responsible for managing all sensors of the automobile, wherein the data comprises pure virtual functions, and three receiving subclass objects of Ethernet data, Can data and Canfd data are respectively arranged according to the classification of Ethernet, CAN network or CANFd network, and each of the three subclass objects comprises pure virtual functions.
6. The system of claim 5, wherein the perception fusion module SWC sets an abstract parsing parent class, implements data parsing according to a data transmission mode, a manufacturer, a model and a CAN dbc or signal list provided by the sensor, and converts raw data collected by the sensor into a readable and usable data form.
7. The system according to claim 5, wherein the perception fusion module SWC is internally provided with a perception module, unreasonable data is removed according to the data characteristics of each sensor, then target data fusion and lane line data fusion are carried out, target screening is carried out according to the driving area of the vehicle, and targets which have close influence on driving safety and driving performance are output.
8. The system according to any one of claims 1 to 7, characterized in that a time synchronization and space synchronization module is further provided to synchronize the sensors such as cameras and radars which are installed at different positions of the vehicle body, the time synchronization module uses the system absolute time as a time axis, selects the output time of any one of the sensors as a reference time, utilizes a sequence container to perform sequential query, compares the data of each frame of the other sensors with the reference time in sequence, and finds the frame with the minimum time interval as the synchronization time; the space synchronization module takes the front guarantee center point of the vehicle as the origin, and takes the vehicle coordinate system with left negative, right positive, front positive and back negative as the reference coordinate system of all the sensors.
9. A perception fusion-based automatic driving control method for an automobile is characterized by comprising the steps that multiple sensors of a forward-looking camera, a horizon target recognition module and a front millimeter wave radar data transceiver module acquire and recognize road target and barrier data in front of the driving of the automobile in real time, integration and safety function verification are carried out, the signal state is confirmed and signal distribution is completed, a single-thread data processing mode is adopted to fuse the data of the multiple sensors, and a certain amount of target-level data which closely influence the driving safety and the driving performance are screened and output according to the driving area of the automobile; the AEB function SWC module controls the vehicle to avoid the obstacle in the front according to the target level data, and the ACC function SWC controls the vehicle to intelligently and adaptively cruise.
10. The method as claimed in claim 9, wherein the data communication between the forward looking camera data transceiver module and the horizon line target identification module exchanges data through an SPI serial peripheral interface, the road target and obstacle data are encapsulated by using an SPI communication protocol to form each signal API function with a prefix of "Rte _ fc _ signal name", the forward millimeter wave radar data transceiver module exchanges data with the car body system through a car body CAN network interface, and the data exchange between the Rte module and the FR module is performed by using a CAN communication protocol.
11. The method of claim 9, wherein the RTE module encapsulates the received FR data using SPI communication protocol to form each signal API function prefixed by "RTE _ FR _ signal name", and encapsulates all data to be standard external interface functions, wherein FC related signals are encapsulated in RTE _ spi.h, FR related signals and body data signals are encapsulated in RTE _ com.h, and SDF related signals are encapsulated in RTE _ sdf.h.
12. The method according to claim 11, characterized in that the RTE module and all SWC modules perform data interaction through corresponding interface functions, data from the RTE are distinguished by prefix RTE, redefining the same data to obtain signals meeting the requirements of the SWC module, performing valid bit verification by using a signal flag bit, performing data rationality judgment according to a dbc physical signal, and outputting two states of RTE _ E _ OK and RTE _ E _ CHECK through the data judged by validity and rationality.
13. The method according to claim 11, wherein the sensor module is arranged at an input end of the perception fusion module SWC, and receives data which is acquired by all sensors and contains a purely virtual function recevie, and three receiving sub-class objects of ethernet data, CAN data and CANFd data are respectively arranged according to the classification of ethernet, CAN network or CANFd network, and each of the three sub-class objects contains a purely virtual function.
14. The method as claimed in claim 13, wherein the perception fusion module SWC sets an abstract parsing parent class, implements data parsing according to a data transmission mode, a manufacturer, a model, and a CAN dbc or a signal list provided by the sensor, and converts raw data collected by the sensor into a readable and usable data form; the perception fusion module SWC is internally provided with a perception module, unreasonable data are removed according to the data characteristics of each sensor, then target data fusion and lane line data fusion are carried out, and target screening is carried out according to the driving area of the vehicle.
15. The method according to one of claims 9 to 14, wherein the time synchronization module uses system absolute time as a time axis, selects the output time of any sensor as a reference time, utilizes sequence container to query in sequence, compares the data of each frame of the other sensors with the reference time in sequence, and finds the frame with the minimum time interval as the time after synchronization; the space synchronization module takes the vehicle front protective center point as an origin, and takes a vehicle coordinate system with left negative, right positive, front positive and back negative as a reference coordinate system of all the sensors.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111155811.5A CN113734166B (en) | 2021-09-30 | 2021-09-30 | Automatic automobile driving control system and method based on sensing fusion SWC |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111155811.5A CN113734166B (en) | 2021-09-30 | 2021-09-30 | Automatic automobile driving control system and method based on sensing fusion SWC |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113734166A true CN113734166A (en) | 2021-12-03 |
| CN113734166B CN113734166B (en) | 2023-09-22 |
Family
ID=78741913
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111155811.5A Active CN113734166B (en) | 2021-09-30 | 2021-09-30 | Automatic automobile driving control system and method based on sensing fusion SWC |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113734166B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114684154A (en) * | 2022-03-24 | 2022-07-01 | 重庆长安汽车股份有限公司 | Method for correcting visual detection target course angle based on radar point cloud and storage medium |
| CN115497195A (en) * | 2022-08-09 | 2022-12-20 | 重庆长安汽车股份有限公司 | Data control system, method, device and medium for realizing driving record |
| CN118144806A (en) * | 2024-05-06 | 2024-06-07 | 北京茵沃汽车科技有限公司 | Camera sensor and fault detection method thereof |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20100070590A (en) * | 2008-12-18 | 2010-06-28 | 재단법인대구경북과학기술원 | Method for authenticating control signal of vehicle's electronic control unit, control signal transmitter and ecu using the same |
| CN103318176A (en) * | 2013-06-28 | 2013-09-25 | 郑州宇通客车股份有限公司 | Coach self-adaptive cruise control system and control method thereof |
| CN108445885A (en) * | 2018-04-20 | 2018-08-24 | 鹤山东风新能源科技有限公司 | A kind of automated driving system and its control method based on pure electric vehicle logistic car |
| CN110147109A (en) * | 2019-05-21 | 2019-08-20 | 重庆长安汽车股份有限公司 | A kind of archetype development system of automated driving system |
| CN111469838A (en) * | 2020-04-22 | 2020-07-31 | 芜湖伯特利汽车安全系统股份有限公司 | Collaborative ACC/AEB decision management system based on Internet of vehicles and vehicle |
| CN111634290A (en) * | 2020-05-22 | 2020-09-08 | 华域汽车系统股份有限公司 | Advanced driving assistance forward fusion system and method |
| CN113313154A (en) * | 2021-05-20 | 2021-08-27 | 四川天奥空天信息技术有限公司 | Integrated multi-sensor integrated automatic driving intelligent sensing device |
-
2021
- 2021-09-30 CN CN202111155811.5A patent/CN113734166B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20100070590A (en) * | 2008-12-18 | 2010-06-28 | 재단법인대구경북과학기술원 | Method for authenticating control signal of vehicle's electronic control unit, control signal transmitter and ecu using the same |
| CN103318176A (en) * | 2013-06-28 | 2013-09-25 | 郑州宇通客车股份有限公司 | Coach self-adaptive cruise control system and control method thereof |
| CN108445885A (en) * | 2018-04-20 | 2018-08-24 | 鹤山东风新能源科技有限公司 | A kind of automated driving system and its control method based on pure electric vehicle logistic car |
| CN110147109A (en) * | 2019-05-21 | 2019-08-20 | 重庆长安汽车股份有限公司 | A kind of archetype development system of automated driving system |
| CN111469838A (en) * | 2020-04-22 | 2020-07-31 | 芜湖伯特利汽车安全系统股份有限公司 | Collaborative ACC/AEB decision management system based on Internet of vehicles and vehicle |
| CN111634290A (en) * | 2020-05-22 | 2020-09-08 | 华域汽车系统股份有限公司 | Advanced driving assistance forward fusion system and method |
| CN113313154A (en) * | 2021-05-20 | 2021-08-27 | 四川天奥空天信息技术有限公司 | Integrated multi-sensor integrated automatic driving intelligent sensing device |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114684154A (en) * | 2022-03-24 | 2022-07-01 | 重庆长安汽车股份有限公司 | Method for correcting visual detection target course angle based on radar point cloud and storage medium |
| CN115497195A (en) * | 2022-08-09 | 2022-12-20 | 重庆长安汽车股份有限公司 | Data control system, method, device and medium for realizing driving record |
| CN118144806A (en) * | 2024-05-06 | 2024-06-07 | 北京茵沃汽车科技有限公司 | Camera sensor and fault detection method thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113734166B (en) | 2023-09-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113734166B (en) | Automatic automobile driving control system and method based on sensing fusion SWC | |
| US20210266295A1 (en) | Deterministic Container-Based Network Configurations for Autonomous Vehicles | |
| Obst et al. | Multi-sensor data fusion for checking plausibility of V2V communications by vision-based multiple-object tracking | |
| CN202827403U (en) | Vehicle controller | |
| CN106448190B (en) | Real-time monitoring and early warning device and method for traffic flow around self-vehicle on highway | |
| CN106394446B (en) | A kind of novel body control system and control method | |
| CN108334087B (en) | Software definition-based platform advanced driving assistance system | |
| CN111762145B (en) | Intermediary agent device, electric control vehicle driving system and operation method thereof | |
| US11897511B2 (en) | Multi-hypothesis object tracking for automated driving systems | |
| CN111209956A (en) | Sensor data fusion method, and vehicle environment map generation method and system | |
| CN108958248A (en) | Standby system | |
| CN112590784A (en) | Domain control system and method for passenger car | |
| US10576977B2 (en) | Vehicle device | |
| CN116901875A (en) | Perception fusion system, vehicle and control method | |
| US11691566B2 (en) | Intelligent vehicle systems and control logic for surround view augmentation with object model recognition | |
| CN110794712A (en) | Automatic driving virtual scene in-loop test system and method | |
| CN212275970U (en) | Vehicle-mounted millimeter wave radar system based on wireless communication | |
| CN109017634B (en) | Vehicle-mounted network system | |
| CN112351407A (en) | AEB strategy method based on 5G hierarchical decision | |
| CN115792944B (en) | Road rapid calibration method and system matched with laser radar | |
| CN220430132U (en) | Vehicle collision early warning system | |
| Ali et al. | An Efficient Design of a Basic Autonomous Vehicle Based on CAN Bus | |
| CN102955156B (en) | Blind spot detection system | |
| CN114212079B (en) | ACC-based vehicle control method, device and system | |
| US11987266B2 (en) | Distributed processing of vehicle sensor data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |