TW202226063A - Multi-sensor autonomous system - Google Patents

Abstract

A multi-sensor autonomous system is disclosed. The multi-sensor autonomous system includes an image signal receiving module, for receiving continuous image signals; a sensing signal receiving module, for receiving continuous sensing signals; and a processing unit, for analyzing the continuous image signals and/or the continuous sensing signals to determine whether an identified target is included in the continuous image signals and/or the continuous sensing signals, and generating a trigger signal while the identified target is confirmed to be included, wherein the processing unit marks the corresponding image signal and/or the sensing signal according to the trigger signal.

Description

Multi-sensor autonomous driving system

The invention relates to the field of automatic driving, in particular to a multi-sensor automatic driving system.

Environmental awareness is a very important key technology for autonomous driving technology. A robust autonomous driving system needs to be equipped with sensors with different characteristics, and can only grasp the dynamics of the environment after being judged by artificial intelligence.

However, artificial intelligence needs to mark and learn from a large number of signals collected by different sensor signals, and only after adjusting the parameters can the optimized performance of the system be obtained to improve the ability to perceive the environment. Therefore, when developing an ideal autonomous driving system, the efficiency of signal collection is more important. Nowadays, in-vehicle intelligence has sensors with different functions such as cameras, millimeter-wave radar, ultrasonic radar and LiDAR. It is necessary to build an integrated platform to achieve efficient signal collection, which can be used as a function of signal synchronization and integration, and sensor signals occupy a huge storage space when collected as image signals, so how to effectively record a large number of sensors at the same time. Signals are then classified, marked and learned, improving storage space utilization, and even sorting and filtering before back-end signal processing, which will be problems that must be solved in the development of autonomous driving technology. In addition, to verify whether the developed artificial intelligence can effectively realize the functions required by autonomous driving, it is necessary to integrate the developed artificial intelligence into a hardware platform, and the relevant specifications and settings of the hardware platform are different from the signal collection system. , and the hardware platform needs to be installed on the real vehicle for verification.

In order to solve the above technical problems, a multi-sensor automatic driving system is provided.

The object of the present invention can be achieved through the following technical solutions:

A multi-sensor automatic driving system includes an image signal receiving module for receiving continuous image signals, a sensor signal receiving module for receiving continuous sensing signals, and a processing unit for converting the continuous image signals The signal or/and the continuous sensing signal are analyzed to determine whether an identification target is included, and when it is confirmed that the identification target is included, a trigger signal is generated, wherein the processing unit marks the corresponding identification target according to the trigger signal. Image signal or/and sensing signal.

It also includes a storage module, characterized in that the processing unit stores the continuous image signal or/and the continuous sensing signal in the storage module.

It also includes a storage module, characterized in that the processing unit stores the marked image signal or/and the marked sensing signal in the storage module.

The recognition target can be set by multiple conditions, and can be set by means of intersection or union according to the multiple conditions.

The image signal is an LVDS format signal.

The sensing signal is at least one of an ultrasonic signal, a lidar signal, a millimeter-wave radar signal, a GPS signal, and a driving data signal.

The sensor signal receiving module further includes a sensor signal receiving block with an Ethernet interface for receiving the LiDAR signal.

The processing unit performs time synchronization on the continuous image signal and the continuous sensing signal. Compared with the prior art, the present invention has the following advantages: 1. After the multi-sensor automatic driving system of the present invention has set the recognition target and set the storage method, it marks the received continuous image signal and continuous sensor signal, and finally the stored signal can be classified into the system through the system. , the developer of the autonomous driving system can then perform effective back-end signal processing on the collected image signals or/and sensing signals, which is conducive to the development of in-vehicle artificial intelligence; 2. Through the system configuration of the present invention, it includes an image signal receiving module and a sensor signal receiving module, and further includes a sensor signal receiving block with an Ethernet interface for receiving the LiDAR signal. . Among them, the image signal is in LVDS signal format, and the transmitted image quality is better, which is more favorable for the development of image artificial intelligence. In addition, the system provides a sensor signal receiving module with a CAN bus interface, which can simultaneously receive different types of sensing A fast Ethernet interface is also provided for the LiDAR signal for transmission. The artificial intelligence developed by the developer can directly use the system of the present invention to be loaded on the vehicle to realize the real vehicle verification.

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention. Accordingly, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, It is not indicated or implied that the device or element referred to must have a particular orientation, be constructed and operate in a particular orientation, and therefore should not be construed as limiting the invention.

In the description of the present invention, "plurality" means two or more, unless otherwise expressly and specifically defined.

In the present invention, unless otherwise expressly specified and limited, the terms "installed", "connected", "connected", "fixed" and other terms should be understood in a broad sense, for example, it may be a fixed connection or a detachable connection , or integrated; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of the two elements or the interaction relationship between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

The present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments:

Referring to FIG. 1 , the present invention discloses a multi-sensor automatic driving system 100 including an image signal receiving module 101 , a sensor signal receiving module 102 , a processing unit 103 and a storage module 104 .

The image signal receiving module 101 is used for receiving continuous image signals, and has an image processing unit (ISP, Image Signal Processor), which can process such as lens correction, image element correction, color interpolation, Bayer noise removal, white balance correction, color correction Correction, gamma correction, color space conversion and other functions. The image signal receiving module 101 generally has a LVDS (Low Voltage Differential Signaling, low voltage differential signal) or MIPI CSI transmission interface. The image signal comes from one or more cameras connected to the vehicle. The camera is usually installed around the vehicle to capture driving images. In order to obtain better image quality, the image signal can be in LVDS format.

The sensor signal receiving module 102 is used for receiving continuous sensing signals, and the sensing signals are at least one of ultrasonic signals, lidar signals, millimeter-wave radar signals (mmWave Radar), GPS signals, and driving data signals. Among them, ultrasonic signals, millimeter-wave radar signals, GPS signals, and driving data signals (including, for example, vehicle speed, engine speed, steering angle, acceleration, file position and other sensor signals, etc.) can be received through the sensor signal receiving module 102 CAN bus interface transmission. The sensor signal receiving module 102 further includes a sensor signal receiving block 1020 with an Ethernet interface for receiving the LiDAR signal, and high-speed LiDAR signal transmission can be realized through the Ethernet interface .

The processing unit 103 is the main operation unit of the system of the present invention, and has built-in optimized artificial intelligence. Usually, the processing core is a DSP (digital signal processor, digital signal processor). DSP is suitable for various multiplication and addition operations (SOP: Sum of Products), such as: Finite Impulse Response Filtering (FIR: Finite Impulse Response), Infinite Impulse Response (IIR: Infinite Impulse Response), Discrete Fourier Transform (DFT) : Discrete Fourier Transform), discrete cosine transform (DCT: Discrete Cosine Transform), dot product operation (Dot product), convolution operation (Convolution), and matrix polynomial evaluation operations. The processing unit 103 has a memory transmission interface and can execute instructions built into the memory. The processing unit 103 also has an image transmission interface, is connected with the image signal receiving module, and operates the image signal converted from the image signal receiving module. The processing unit 103 also includes a CAN bus interface, which is connected to the CAN bus of the sensor signal receiving module, and calculates sensing signals such as ultrasonic signals, millimeter-wave radar signals, GPS signals, and driving data signals. In addition, the processing unit 103 also has a high-speed transmission interface, supports high-speed Ethernet network interface transmission, and mainly operates the lidar signal from the sensor signal receiving module.

The main function of the storage module 104 is to store image signals or/and various sensing signals. The storage module 104 may be a built-in integrated circuit memory, or may be an external storage device, such as an SSD or an SD card. The storage file is stored in the form of frame numbers. Each data in the file must conform to the corresponding storage format. For example, the field identification data starts with the field data format, such as Camera image, Radar, LiDAR..., field data Length, the field is synchronized with the current time and other formats.

Please refer to FIG. 1 and FIG. 2 together for an embodiment of a process for AI signal collection by a multi-sensor autonomous driving system 100 according to the present disclosure. When the multi-sensor automatic driving system 100 of the present invention is installed on a real vehicle, the following steps are performed:

Step S1, setting the recognition target and the storage mode. The setting method can be used to operate the multi-sensor automatic driving system 100 through a user interface (not shown) when the system is started, or to preload the identification target into the system through a program when the system is started. , the way of loading can be through physical transmission devices, such as USB socket, memory card, etc., or through wireless transmission, such as Bluetooth, wireless network, etc. The recognition target can be set with a single setting or multiple conditions, and can be set in the form of intersection or union according to multiple conditions, that is, two or more different recognition targets can be set, and the intersection means that different recognition targets exist at the same time. Union means that any recognition target exists. The storage mode is to store different data types according to the identification target, which can be determined according to different sensor signals, the format of the stored signal, the size of the data of the stored signal, etc. In the storage mode of the present invention, it can be set when the system detects When the target is identified, whether to store the relevant image or other sensor signals when the target is identified, and even the number of frames to be stored can be adjusted according to the settings, such as the frame before and after the system detects the target. number setting. On the contrary, the system of the present invention can be configured to store relevant images or other sensor signals when the identification target is not present.

Generally speaking, the recognition target may include behaviors such as specific objects, events, manipulations, etc. In particular, the specific target that is judged by artificial intelligence mainly based on images can also be the target that needs artificial intelligence to judge other sensing signals. Examples of specific objects may include pedestrians, various types of vehicles, street signs, sign signs, utility poles, lane markings, sidewalks, lane types, potholes, curbs, parking grids, floor locks, stop bars, and the like. Examples of specific events may include rain, fog, or snow driving weather in the environment, as well as ambient illuminance while driving, or even traffic jams, vehicle breakdowns, vehicle crashes, sensor failures, or shading affecting functionality. Examples of specific manipulations can be failure of automatic driving functions, such as failure of automatic parking, failure of automatic lane change, etc., or intervention of driver behavior, such as emergency braking, rapid steering and other behaviors. The above identification targets are mainly based on artificial intelligence judgment, but the present invention is not limited to this, of course, it can also include sensor signals, such as GPS setting range, vehicle speed, ambient temperature and humidity, driving intervention time and so on.

Step S2, receiving an image signal or/and a sensing signal. After the setting in the above-mentioned steps is completed, when the vehicle moves, the image signal receiving module 101 is configured through the system of the present invention to receive continuous image signals, and the sensor signal receiving module 102 is used to receive continuous image signals. sensing signal. The continuous image signal and the continuous sensing signal will be temporarily stored in the shared memory of the dynamic memory (not marked) built in the system before entering the processing unit 103 of the system of the present invention to analyze and determine whether there is an identification target. In the shared memory area, the processing unit 103 performs time synchronization on the continuous image signals and the continuous sensing signals accessed in the shared memory area.

Step S3, processing unit analysis. The multi-sensor automatic driving system 100 of the present invention has a processing unit 103, and artificial intelligence has been built in the processing unit 103, which can identify the recognition target, that is, the multi-sensor automatic driving of the present invention The processing unit 103 of the system 100 has been built with optimized artificial intelligence, which can identify the existing identification targets of the system, improve the efficiency of signal collection or the convenience of subsequent signal screening and processing, so as to develop other automatic driving functions. Signal collection operations. The processing unit 103 analyzes the continuous image signal or/and the continuous sensing signal according to the recognition target set in step S1 to determine whether the recognition target is included, and when it is confirmed that the recognition target is included, Generate trigger signal. That is to say, the processing unit 103 of the present invention can select the analyzed signal according to the identification target feature selected in step S1. If the operation is only an artificial intelligence operation of the image signal, it will only analyze the image signal without analyzing other sensing signals. signal to save system computation.

The trigger signal is generated by the processing unit 103 and stored in the dynamic memory (not marked) built into the system of the present invention, and the trigger signal may include the processing unit 103 for the system of the present invention. Relevant actions, such as controlling the system (or vehicle), stopping the system (or vehicle), etc., or performing signal processing such as marking and storing the continuous image signal or/and the continuous sensing signal.

Step S4, mark storage. The processing unit 103 marks the continuous image signal or/and the continuous sensing signal corresponding to the trigger signal generated in the foregoing steps. The mark is mainly generated according to the setting in step S1. The specific method can be that the processing unit 103 generates text data, time start and end data, type mark of the recognition target, identification of the continuous image signal or/and the continuous sensing signal. At least one of the target's checkmark, the number of frames stored, or the confidence index per frame, etc. For the marked continuous image signal or/and the continuous sensing signal, the processing unit 103 calls the storage command of the dynamic memory built in the system of the present invention to store the image signal or/and the sensing signal in the storage module 104 of the system of the present invention. The storage of the signal can be performed according to the setting in step S1, and the optional method is: (1) The processing unit 103 stores the continuous image signal or/and the continuous sensing signal in the storage module 104, that is, the storage system After the setting is activated according to step S1, the image signal or/and the sensing signal starts to be stored, but part of the continuous image signal and part of the continuous sensing signal already have corresponding mark signals. (2) The processing unit 103 stores the marked image signal or/and the marked sensing signal in the storage module 104, that is, only the marked signal is stored, and the other unmarked signals are stored. The signal is not stored in the storage module 104 .

Through the stored signal obtained in step S4, the system of the present invention classifies and marks the stored signal, and the developer of the automatic driving system can re-mark, filter, and parameterize the collected image signal or/and sensing signal through the signal. Adjustment, algorithm improvement and other related artificial intelligence learning and training, and finally the trained artificial intelligence can be updated for offline verification, or the artificial intelligence real vehicle verification can be carried out through the hardware platform built by the system of the present invention. For signal collection, the aforementioned steps S1 to S4 can be repeated until the developed automatic driving function is optimized.

Now, the multi-sensor automatic driving system 100 of the present invention is implemented as follows, and please refer to the description of FIG. 1 and FIG. 2 together.

For example, the multi-sensor automated driving system 100 is described as an implementation using the storage of a marker whose identification target is a vehicle frame. The common automatic driving system needs to collect the corresponding vehicle frame signal when realizing the automatic parking function. Therefore, in the step S1 of the system of the present invention, the identification target and the storage mode need to be set. In identifying the target, since the grid signal includes the grid line, the grid number, other vehicles, ground obstacles (such as ground locks, limit bars, etc.) and other obstacles (such as pedestrians, animals, etc.), or Ambient illumination, etc. Developers can select single or multiple conditions according to different recognition targets. For example, only a single recognition target can be selected when the grid line is double, or when the grid line is selected at the same time and the ambient illumination is below 100 LUX The recognition target is even for the recognition target that selects the grid line as a single line or a double line. The storage mode can be set to store the type of signal with the identification target, and can further select to store only the image signal without storing other sensor signals, or store the image signal and other sensor signals at the same time. Furthermore, the storage amount of the stored image signal can be set, for example, the number of frames from when the system (processing unit 103 ) detects the start of the recognition target to the end of the detection of the recognition target.

After the identification target and the setting of the storage mode are completed, in step S2, the image signal receiving module 101 is turned on to receive continuous image signals, and the sensor signal receiving module 102 is turned on to receive continuous sensing signals.

Then, the processing unit 103 performs the analysis in step S3 for the continuous image signal received by the image signal receiving module 101 or/and the continuous sensing signal received by the sensor signal receiving module 102 . Since the processing unit 103 of the multi-sensor automatic driving system 100 of the present invention already has artificial intelligence, the signal can be distinguished according to the setting of step S1. For example, when a single recognition target is selected when the grid line is a double line, and only the image signal is stored without storing other sensor signals, the processing unit 103 calculates the artificial intelligence according to the characteristics of the grid line, and only analyzes the image signal and obtains it. Without analyzing other sensing signals, if the processing unit 103 analyzes that the image signal has the identification target set in step S1, it will generate a trigger signal. When the trigger signal is a storage command in this example, the trigger signal will be stored in the signal. The storage module 104 enters step S4, and only stores the image signal and does not store other sensing signals.

The content of the stored signal in step S4 can also be stored in different ways according to the setting in step S1, and the options are: (1) The processing unit 103 stores the continuous image signal or/and the continuous sensory signal. The measurement signal is stored in the storage module 104, that is, after the system is started, the image signal or/and the sensing signal starts to be stored, but part of the continuous image signal and part of the continuous sensing signal The signal has the flag signal that the step S3 has been completed. (2) The processing unit 103 stores the marked image signal or/and the marked sensing signal in the storage module 104, that is, only the marked signal is stored, and the other unmarked signals are stored. The signal is not stored in the storage module 104 . Specifically, the storage method of the present invention can store relevant images or other sensing signals when the recognition target is not present. For example, the system of the present invention can choose not to store the vehicle compartment recognized by the processing unit 103 as It only stores the image signal of the grid line when the grid line is a single line, and only stores the grid line image signal of the grid line other than the single line.

The following is a description of the implementation of the multi-sensor autonomous driving system 100 , for example, using the storage of markers identifying the target as an obstacle. In the development of automatic driving functions, the detection of obstacles usually requires more than two sensor signals for analysis and judgment. Usually the sensors are cameras and ultrasonic sensors. The purpose is to receive signals from different sensors. , through the artificial intelligence calculation, it can better identify the obstacles, such as grasping the obstacle type and obstacle distance, and even merging different sensor signals. Taking the space detection in the development of automatic parking as an example, in step S1, multiple types of obstacles can be set as the identification targets for signal collection, such as vehicles parked in the car compartment, ground locks, limit levers, etc. These multiple identification targets are selected in the way of union. The storage mode can be set to only store the sensor signal when the processing unit 103 generates a trigger signal after analyzing the recognition target, where the sensor signal includes all image signals and ultrasonic signals, and starts with the detection of the recognition target All frames from the time to the end of detecting and identifying the target are stored.

After the setting is completed, similarly, in step S2, the image signal receiving module 101 is turned on to receive continuous image signals, and the sensor signal receiving module 102 is turned on to receive continuous sensing signals. In this example, the sensor signal At least include ultrasonic signals.

In step S3, the processing unit 103 analyzes the continuous image signals received by the image signal receiving module 101 or/and the continuous sensing signals received by the sensor signal receiving module 102. Since the processing unit 103 of the multi-sensor automatic driving system 100 of the present invention has artificial intelligence, it can analyze the image signal and the ultrasonic signal at the same time according to the obstacle setting condition in step S1. If the processing unit 103 analyzes the image signal, step S1 When the identification target is set, a trigger signal (storage command) is generated, and according to the trigger signal, the signal is stored in the storage module 104, that is, the step S4 is entered. According to the step S1, the image signal and the ultrasonic signal are stored at this time. . Specifically, when the processing unit 103 generates a trigger signal for the identification target (vehicle parked in the compartment, ground lock, limit lever), the collected signal at this time is in addition to the identification target type of the image signal and the image calculated In addition to the distance signal, it also includes the distance signal corresponding to the ultrasonic signal. Therefore, the system of the present invention can effectively screen, classify and store these huge signals for the storage of the mark of the obstacle or the signal collection before the fusion of the signals of different transmitters.

For developers, in addition to the more efficient collection of sensor signals through the system of the present invention, the system configuration of the present invention includes an image signal receiving module 101 and a sensor signal receiving module 102 , and also includes a sensor signal receiving block 1020 with an Ethernet interface, which is used to receive the lidar signal. The artificial intelligence developed by the developer can directly use the system of the present invention to be loaded on the vehicle to achieve Vehicle verification.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention in any form. Although the present invention has been disclosed above with preferred embodiments, it is not intended to limit the present invention. Personnel, without departing from the scope of the technical solution of the present invention, can make some changes or modifications to equivalent examples of equivalent changes by using the methods and technical contents disclosed above, but all content that does not depart from the technical solution of the present invention, according to the present invention. The technical essence of the invention Any simple modifications, equivalent changes and modifications made to the above embodiments still fall within the scope of the technical solutions of the present invention.

100: Multi-sensor autonomous driving systems 101: Video signal receiving module 102: Sensor signal receiving module 103: Processing unit 104: Storage Module 1020: The sensor signal receiving block of the Ethernet interface

FIG. 1 is a schematic structural diagram of a multi-sensor automatic driving system of the present invention. FIG. 2 is a schematic diagram of a process of a multi-sensor automatic driving system for artificial intelligence signal collection according to the present invention.

100: Multi-sensor autonomous driving systems

101: Video signal receiving module

102: Sensor signal receiving module

103: Processing unit

104: Storage Module

1020: The sensor signal receiving block of the Ethernet interface

Claims (8)

A multi-sensor autonomous driving system comprising: an image signal receiving module for receiving continuous image signals; a sensor signal receiving module for receiving continuous sensing signals; and a processing unit that analyzes the continuous image signal or/and the continuous sensing signal to determine whether an identification target is included, and generates a trigger signal when it is confirmed that the identification target is included, The processing unit marks the corresponding image signal or/and the sensing signal according to the trigger signal. The multi-sensor automatic driving system according to claim 1, further comprising a storage module, wherein the processing unit stores the continuous image signal or/and the continuous sensing signal in the storage module. The multi-sensor automatic driving system of claim 1, further comprising a storage module, wherein the processing unit stores the marked image signal or/and the marked sensor The signal is stored in the storage module. The multi-sensor automatic driving system according to claim 1, wherein the identification target can be set by multiple conditions, and can be set by means of intersection or union according to the multiple conditions. The multi-sensor automatic driving system according to claim 1, wherein the image signal is a LVDS format signal. The multi-sensor automatic driving system according to claim 1, wherein the sensing signal is at least one of an ultrasonic signal, a lidar signal, a millimeter-wave radar signal, a GPS signal, and a driving data signal. 6. The multi-sensor automatic driving system of claim 6, wherein the sensor signal receiving module further comprises a sensor signal receiving block having an Ethernet interface for receiving the light signal. The multi-sensor automatic driving system according to claim 1, wherein the processing unit performs time synchronization on the continuous image signal and the continuous sensing signal.

Images