WO2021026768A1

WO2021026768A1 - Automatic driving method and apparatus based on data stream, and electronic device and storage medium

Info

Publication number: WO2021026768A1
Application number: PCT/CN2019/100382
Authority: WO
Inventors: 姜浩; 蔡权雄; 牛昕宇
Original assignee: 深圳鲲云信息科技有限公司
Priority date: 2019-08-13
Filing date: 2019-08-13
Publication date: 2021-02-18
Also published as: CN112840284A

Abstract

An automatic driving method and apparatus based on a data stream, and an electronic device and a storage medium. The method comprises: acquiring a neural network diagram and parameters of an automatic driving model, wherein the parameters are pre-trained parameters (201); performing, according to the neural network diagram and the parameters of the automatic driving model, configuration on a data stream architecture to obtain a data stream automatic driving model corresponding to the automatic driving model (202); acquiring image information used for automatic driving (203); inputting the image information into the data stream automatic driving model for processing to obtain an image processing result (204); and sending the image processing result to a driving decision-making module to create a driving decision (205). The method can increase the identification and detection speed for an image during automatic driving, so as to improve the real-time performance of automatic driving; in addition, a data stream automatic driving model has low requirements for hardware, such that the cost and power consumption of automatic driving can be reduced.

Description

Automatic driving method, device, electronic equipment and storage medium based on data stream

Technical field

This application relates to the field of artificial intelligence, and more specifically, to a data stream-based automatic driving method, device, electronic equipment, and storage medium.

Background technique

Artificial neural network (artificial neural network, abbreviation ANN), abbreviated as neural network (neural network, abbreviation NN) or neural network, is a kind of imitating the structure and function of biological neural network (animal's central nervous system, especially the brain) Mathematical model or calculation model, used to estimate or approximate the function.

The neural network is mainly composed of: input layer, hidden layer, and output layer. When there is only one hidden layer, the network is a two-layer neural network. Since the input layer has not undergone any transformation, it can not be regarded as a separate layer. In practice, each neuron in the input layer of the network represents a feature, and the number of output layers represents the number of classification labels (when doing binary classification, if a sigmoid classifier is used, the number of neurons in the output layer is 1 ; If the softmax classifier is used, the number of neurons in the output layer is 2), and the number of hidden layers and hidden layer neurons are manually set.

Neural network application works well in classification problems. Classification problems are mostly in the industry. LR or linear SVM is more suitable for linear classification. If the data is non-linear and separable (mostly non-linear in real life), LR usually needs to rely on feature engineering to do feature mapping, adding Gaussian terms or combination terms; SVM needs to select a kernel. The addition of Gaussian terms and combination terms will produce many useless dimensions and increase the amount of calculation. GBDT can be combined into a strong classifier using weak linear classifiers, but the effect may not be good when the dimensionality is high. When the neural network has three or more layers, it can perform nonlinear separability well.

Deep learning has been applied to various fields at present, and the application scenarios are roughly divided into three categories: object recognition, target detection, and natural language processing.

Target detection can be understood as the integration of object recognition and object positioning, not only to identify which category the object belongs to, but more importantly to get the specific location of the object in the picture. In order to accomplish these two tasks, target detection models are divided into two categories. One type is two-stage, which divides object recognition and object positioning into two steps and completes them separately. Typical representatives of this type are R-CNN, fast R-CNN, and faster-RCNN families. They have low recognition error rates and low missed recognition rates, but they are slower and cannot meet real-time detection scenarios. In order to solve this problem, another type of method appeared, called one-stage, typical representatives are Yolo, SSD, YoloV2, etc. Their recognition speed is very fast, can meet the real-time requirements, and the accuracy rate can basically reach the level of faster R-CNN.

Currently, there are camera-based solutions and solutions based on other sensors such as lidar for sensing modules for autonomous driving on the market. There are two main ways to use the camera, one is based on traditional feature extraction algorithms, using DSP and other hardware for computational acceleration; the other is based on deep learning algorithms, mainly using GPU for computational acceleration. The first solution has problems such as insufficient recognition accuracy, difficulty in adapting to multiple scenes, and low robustness; the second solution uses neural networks for target detection and segmentation algorithms, which solves the problem of the first algorithm Shortcomings such as low accuracy, but higher calculations lead to higher power consumption, high hardware requirements, high cost, high power consumption, difficult heat dissipation, and low real-time performance.

Application content

The purpose of this application is to provide a data stream-based automatic driving method, device, electronic equipment, and storage medium in response to the above-mentioned defects in the prior art, which solves the problem of target detection when the existing neural network is used to make target automatic driving decisions. During the segmentation process, the amount of calculation required is high, which leads to the problems of high cost, high power consumption, and difficult heat dissipation.

The purpose of this application is achieved through the following technical solutions:

In a first aspect, a data stream-based deep network acceleration method is provided, the method includes:

Acquiring a neural network diagram and parameters of the autonomous driving model, where the parameters are pre-trained parameters;

According to the neural network diagram and parameters of the automatic driving model, configure on the data flow architecture to obtain the data flow automatic driving model corresponding to the automatic driving model;

Obtain image information for automatic driving;

Inputting the image information into the data stream automatic driving model for processing to obtain an image processing result;

The image processing result is sent to the driving decision module to form a driving decision.

Optionally, the configuring and obtaining the data flow automatic driving model corresponding to the automatic driving model on the data flow architecture according to the neural network diagram and parameters of the automatic driving model includes:

According to the neural network diagram, configure parallel or serial between multiple neural network layers;

According to the parameters, allocate the data stream memory corresponding to each neural network layer, and the data stream is internally used to store the parameters of the corresponding neural network layer;

Forming a data flow path between the multiple neural network layers based on the parallel or serial between the multiple neural network layers and allocating data flow memory corresponding to each neural network layer;

According to the data flow path, the data flow automatic driving model is formed.

Optionally, the allocating data stream memory corresponding to each neural network layer according to the parameters includes:

Specify a starting memory address for each parameter data block preloaded in the neural network layer;

Starting from the designated starting memory address, a memory space with the same size as the parameter data block is opened up and allocated to the parameter data block for loading.

Optionally, the acquiring image information used for automatic driving includes:

Obtain image information from the image source and store the obtained image information in the image memory;

Read image information from the image memory.

Optionally, the method further includes:

If reading the image information from the image memory fails, the reading is performed again within a predetermined time.

Optionally, after the input of the image information into the data stream automatic driving model for processing, the method further includes:

The result obtained after the data flow automatic driving model is processed is post-processed to obtain an image processing result.

Optionally, the image processing result includes the category and coordinate data of the object feature, and the sending the image processing result to the driving decision module to form a driving decision includes:

The category and coordinate data of the object features are sent to the driving decision module to form a driving decision.

In a second aspect, an automatic driving device based on data stream is also provided, the device including:

The first acquisition module is used to acquire the neural network diagram and parameters of the automatic driving model, where the parameters are pre-trained parameters;

The configuration module is configured to configure the data flow automatic driving model corresponding to the automatic driving model on the data flow architecture according to the neural network diagram and parameters of the automatic driving model;

The second acquisition module is used to acquire image information for automatic driving;

A processing module, configured to input the image information into the data stream automatic driving model for processing to obtain an image processing result;

The sending module is used to send the image processing result to the driving decision module to form a driving decision.

In a third aspect, an electronic device is provided, including: a memory, a processor, and a computer program stored on the memory and capable of running on the processor, and the processor implements the implementation of the application when the processor executes the computer program Examples provide steps in the data stream-based automatic driving method.

In a fourth aspect, a computer-readable storage medium is provided, and a computer program is stored on the computer-readable storage medium. The computer program, when executed by a processor, implements the data stream-based automatic driving method provided in the embodiments of the present application A step of.

The beneficial effects brought by this application: the corresponding data flow automatic driving model is constructed through the neural network graph and parameters. Since the data flow automatic driving model is a non-instruction set model, there is no instruction idle overhead, which can accelerate the recognition of images in automatic driving In addition, the data flow automatic driving model does not require high hardware requirements, which can reduce the cost and power consumption of automatic driving.

Description of the drawings

FIG. 1 is a schematic diagram of an optional implementation architecture of a data stream-based automatic driving method provided by an embodiment of this application;

2 is a schematic flowchart of a data stream-based automatic driving method provided by an embodiment of the application;

FIG. 3 is a schematic flowchart of another automatic driving method based on data flow provided by an embodiment of the application;

FIG. 4 is a schematic diagram of a data stream-based automatic driving device provided by an embodiment of the application;

FIG. 5 is a schematic diagram of a specific structure of a configuration module 402 provided by an embodiment of the application;

FIG. 6 is a schematic diagram of a specific structure of an allocation unit 4022 provided by an embodiment of the application;

FIG. 7 is a schematic diagram of a specific structure of the first obtaining module 401 provided by an embodiment of the application;

FIG. 8 is a schematic diagram of another automatic driving device based on data flow provided by an embodiment of the application;

FIG. 9 is a schematic diagram of another automatic driving device based on data flow provided by an embodiment of the application.

detailed description

The preferred embodiments of the present application are described below. Those of ordinary skill in the art will be able to implement them with related technologies in the field according to the following description, and will be able to better understand the innovations and benefits of the present application.

In order to further describe the technical solution of the present application, please refer to FIG. 1. FIG. 1 is a schematic diagram of an optional implementation architecture of a data stream-based automatic driving method provided by an embodiment of the present application. As shown in FIG. The storage module (DDR) 101 and the CPU are connected by interconnection. The architecture 103 includes: a first storage module 104, a global data flow network 105 and a data flow engine 106. The first storage module 104 is connected to the off-chip storage through interconnection. The module 101 is also connected to the global data flow network 105 through interconnection, and the data flow engine 106 is connected to the global data flow network 105 through the interconnection so that the data flow engine 106 can realize parallel or serial. The aforementioned data flow engine 106 may include: a computing core (or called a computing module), a second storage module 108, and a local data flow network 107. The computing core may include a core for computing, such as a convolution core 109 and a pooling core. 110 and activation function core 111, etc. Of course, it can also include other calculation cores besides the example convolution core 109, pooling core 110, and activation function core 111, which are not limited here, and can also be included in the neural network. The kernel used for calculation. The above-mentioned first storage module 104 and the above-mentioned second storage module 108 may be on-chip cache modules, or may be DDR or high-speed DDR memory modules. The above-mentioned data stream engine 106 can be understood as a computing engine that supports data stream processing, and can also be understood as a computing engine dedicated to data stream processing. The above-mentioned data flow architecture can be implemented on an FPGA programmable gate array.

This application provides a data stream-based automatic driving method, device, equipment and storage medium.

For the first aspect, please refer to FIG. 2. FIG. 2 is a schematic flowchart of a data stream-based automatic driving method provided by an embodiment of the present application. As shown in FIG. 2, the method includes the following steps:

201. Acquire a neural network diagram and parameters of an automatic driving model, where the parameters are pre-trained parameters.

The above-mentioned neural network diagram can be understood as a neural network structure, and further, can be understood as a neural network structure for an autonomous driving model. The above-mentioned neural network structure uses layers as calculation units, including but not limited to: convolutional layer, pooling layer, ReLU, fully connected layer, etc. The aforementioned parameters refer to the parameters corresponding to each layer in the neural network structure, and may be weight parameters, bias parameters, and so on. The aforementioned autopilot model can be a pre-trained autopilot model. Since the autopilot model is pre-trained, the attributes of its parameters are also trained. Therefore, the configured data flow autopilot model can be directly based on the configured parameters. In use, there is no need to train the data stream autopilot model. According to the pre-trained autopilot model, a unified description can be made through neural network graphs and parameters. The above-mentioned neural network diagram and parameters of the autopilot model can be obtained locally or on a cloud server. For example, the above-mentioned neural network diagram and parameters can be stored locally in complete sets, and they can be automatically selected during use or The user makes a selection, or uploads the neural network diagram and parameters to the cloud server, and downloads the neural network diagram and parameters in the cloud server through the network when in use.

202. According to the neural network diagram and parameters of the automatic driving model, configure on the data flow architecture to obtain a data flow automatic driving model corresponding to the automatic driving model.

The above neural network diagram includes the connection relationship between the data flow engine, the first data flow storage module, and the global data flow network. The above connection relationship may include the number of connections of the data flow engine, the connection sequence, etc., and the data flow engine Connect with the global data flow network through interconnection to form a corresponding autonomous driving model. In addition, different neural networks can be formed according to different neural network diagrams. The above-mentioned parameters correspond to each neural network layer. By allocating different data stream buffer areas in the first data stream storage module, the parameters of each neural network layer are allocated to different data stream buffer areas for reading. The above-mentioned data flow model is based on a non-instruction set model, so there is no instruction idle overhead, which can improve the hardware acceleration efficiency of the neural network. For example, the convolution algorithm of an image is y _i =x _i* c+y _i-1 . The algorithm based on the instruction set is MULT(x _i , c, r), ADD(r, y _i-1 , y _i ), which means that the instruction of r=x _i* c is executed first, and the first result is obtained and then stored, and then The first result is read and used to execute the instruction of y _i =r+y _i-1 . When the MULT is executed, ADD needs to wait and read the execution result of the MULT, which results in the ADD being in an idle state. The data flow model reads x _i and c from the memory to the multiplication calculation core for operation. At the same time, reads _yi-1 from the memory to the addition calculation core, and adds the result of the multiplication operation. Operation, no instruction is idle. The aforementioned data stream engine includes a second data stream storage module and a calculation core corresponding to the neural network operator. The aforementioned second data stream storage module stores parameters corresponding to the calculation core in a partition, and stores the second data stream The parameters in the module are called, so that the second data flow storage module and multiple computing cores form a data flow path, thereby forming a data flow engine.

203. Acquire image information used for automatic driving.

In this step, the above-mentioned image information may be picture data taken in real time by a camera, picture data read from a local folder, or picture data transmitted through a transmission protocol such as TCP.

204. Input the image information into the data stream automatic driving model for processing to obtain an image processing result.

In this step, the above-mentioned image information is obtained through step 203. The above-mentioned image information is input into the data stream automatic driving model for object detection and image segmentation, and abstract features of the object are extracted, and then the object category, probability and Coordinates and other results.

205. Send the image processing result to the driving decision module to form a driving decision.

The above-mentioned image processing results may include the result of object category, probability and coordinates. The above-mentioned driving decision module may be an existing driving decision module, and the above-mentioned driving decision module may be a local driving decision module or a driving in a cloud server. Decision-making side.

In a possible embodiment, the above-mentioned image processing result may be the abstract result obtained by the data flow automatic driving model. The abstract result is sent to the driving decision module set in the cloud server for processing, and the processed result includes the object category , Probability, coordinates and other results, in this way, post-processing and decision-making modules and other hardware can be deployed on the cloud server, thereby reducing the power consumption of the autonomous driving system in the vehicle.

In this embodiment, the corresponding data flow automatic driving model is constructed through the neural network graph and parameters. Since the data flow automatic driving model is a non-instruction set model, there is no instruction idle overhead, which can accelerate the recognition of objects in the image during automatic driving And the detection speed, so that the automatic driving decision module can obtain the image processing results faster, make driving decisions faster, and improve the real-time performance of automatic driving. In addition, the data flow automatic driving model does not require high hardware requirements, which can Reduce the cost and power consumption of autonomous driving.

It should be noted that the data stream-based automatic driving method provided in the embodiments of the present application can be applied to data stream automatic driving equipment, such as computers, servers, mobile phones, vehicle central control, etc., which can perform data stream-based automatic driving. equipment.

Please refer to FIG. 3. FIG. 3 is a schematic flowchart of another data stream-based automatic driving method provided by an embodiment of the present application. As shown in FIG. 3, the method includes the following steps:

301. Acquire a neural network diagram and parameters of an automatic driving model, where the parameters are pre-trained parameters.

302. Configure parallel or serial connections among multiple neural network layers according to the neural network diagram.

303. According to the parameters, allocate data stream memory corresponding to each neural network layer, and the data stream is internally used to store the parameters of the corresponding neural network layer.

304. Form a data flow path between the multiple neural network layers based on the parallel or serial between the multiple neural network layers and allocating data flow memory corresponding to each neural network layer.

305. Form the data flow automatic driving model according to the data flow path.

306. Acquire image information used for automatic driving.

307. Input the image information into the data stream automatic driving model for processing, to obtain an image processing result.

308. Send the image processing result to the driving decision module to form a driving decision.

In this embodiment, the neural network diagram includes parallel or serial relationships between multiple neural network layers, and the multiple neural network layers under the data flow are configured according to the parallel or serial relationship between the multiple neural network layers Parallel or serial between. Parallel or serial under the above data flow is the parallel or serial embodiment of the data flow engine, and the above data flow engine provides computing resources for the corresponding neural network layer. The above-mentioned first data stream storage module may be a cache, a DDR or a high-speed access DDR. In the embodiment of the present application, it is preferably a cache. Specifically, a controllable read-write address generating unit may be provided in the cache. Depending on the input data format and the calculations required in the data path, the address generation unit will generate an adapted address sequence to index the data in the cache. The data stream is stored through the first data stream storage module, and the data is controlled to flow to multiple neural network layers for calculation, so that the data processing can be processed in the data stream model like a pipeline, no instruction is idle, and the image processing is improved. effectiveness.

In this embodiment, the aforementioned parameter data block can be a parameter or a group of parameters. The aforementioned initial memory address can be generated by the address generation unit in the first data stream storage module and assigned to the corresponding Parameter data block. The above-mentioned memory space is used to store the parameter data blocks that the corresponding neural network layer needs to be preloaded. In this way, storing the parameter data block in the memory space can speed up the IO speed of the data, and the input and output between the neural network layers does not need to be IO through the off-chip memory, which improves the processing efficiency of image data.

Read image information from the image memory.

In this embodiment, the above-mentioned image source can be a camera, a local folder, or a picture library transmitted through a transmission protocol such as TCP. Storing the image information obtained from the image source in the image memory can make the reading of the image information more efficient. In addition, it should be noted that the above-mentioned image memory may be the first data stream storage module, or a part of the storage area in the first data stream storage module, or another memory storage module. In this way, the image information is stored in the image memory in advance. After the data flow autopilot model is configured according to the neural network diagram and parameters, the image information can be quickly read from the image memory for processing.

Optionally, the method further includes:

In this embodiment, when the image information fails to be read from the image memory, it means that the data flow automatic driving model fails to obtain the input information, and the image information can be read again in the image memory. The predetermined time for re-reading the image information can be The time in milliseconds. In a possible embodiment, if the number of consecutive reading failures in the image memory exceeds a certain number of times, the image information will be obtained from the image source. If there is a failure to read the image information, it is read in the image memory again. Since the reading speed in the memory is faster, the re-reading time is also very fast.

The above post-processing can be understood as the data characterization of the output result of the neural network. The output result of the neural network is the feature value, which can be understood as an abstract representation of the input picture or data. The above-mentioned post-processing can be achieved through some The calculation method converts the abstract representation into meaningful output, such as the object category and corresponding probability in the classification problem, the object category contained in the picture in the detection problem, the probability and the coordinates, etc.

Optionally, the image processing result includes coordinate data of object features, and the sending the image processing result to a driving decision module to form a driving decision includes:

The above-mentioned object feature categories include vehicles, animals, people, stones, trees, road markings, road signs, etc. The above-mentioned coordinate data may be image-based coordinate data or vehicle environment-based coordinate data. The aforementioned driving decision module may be a local driving decision module or a driving decision module deployed in a cloud server. The above-mentioned driving decision-making is obtained by the driving decision-making module according to the object feature category and coordinate data.

The above-mentioned optional implementation manners can implement the data stream-based automatic driving method in the corresponding embodiment of FIG. 1 and FIG. 2 to achieve the same effect, and will not be repeated here.

For the second aspect, please refer to FIG. 4. FIG. 4 is a schematic structural diagram of a data stream-based automatic driving device provided by an embodiment of the present application. As shown in FIG. 4, the device 400 includes:

The first obtaining module 401 is configured to obtain a neural network diagram and parameters of an automatic driving model, where the parameters are pre-trained parameters;

The configuration module 402 is configured to configure the data flow automatic driving model corresponding to the automatic driving model on the data flow architecture according to the neural network diagram and parameters of the automatic driving model;

The second acquisition module 403 is used to acquire image information for automatic driving;

The processing module 404 is configured to input the image information into the data flow automatic driving model for processing, and obtain an image processing result;

The sending module 405 is configured to send the image processing result to the driving decision module to form a driving decision.

Optionally, as shown in FIG. 5, the configuration module 402 includes:

The configuration unit 4021 is configured to configure parallel or serial between multiple neural network layers according to the neural network diagram;

The allocation unit 4022 is configured to allocate data stream memory corresponding to each neural network layer according to the parameters, and the data stream is internally used to store the parameters of the corresponding neural network layer;

The first path unit 4023 is configured to form a data flow path between the multiple neural network layers based on the parallel or serial between the multiple neural network layers and allocating data flow memory corresponding to each neural network layer;

The second path unit 4024 is configured to form the data flow automatic driving model according to the data flow path.

Optionally, as shown in FIG. 6, the allocating unit 4022 includes:

The address subunit 40221 is used to specify a starting memory address for the parameter data block preloaded in each neural network layer;

The allocation subunit 40222 starts from the designated starting memory address, opens up a memory space the same size as the parameter data block, and allocates it to the parameter data block for loading.

Optionally, as shown in FIG. 7, the first obtaining module 401 includes:

The storage unit 4011 obtains image information from an image source, and stores the obtained image information in the image memory;

The reading unit 4012 is used to read image information from the image memory.

Optionally, as shown in FIG. 8, the apparatus 400 includes:

The third acquisition module 406 is configured to re-read the image information within a predetermined time if the image information from the image memory fails to be read.

Optionally, as shown in FIG. 9, the apparatus 400 further includes:

The post-processing module 407 is configured to perform post-processing on the result obtained after processing the data stream automatic driving model to obtain an image processing result.

Optionally, the sending module 405 is further configured to send the category and coordinate data of the object feature to the driving decision module to form a driving decision.

In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and capable of running on the processor. When the processor executes the computer program The steps in the data stream-based automatic driving method provided in the embodiments of this application are implemented.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium with a computer program stored on the computer-readable storage medium. When the computer program is executed by a processor, the data stream-based Steps in an autonomous driving method. That is, in the specific embodiment of the present invention, when the computer program of the computer-readable storage medium is executed by the processor, the steps of the above-mentioned neural network processing method based on data flow are realized, which can reduce the nonlinearity of the digital circuit control capacitance.

Exemplarily, the computer program in the computer-readable storage medium includes computer program code, and the computer program code may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: capable of carrying the computer program code

Any entity or device, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, Telecommunications signals and software distribution media, etc.

It should be noted that, since the computer program of the computer-readable storage medium is executed by the processor to implement the steps of the aforementioned neural network processing method based on data flow, all the embodiments of the aforementioned neural network processing method based on data flow are applicable to The computer-readable storage medium can achieve the same or similar beneficial effects.

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through a computer program. The program can be stored in a computer readable storage medium. During execution, it may include the procedures of the above-mentioned method embodiments.

In addition, the specific embodiment of the present invention also provides an acceleration hardware board 303 that can interact with the processor 301 for data flow acceleration neural network, which is applied to the algorithm acceleration of the automatic driving perception module.

It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that this application is not limited by the described sequence of actions. Because according to this application, some steps can be performed in other order or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the involved actions and modules are not necessarily required by this application.

In the above-mentioned embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in an embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed device may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or may be Integrate into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be realized in the form of hardware or software program module.

If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer readable memory. Based on this understanding, the technical solution of the present application essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, A number of instructions are included to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other various media that can store program codes.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable memory, and the memory can include: flash disk , Read-only memory (English: Read-Only Memory, abbreviation: ROM), random access device (English: Random Access Memory, abbreviation: RAM), magnetic disk or optical disc, etc.

The above content is a further detailed description of the application in conjunction with specific preferred embodiments, and it cannot be considered that the specific embodiments of the application are limited to these descriptions. For those of ordinary skill in the technical field to which this application belongs, a number of simple deductions or substitutions can be made without departing from the concept of this application, which should be regarded as falling within the protection scope of this application.

Claims

A data stream-based automatic driving method, characterized in that the method includes:

Acquiring a neural network diagram and parameters of the autonomous driving model, where the parameters are pre-trained parameters;

According to the neural network diagram and parameters of the automatic driving model, configure on the data flow architecture to obtain the data flow automatic driving model corresponding to the automatic driving model;

Obtain image information for automatic driving;

Inputting the image information into the data stream automatic driving model for processing to obtain an image processing result;

The image processing result is sent to the driving decision module to form a driving decision.
The method according to claim 1, wherein the configuring on a data flow architecture to obtain the data flow automatic driving model corresponding to the automatic driving model according to the neural network diagram and parameters of the automatic driving model comprises:

According to the neural network diagram, configure parallel or serial between multiple neural network layers;

According to the parameters, allocate data stream memory corresponding to each neural network layer, and the data stream is internally used to store the parameters of the corresponding neural network layer;

Forming a data flow path between the multiple neural network layers based on the parallel or serial between the multiple neural network layers and allocating data flow memory corresponding to each neural network layer;

According to the data flow path, the data flow automatic driving model is formed.
The method according to claim 2, wherein the allocating data flow memory corresponding to each neural network layer according to the parameter comprises:

Specify a starting memory address for each parameter data block preloaded in the neural network layer;

Starting from the designated starting memory address, a memory space with the same size as the parameter data block is opened up and allocated to the parameter data block for loading.
The method according to claim 1, wherein said acquiring image information for automatic driving comprises:

Obtain image information from the image source and store the obtained image information in the image memory;

Read image information from the image memory.
The method according to claim 4, wherein the method further comprises:

If reading the image information from the image memory fails, the reading is performed again within a predetermined time.
The method according to claim 1, wherein after said inputting said image information into said data stream automatic driving model for processing, said method further comprises:

The result obtained after the data flow automatic driving model is processed is post-processed to obtain an image processing result.
The method according to claim 1, wherein the image processing result includes the category of object characteristics and coordinate data, and the sending the image processing result to the driving decision module to form a driving decision includes:

The category and coordinate data of the object features are sent to the driving decision module to form a driving decision.
A data stream-based automatic driving device, characterized in that the device includes:

The first acquisition module is used to acquire the neural network diagram and parameters of the automatic driving model, where the parameters are pre-trained parameters;

The configuration module is used to configure the data flow automatic driving model corresponding to the target automatic driving model on the data flow architecture according to the neural network diagram and parameters of the automatic driving model;

The second acquisition module is used to acquire image information for automatic driving;

A processing module, configured to input the image information into the data stream automatic driving model for processing to obtain an image processing result;

The sending module is used to send the image processing result to the driving decision module to form a driving decision.
An electronic device, characterized by comprising: a memory, a processor, and a computer program stored on the memory and capable of running on the processor. The processor executes the computer program as claimed in claim 1. Steps in the data stream-based automatic driving method described in any one of to 7.
A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the data-based system according to any one of claims 1 to 7 is implemented. Steps in the flow of automated driving methods.