WO2022011722A1 - 数据处理方法、装置及智能车辆 - Google Patents

数据处理方法、装置及智能车辆 Download PDF

Info

Publication number
WO2022011722A1
WO2022011722A1 PCT/CN2020/102868 CN2020102868W WO2022011722A1 WO 2022011722 A1 WO2022011722 A1 WO 2022011722A1 CN 2020102868 W CN2020102868 W CN 2020102868W WO 2022011722 A1 WO2022011722 A1 WO 2022011722A1
Authority
WO
WIPO (PCT)
Prior art keywords
machine learning
learning model
data processing
fusion
neural network
Prior art date
Application number
PCT/CN2020/102868
Other languages
English (en)
French (fr)
Inventor
林孝盈
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN202080004591.3A priority Critical patent/CN112585583B/zh
Priority to PCT/CN2020/102868 priority patent/WO2022011722A1/zh
Priority to EP20945053.5A priority patent/EP4141663A4/en
Publication of WO2022011722A1 publication Critical patent/WO2022011722A1/zh
Priority to US18/154,148 priority patent/US20230169343A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/0098—Details of control systems ensuring comfort, safety or stability not otherwise provided for
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W60/00—Drive control systems specially adapted for autonomous road vehicles
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W2050/0001—Details of the control system
    • B60W2050/0002—Automatic control, details of type of controller or control system architecture
    • B60W2050/0004—In digital systems, e.g. discrete-time systems involving sampling
    • B60W2050/0006—Digital architecture hierarchy
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing

Definitions

  • the present application relates to the technical field of Internet of Vehicles, and in particular, to a data processing method, device and intelligent vehicle.
  • the in-vehicle terminal installed in the vehicle uses the detection data as a machine learning model, and after further analysis and processing, outputs the information that can control the vehicle to adjust the driving parameters to achieve assisted driving.
  • the machine learning model used to analyze the sensor data of the in-vehicle terminal is usually provided by the supplier of the in-vehicle terminal, and the supplier will spend a lot of manpower and material resources in order to obtain the machine learning model. Collect actual road data as samples, and obtain a machine learning model that can be applied to in-vehicle terminals after a lot of training calculations.
  • the Chamber of Commerce ensures the security of machine learning models through a number of technologies.
  • the supplier sets encryption parameters for the machine learning model set in the vehicle terminal, encrypts the machine learning model and then deploys it in the vehicle terminal, but this technology will cause the machine learning model to perform each step. All calculations require encryption and decryption, which increases the computational overhead of the machine learning model and brings about a large processing delay; or, in another technology, the machine learning model runs in a specially set trusted execution environment (trusted execution environment).
  • TEE machine learning model environment
  • this technology is limited by the computing power and space of TEE, each time only part of the ongoing calculation of the machine learning model can be taken out of the TEE to the general environment, and after the calculation is completed, the Returning the calculation results to the TEE increases the number of interactions between the TEE and the general environment during the calculation of the machine learning model, which also brings processing delays.
  • the present application provides a data processing method, device and intelligent vehicle, which are used to solve the technical problem in the prior art that the vehicle-mounted terminal cannot increase the time delay under the condition of ensuring the safety of the model.
  • a first aspect of the present application provides a data processing device.
  • the first neural network model in all the multiple virtual machines combines multiple sensor groups.
  • the output results of the detected data are jointly output to the second neural network model, and the second neural network model finally obtains a fusion output result that can be used to indicate the driving parameter information according to the multiple output results.
  • the machine learning model is divided into different parts and stored in the multiple virtual machines through multiple virtual machines set in the data processing device, thereby increasing the number of users of the data processing device.
  • the difficulty of breaking and obtaining the machine learning model is that the attacker needs to crack all the virtual machines before they can obtain the machine learning model in the data processing device in reverse.
  • the delay introduced by the present application to protect the machine learning model is only caused by the interaction between the virtual machine and the data processing device, and the delay between the virtual machine and the data processing device
  • the interaction technology is relatively mature and the delay is relatively small, so that the application will not increase too much computational overhead compared with the machine learning model encryption technology, and will not increase too much interaction compared with the machine learning model running in the TEE.
  • the machine learning model of the present application reduces the computational delay introduced on the basis of improving security.
  • the data processing apparatus provided by the present application can ensure the security of the machine learning model without excessively increasing the computational overhead and delay of the machine learning model.
  • the data processing apparatus not only protects the first machine learning model used to calculate the sensor by using multiple virtual machines, but also uses the fusion virtual machine to perform processing on the output results of the multiple virtual machines.
  • the fusion obtains the second machine learning model of the fusion output for protection.
  • the data processing apparatus introduces the time delay caused by the interaction between the virtual machine and the fusion virtual machine, the time delay is still small compared with the existing technology, so that the machine learning model The first machine learning model and the second machine learning model divided as a whole are put into different virtual machines, which further ensures the overall security of the machine learning model, and can also reduce the computing time of the machine learning model compared with the prior art. At the same time, higher security performance and certain computing efficiency of the data processing device are realized.
  • the entire machine learning model in the data processing device is further divided into multiple branch neural networks and one fusion neural network, wherein the multiple branch neural networks are independently set in multiple In the virtual machine, it is used to process the detection data of the sensor group connected to the virtual machine to obtain the output result, and the fusion neural network is used to perform feature fusion on the output results calculated by all branch neural networks to obtain the fusion output result.
  • this embodiment is based on the branch network and the fusion network fused later, and the detection data of all sensor groups can be processed separately through the branch neural network, and then the output results can be fused. After all branch neural networks have calculated the output results, more accurate fusion output results can be obtained through feature fusion, instead of relying on the output results corresponding to a certain sensor group.
  • the machine learning model is equivalently divided into multiple branch neural network models and a fusion neural network model, when an attacker wants to steal the machine learning model stored in the data processing device by the supplier, in addition to To break through the operating system of the data processing device itself to obtain the fusion neural network model, it is necessary to further attack multiple virtual machines in the data processing device one by one.
  • the entire machine learning model can be stolen.
  • the number of sensors is large, the number of virtual machines that need to be broken will increase, which will undoubtedly increase the difficulty of stealing the machine learning model, making it more difficult for attackers to steal from the data device.
  • machine learning model thereby improving the security performance of the machine learning model stored in the data processing device.
  • the machine learning model is divided into multiple branch neural networks and a fusion neural network, and each branch neural network only needs to process the detection data of one sensor, so the branch neural network can It is provided by the supplier of the sensor group, and the fusion neural network needs to fuse the output results of multiple sensors to obtain the fusion result. Therefore, the fusion neural network can be provided by the supplier of the data processing device, that is, the supplier of the data processing device in this embodiment. Trained to get.
  • the multiple branched neural networks and one fusion neural network split by the machine learning model can be provided by different suppliers, which reduces the time required for one supplier to calculate all the machine learning models. Difficulty, through the joint calculation of different suppliers to improve the efficiency of the machine learning model. And each supplier can also use the private training data corresponding to its sensors for calculation, and can also ensure the security of each supplier's data.
  • training data may be shared, for example, the training data collected by the supplier of each sensor group All files are uploaded to a storage space such as a server in the Internet for storage.
  • suppliers of all sensor groups can use the same data to train their respective networks, so that although the entire machine learning model is split into different networks, each network trains The data used in the training process is the same, and the data in the entire training process is kept consistent, making the final machine learning model more accurate as a whole.
  • sharing training data by different suppliers can enrich the amount of data for machine learning model training and reduce the human and material investment of each supplier when acquiring training data.
  • the manufacturers of machine learning models corresponding to different sensor groups It can also guarantee the privacy of its machine learning models.
  • a second aspect of the present application provides a data processing method, which can be applied to the data processing apparatus provided in the first aspect of the present application.
  • detection data wherein, each virtual machine includes a first machine learning model; using the detection data of a group of sensors corresponding to each virtual machine as the input of the first machine learning model in the virtual machine, multiple outputs are obtained Result; take multiple output results of the first machine learning model in the multiple virtual machines as the input of the second machine learning model, and obtain a fusion output result; the fusion output result is used to indicate the driving of the intelligent vehicle Parameter information.
  • the second machine learning model is set in the fusion virtual machine; the multiple output results of the first machine learning model in the multiple virtual machines are used as the second Before obtaining the fusion output result from the input of the machine learning model, the method further includes: acquiring multiple output results of the first machine learning model in the plurality of virtual machines through the fusion virtual machine.
  • the first machine learning model includes: a branched neural network model; the second machine learning model includes: a fusion neural network model.
  • the branch neural network model included in each virtual machine is obtained by training from a supplier of a group of sensors corresponding to the virtual machine according to a training data set;
  • the training data set includes A plurality of corresponding relationships, in each of the corresponding relationships, the detection data of a group of sensors of the intelligent vehicle corresponds to one piece of driving parameter information.
  • the fusion neural network model is obtained by training according to the training data set by the supplier of the data processing device.
  • the correspondence included in the training data set is provided by the supplier of the data processing apparatus and the supplier of the multiple groups of sensors.
  • a third aspect of the present application provides an intelligent vehicle, including the data processing apparatus according to any one of the first aspect of the present application.
  • a fourth aspect of the present application provides an electronic device, comprising: a processor and a memory; instructions are stored in the memory, and when the processor executes the instructions, the processor executes any one of the methods described in the second aspect of the present application. method described in item.
  • a fifth aspect of the present application provides a storage medium, where a computer program is stored in the storage medium, and when the computer program runs on a computer, the computer is made to execute the method according to any one of the second aspect of the present application.
  • 1 is a schematic diagram of an application scenario of the application
  • Fig. 2 is a structural schematic diagram of a machine learning model
  • FIG. 3 is a schematic structural diagram of a branch network of early fusion
  • FIG. 4 is a schematic structural diagram of a branch network of late fusion
  • FIG. 5 is a schematic structural diagram of a branch network of mid-term deep fusion
  • FIG. 6 is a schematic diagram of a relevant scenario for the generation and use of a machine learning model
  • FIG. 7 is a schematic structural diagram of an embodiment of a data processing apparatus provided by the present application.
  • FIG. 8 is a schematic structural diagram of a virtual machine provided by the present application.
  • FIG. 9 is a schematic structural diagram of an embodiment of a data processing apparatus provided by the present application.
  • FIG. 10 is a schematic structural diagram of an embodiment of a data processing apparatus provided by the present application.
  • Figure 11 is the detection data of the lidar sensor
  • Figure 12 is the detection data of the front camera
  • Figure 13 is the detection data of the rear camera
  • FIG. 15 is a schematic structural diagram of an embodiment of an electronic device provided by the present application.
  • FIG. 1 is a schematic diagram of an application scenario of the application.
  • an intelligent vehicle 1 has an assisted driving function, and a plurality of sensors 2 are provided on the intelligent vehicle 1, such as imaging sensors such as cameras, A radar sensor (radar), a lidar sensor (lidar), an infrared sensor, etc., these sensors 2 are arranged at different positions on the intelligent vehicle 1, and can be used to detect the environment around the intelligent vehicle 1 to generate detection data.
  • the intelligent vehicle 1 is also provided with an on-board terminal 3, which can be used to analyze the detection data of the multiple sensors 2, and further control the intelligent vehicle 1 according to the analysis results, so as to realize the driver of the intelligent vehicle 1.
  • Various functions for assisting driving such as: automatic braking, automatic parking, blind spot detection, automatic cruise and lane departure warning, etc.
  • FIG. 2 is a schematic structural diagram of a machine learning model, wherein, taking a deep learning (deep learning, DL) model in the machine learning model as an example, because it uses a neural network, it can also be called a deep neural network model,
  • the input layer of the deep learning model includes two input values X1 and X2.
  • the detection data obtained by two different sensors 2 set on the intelligent vehicle 1 can be used as the input of the machine learning model.
  • the middle layer has three neurons h1-h3 , which is used to calculate the vector inner product and nonlinear function of the input value of the input layer, and the probability value of type 1 and type 2 is represented by the two output values of the output layer as the output of the machine learning model.
  • the probability value can be used The selection of a corresponding driving assistance control scheme for the intelligent vehicle 1 is performed subsequently.
  • the machine learning model shown in FIG. 2 is only used to describe its structure, and the number of intermediate layers and the number of nodes included in each layer may be multiple during specific implementation, which is not limited.
  • the machine learning model 31 can adopt the method of fusing different branch networks to The detection data of the sensor 2 is processed, and finally the probability value of the output layer is obtained according to the detection data of the plurality of sensors 2 , which is used for the subsequent selection of the corresponding assisted driving control scheme for the intelligent vehicle 1 .
  • the fusion method adopted by the machine learning model 31 provided in this application includes early fusion, late fusion or deep fusion.
  • Figure 3 is a schematic diagram of the structure of an early fusion branch network, in which the machine learning model takes the detection data of multiple different sensors as input. The fused detection data is processed by multiple intermediate layers, and finally an output result that combines all the input detection data and processed by the intermediate layers is output.
  • Figure 4 is a schematic diagram of the structure of a later fusion branch network, in which the machine learning model will receive the detection data of different sensors as the input of multiple input layers, and process them through the middle layer respectively, and then all the detection data will be processed by the middle layer. The output results of the middle layer are fused through the fusion network, and finally an output result obtained by fusion of all the detection data after being processed by the middle layer is output.
  • Figure 5 is a schematic diagram of the structure of a mid-term deep fusion branch network, wherein the machine learning model takes the detection data of multiple different sensors as input, and processes the data of multiple input layers through multiple cycles of intermediate layer calculation and fusion processing After that, the final output is an output result that combines all the input detection data after multiple intermediate layer processing and fusion.
  • the machine learning model 31 can be provided by the supplier of the vehicle-mounted terminal 3 .
  • FIG. 6 is a schematic diagram of a scene related to the generation and use of the machine learning model.
  • the supplier's staff collects test data by driving smart vehicles on the road, for example, collecting the detection data of sensors under different road conditions as the input of the machine learning model 31, and collecting or manually marking the data under the above-mentioned different road conditions
  • the driving parameter information corresponding to the assisted driving strategy to be performed is used as the output of the machine learning model 31 .
  • step 2 using known detection data as input data and known driving parameter information as output data, in step 2, the middle layer of the machine learning model 31 is learned and trained to obtain a usable machine learning model 31 .
  • step 3 the supplier can store the machine learning model 31 in the on-board terminal 3 of the intelligent vehicle 1, so that the intelligent vehicle storing the machine learning model 31 can use the model to detect the real-time acquisition of the sensor of the intelligent vehicle.
  • the data is used as the input of the machine learning model, and the driving parameter information corresponding to the real-time detection data is output, so as to realize the assisted driving function in the scene shown in Figure 1.
  • the supplier will invest a lot of manpower and material resources to collect a large amount of test data in step 1 to cover more possible driving scenarios and collect more abundant driving scenarios. road conditions, and invest a lot of time and computational cost in step 2, repeatedly analyze and calculate a large amount of test data collected in step 1, and finally obtain a machine learning model 31 that can be deployed in the smart vehicle 1 .
  • the machine learning model 31 stored in the in-vehicle terminal 3 of the smart vehicle 1 gives some illegal attackers the possibility to steal the machine learning model 31 .
  • the attacker may steal the machine learning model 31 stored in it after breaking the vehicle terminal 3, and then copy and copy the stolen machine learning model 31 on other vehicles for profit.
  • Such attackers may attack, in addition to stealing the machine learning model 31 provided by the supplier and causing loss of economic benefits to the supplier, for the machine learning model 31 copied by the attacker to other vehicles due to the lack of subsequent support and updates, etc.
  • the service may also bring great safety hazards to the normal driving of the vehicle. Therefore, in order to ensure that the machine learning model 31 deployed in the in-vehicle terminal 3 will not be easily stolen by an attacker, the supplier may protect the machine learning model 31 by encrypting, setting a security environment, or the like.
  • the supplier sets special encryption parameters for the machine learning model set in the vehicle terminal, encrypts the whole machine learning model before deploying it into the vehicle terminal, so that even if the attacker breaks the
  • the on-board terminal cannot directly determine the specific content of the machine learning model stored in it.
  • the encryption method used for the machine learning model in this first technology is usually a homomorphic encryption algorithm. This algorithm not only has a large computational cost, but also increases the data volume of the machine learning model itself, and also affects the machine learning model. The calculation accuracy of the machine learning model will affect the performance of the assisted driving function of the machine learning model.
  • the machine learning model can be stored in the trusted execution environment (TEE) in the vehicle terminal, so that based on the mature TEE technology, even if the attacker breaks the vehicle terminal, There is also no further access to the machine learning model stored in the TEE.
  • TEE trusted execution environment
  • the machine learning model needs to be in a general execution environment (rich execution environment, REE) outside the TEE when it actually performs calculations. conduct.
  • REE rich execution environment
  • the result is calculated in the general execution environment and returned to the TEE, so that the machine learning model 31 obtains the calculation result.
  • the calculation of some nodes in the machine learning model 31 is sent to the general execution environment each time according to a certain method, such as the order between nodes. , and in the general execution environment, the results of some nodes are calculated and returned to the TEE.
  • the two calculation methods will increase a large amount of data interaction between the TEE and the general execution environment, thereby increasing the time overhead of the machine learning model in the calculation, thereby increasing the The delay of the machine learning model in implementing the assisted driving function is reduced.
  • the present application provides a data processing method, device and intelligent vehicle.
  • a machine learning model set in an on-board terminal of an intelligent vehicle By dividing a machine learning model set in an on-board terminal of an intelligent vehicle into multiple parts and storing them in different multiple virtual machines, different virtual machines can be divided into different parts.
  • the part stored in the computer runs independently and provides protection at the same time, so as to achieve the technical effect of protecting the machine learning model without increasing the computational time overhead of the machine learning model.
  • FIG. 7 is a schematic structural diagram of an embodiment of the data processing device provided by the application.
  • FIG. 7 shows the structure of the intelligent vehicle 1 and the data processing device 3 specifically set in the intelligent vehicle 1 in the scenario shown in FIG. 1 ,
  • the intelligent vehicle 1 shown in FIG. 7 includes: a plurality of sensors, a data processing device 3 and an automatic driving module 4 .
  • each sensor 2 may be disposed on the body of the intelligent vehicle 1 to detect the environmental information around the current location of the intelligent vehicle 1 and generate detection data.
  • sensors include imaging sensors such as cameras, radar sensors, lidar sensors, and infrared sensors.
  • the multiple sensors provided on the smart vehicle 1 may be divided into different sensor groups.
  • the sensors provided on the smart vehicle 1 are divided into N different sensor groups as an example, where N > 1.
  • Each sensor group may include one or more sensors.
  • the embodiments of the present application do not limit the specific basis for dividing different sensor groups.
  • the sensors can be divided into sensor groups according to the format of the collected data, and the cameras, lidars, and other sensors used for collecting image data provided on the intelligent vehicle 1 1.
  • Divide sensors such as GPS, infrared radar, etc. that are used for ranging to obtain distance data into sensor group 2, etc.; or, according to the location of the sensor setting, the sensor set in front of the intelligent vehicle 1 is divided into sensor group 1, and the The sensors provided behind the smart vehicle 1 are divided into sensor groups 2 and the like.
  • the data processing device 3 can be a vehicle-mounted terminal as shown in FIG. 1, or a device in the vehicle-mounted terminal.
  • the data processing device is used to process the detection data input by the N groups of sensors 2, and then output information to the automatic driving module 4, The information can be used to indicate driving parameter information of the intelligent vehicle.
  • the automatic driving module 4 can be used to adjust the driving parameters such as the speed and direction of the intelligent vehicle 1, and realize the assisted driving function of assisting the driver of the intelligent vehicle 1 to drive.
  • the automatic driving module 4 can be an independent device, or can be set in the data. Modules in processing device 3 . Finally, when the fusion result is received, the automatic driving module 4 can control the intelligent vehicle 1 to adjust the driving parameters according to the information sent by the in-vehicle terminal 3 to realize the assisted driving function.
  • a plurality of virtual machines 31 are provided, and each of the virtual machines 31 corresponds to the plurality of sensor groups 2 provided in the intelligent vehicle 1 in one-to-one correspondence.
  • the intelligent vehicle 1 is provided with N sensor groups 2 labeled 1-N in total
  • the data processing device 3 is also set with N virtual machines 31 labeled 1-N in total
  • N The sensors are connected to N virtual machines in one-to-one correspondence.
  • multiple virtual machines 31 may run on the operating system of the data processing apparatus 3, and the types and specific implementation manners of the multiple virtual machines may be the same or different, and the principles and implementations of the virtual machines are not limited in this application.
  • Each virtual machine can be used to implement different functions, and the multiple virtual machines are independent of each other and do not affect each other when implementing different functions.
  • each virtual machine 31 connected to the sensor group 2 can be used to further analyze and process the detection data obtained by the respective connected sensor group 2 , and obtain the corresponding data of the respective sensor 2 . Output the result.
  • each virtual machine 31 includes a first machine learning model, and after acquiring the detection data of a corresponding set of sensors, the virtual machine 31 may specifically use the detection data as the first machine learning model The first machine learning model analyzes the detection data and outputs the detection data, and the obtained detection data is used as the output result.
  • FIG. 8 is a schematic structural diagram of a virtual machine provided by the present application.
  • the example shown in FIG. 8 is based on the example shown in FIG. 7 .
  • the machines 31 are respectively connected with the sensor groups 2 labeled 1-3, and each virtual machine 31 includes a first machine learning model, the first machine learning model may be a branch neural network model, and its structure may be as shown in FIG. 2
  • the basic structure of the machine learning model is based on the basic structure of the machine learning model, after the virtual machine 31 obtains the detection data of the corresponding sensor group 2, the detection data is used as the input of the branch neural network model. After the calculation of the intermediate layer, the probability value output from the output layer can be used as The output result corresponds to the output result of the detection data of the sensor group 2 .
  • the detection data obtained by the sensor group is the image in front of the smart vehicle captured by the camera
  • the image is input as the detection data to the branch neural network model in the virtual machine, and the calculation in the middle layer is performed.
  • the machine learning model outputs through its output layer a probability value representing the presence of an obstacle in the image in front of the smart vehicle, or a probability value representing the location of an obstacle in front of the smart vehicle.
  • the branch neural network model in each virtual machine 31 will output an output result according to the detection data of the corresponding sensor group 2 connected to each other.
  • the output results are fused, and the data processing device 3 provided by this application further includes: a second machine learning model 32, which is used to fuse the output results calculated by the multiple virtual machines 31 to obtain parameters that can be used to indicate the driving of the intelligent vehicle.
  • the information fusion output results, and the fusion output results are sent to the automatic driving module 4.
  • the network structure of early fusion as shown in FIG. 3 is used as an example, and the feature fusion module 32 is set. It can be understood that the feature fusion module can also be selected as shown in FIG. 4 or FIG. 5 .
  • the mid- or late-stage fusion network structure is used as shown in FIG. 4 or FIG. 5 .
  • the second machine learning model 32 may be a fusion neural network model, and its structure may also be the basic structure of the machine learning model shown in FIG. 2 . Then the second machine learning model 32, after receiving the output results of the sensor groups corresponding to the labels 1-3 sent by the three virtual machines 31 also running on the operating system, uses these three output results as the output results of the fusion neural network model. Input, after the calculation of the intermediate layer, the fusion output result output from the output layer can be a probability value, and this probability value can be used to indicate the driving parameter information of the intelligent car.
  • the automatic driving module 4 can directly adjust or indicate the driving parameter information of the intelligent vehicle to the driver, so as to realize the function of the data processing device assisting the driving of the intelligent vehicle.
  • the automatic driving module 4 can directly control the driving direction or speed of the intelligent vehicle; or, the automatic driving module 4 can prompt the driver of the intelligent vehicle according to the result of the fusion output through the visual interface provided by the vehicle terminal and other devices. Driving direction to get driving speed etc.
  • the second machine learning model 32 may run in the operating system of the data processing device 3 , that is, the operating system of the data processing device 3 runs multiple A virtual machine 31 and a second machine learning model 32, each virtual machine and the second machine learning model are logically equal, and the interaction between the virtual machine 31 and the second machine learning model 32 can be understood as the virtual machine 31
  • the interaction with the host where it is located is based on the parallel computing structure used in the interaction between the virtual machine and the host in the virtual machine technology.
  • Each virtual machine can be parallelized without interaction, so there is no additional performance overhead and no increase in computing delay.
  • the delay caused by the interaction with the host is low or even negligible, so that the delay for the virtual machine 31 to send the output result to the module of the second machine learning model 32 is also low.
  • the data stored in the data processing apparatus can be
  • the machine learning model can be equivalently divided into multiple branch neural network models (the first machine learning model) and one fusion neural network model (the second machine learning model) as a whole.
  • a plurality of different virtual machines are set to store a plurality of branch neural network models respectively, so that each branch neural network model is only calculated in the stored virtual machine.
  • the only visible part of each virtual machine is the input detection data and the output output results, which can be understood as dividing the entire machine learning model in the data processing device into different parts (branch neural network models), and the different parts are in run in different virtual machines.
  • the data processing device can obtain the driving parameter information used to indicate the intelligent vehicle according to the detection data of the sensor through the machine learning model, so as to realize the auxiliary driving function, in order to further ensure the machine learning in the data processing device.
  • Model security set up multiple independent virtual machines to store multiple branch neural network models, and use multiple virtual machines to analyze the detection data of corresponding different sensor groups and output the results.
  • the obtained output result is sent to the fusion neural network model, and the fusion neural network model takes the output results of all multiple virtual machines as input, and performs feature fusion to obtain the fusion output result, which can be used to indicate the driving parameter information of the intelligent vehicle.
  • the machine learning model is protected by adding a virtual machine. Since the virtual machine is only a different running environment, it is different from the existing first technology for the machine learning model. Compared with the encryption protection method, it does not increase the calculation amount of the machine learning model itself, nor does it affect the calculation accuracy of the machine learning model; compared with the existing second technology, the machine learning model is stored in the TEE as a whole for protection. In contrast, since the computing power and storage space of the virtual machine is larger than that of the TEE, the process of repeatedly extracting data from the TEE and returning the results does not need to be repeated during the calculation of the machine learning model, and the data interaction during the calculation of the machine learning model is not increased.
  • the machine learning model in the data data processing apparatus can be divided into multiple parts and stored in different virtual machines, so that different virtual machines can be divided into multiple parts. It runs independently and provides protection, and on the basis of multiple virtual machines to protect the machine learning model, it will not bring additional computing overhead, time overhead and data interaction to the calculation of the machine learning model itself, reducing the machine learning model.
  • the delay in realizing the assisted driving function ultimately ensures the safety performance and computing efficiency of the data processing device.
  • FIG. 9 is a schematic structural diagram of an embodiment of a data processing apparatus provided in this application.
  • the data processing apparatus 3 includes a plurality of virtual machines 31, A fusion virtual machine 33 is also included, and a second machine learning model 32 is stored in the fusion virtual machine 33 .
  • the multiple virtual machines 31 and the converged virtual machines 33 can run in the operating system of the data processing device 3 at the same time, logically belong to a parallel relationship with each other, and run independently without affecting each other.
  • the data processing apparatus not only protects the branch neural network model through multiple virtual machines, but also protects the fusion neural network model through the fusion virtual machine.
  • the delay is still small compared with the existing technology, so by placing the branch neural network model and the fusion neural network model divided by the whole machine learning model into different virtual
  • the storage and calculation in the computer further ensure the overall security of the machine learning model. Compared with the existing technology, it can also reduce the delay in the calculation of the machine learning model. At the same time, it realizes higher security performance and certain computing efficiency of the data processing device. .
  • a generation method of the machine learning model is that the entire machine learning model is provided by the supplier of the data processing device; another generation method is virtual.
  • the branch neural network model in the computer can be provided by suppliers of different sensor groups, and the fusion neural network model can be jointly trained by the supplier of the data processing device combined with the branch neural network models provided by the suppliers of different sensor groups.
  • the supplier of the sensor and the supplier of the data processing device can share the training data used to train the machine learning model respectively.
  • the supplier of sensor group 1 can drive a smart vehicle on the road, collect the detection data of all sensors in the sensor group under different road conditions, and collect or manually mark the above
  • the driving parameter information corresponding to the assisted driving strategy that needs to be carried out under different road conditions, such as driving direction and driving speed, etc. the data collected by the supplier of sensor 1 is recorded as training data set 1.
  • the corresponding relationship between the detection data of the sensor group and the driving parameter information is recorded as training data set 2
  • the data collected by the supplier of sensor group 3 is recorded as training data set 3
  • the data collected by the supplier of data processing device is recorded as training data set 4.
  • the above training data sets 1-4 can be stored in the same network server by uploading to the server, for example, when each supplier trains its own machine learning model, the training data collected by all suppliers stored in the server can be used.
  • Data sets for example, the suppliers of sensor groups 1-3 can use the training data sets 1-4 to train the branch neural network models in the virtual machines corresponding to the respective sensor groups, and the suppliers of the data processing devices can also use the training data sets 1-4.
  • -4 Train the fusion neural network model.
  • the present embodiment can enrich the amount of data during training of the machine learning model, and reduce the input of manpower and material resources for each supplier in acquiring training data.
  • the supplier of each sensor group trains the corresponding machine learning model. Even if the machine learning model is stored in multiple virtual machines of a data processing device, due to the independent settings of different virtual machines, the The machine learning models corresponding to other sensor groups cannot be obtained from each other, and the manufacturers of the machine learning models corresponding to each sensor group can also ensure the privacy of their machine learning models.
  • FIG. 10 is a schematic structural diagram of an embodiment of the data processing device provided by the present application.
  • Three sensor groups are provided, and the three sensor groups respectively include a sensor: a lidar, a front camera, and a rear camera.
  • the data processing device 3 is provided with three virtual machines, which are respectively connected to the three sensor groups, and each pass through the virtual machine.
  • the branch neural network model takes the detection data of the sensor group as the output, and obtains the output result.
  • Figure 11 shows the detection data of the lidar sensor.
  • the output result that can be obtained is that there are a total of five vehicles labeled 1-5 in Figure 11 and the positions of each vehicle, and at the same time , Figure 12 is the detection data of the front camera.
  • the output result that can be obtained is that there are a total of five vehicles labeled 1-5 in Figure 12 and the position of each vehicle.
  • Figure 13 is the detection data of the rear camera.
  • the fusion output result is determined only by the output results obtained from the detection data corresponding to any one of the three sensor groups, the accuracy of driving parameters may be affected due to abnormal conditions such as line of sight occlusion and resolution, thereby reducing the assisted driving. Therefore, after obtaining the output results corresponding to the three sensor groups through the three virtual machines, the virtual machines labeled 1-3 output the output results of the above-mentioned Figures 11-13 to the second machine learning model 32 respectively.
  • the second machine learning model 32 combines the three output results to jointly obtain the fusion output result, and finally can indicate the driving parameter information of the intelligent vehicle according to the fusion output result after the fusion of the output results of the three sensor groups, so that the data processing device It can be more perfect, relying on the fusion output results of all sensor groups to determine the driving parameters, and prevent a certain sensor from being blocked and other abnormal conditions, the fusion output results after fusion of the output results of multiple sensors can still indicate more accurate driving. Parameter information.
  • the second machine learning model 32 can obtain a fusion output result according to the obtained output results corresponding to the multiple sensor groups shown in FIGS. 11-13 and send it to the automatic driving module 4, where the fusion output result can be used for intelligent vehicle maintenance
  • the current driving direction and the driving parameter information of deceleration to 40km/h.
  • FIG. 14 is a schematic flowchart of an embodiment of the data processing method provided by the present application, and the method includes:
  • S101 Obtain detection data of a group of sensors corresponding to each virtual machine through a plurality of virtual machines; wherein each virtual machine includes a first machine learning model.
  • the data processing apparatus uses multiple virtual machines set in the data processing apparatus, and divides the machine learning model into different parts and stores them in the multiple virtual machines respectively.
  • Each of the virtual machines corresponds to a plurality of sensor groups set in the smart vehicle, and in S101 , the data processing apparatus as an execution subject first obtains detection data of the corresponding groups of sensors through the plurality of virtual machines.
  • S102 Use detection data of a group of sensors corresponding to each virtual machine as the input of the first machine learning model in the virtual machine, and obtain multiple output results.
  • the detection data can be used as the input of the first machine learning model, and the first machine learning model can analyze the detection data Then output the detection data, and the obtained detection data is used as the output result.
  • S103 Use multiple output results of the first machine learning model in the multiple virtual machines as the input of the second machine learning model to obtain a fusion output result; the fusion output result is used to indicate the driving parameter information of the intelligent vehicle.
  • the data processing provided in this application The device further includes: a second machine learning model, then in S103, the data processing device as the execution subject can be used to fuse the output results calculated by the multiple virtual machines to obtain a data that can be used to indicate the driving parameter information of the intelligent vehicle. Fuse the output results and send the fusion output results to the autonomous driving module.
  • the machine learning model is divided into different parts and stored in the multiple virtual machines, thereby increasing the number of data processing apparatuses. It is difficult to be hacked and obtain the machine learning model in it. As an attacker, after all virtual machines are cracked, the machine learning model in the data processing device can be obtained in reverse. Therefore, to a certain extent, the machine learning model in the data processing device is realized.
  • the protection of the machine learning model improves the security of the machine learning model; at the same time, the data processing method provided in this application can ensure the security of the machine learning model without excessively increasing the computational overhead and delay of the machine learning model.
  • the second machine learning model may be set in the fusion virtual machine; then before S103 as shown in FIG. 14 , the method further includes: acquiring multiple pieces of the first machine learning model in the plurality of virtual machines through the fusion virtual machine Output the result.
  • this embodiment protects the second machine learning model used for fusing the output results of multiple virtual machines to obtain the fusion output result by fusing the virtual machines.
  • the data processing method provided in this embodiment introduces the time delay caused by the interaction between the virtual machine and the converged virtual machine, the time delay is still small compared with the existing technology, so that the machine learning The first machine learning model and the second machine learning model divided by the whole model are put into different virtual machines, which further ensures the safety of the machine learning model as a whole, and can also reduce the calculation of the machine learning model compared with the prior art. time delay, while achieving higher security performance and certain computing efficiency of the data processing device.
  • the first machine learning model includes: a branch neural network model; the second machine learning model includes: a fusion neural network model.
  • the branch neural network model included in each virtual machine is obtained by training from a supplier of a group of sensors corresponding to the virtual machine according to a training data set; the training data set includes multiple correspondences, and in each In the corresponding relationship, the detection data of a group of sensors of the intelligent vehicle corresponds to a driving parameter information.
  • the fusion neural network model is obtained by training according to the training data set by the supplier of the data processing device.
  • the correspondence included in the training data set is provided by the supplier of the data processing device and the supplier of the plurality of sets of sensors.
  • the data processing method provided in this embodiment can be executed by the corresponding data processing apparatus provided in the foregoing embodiments of the present application, and the implementation manner and principle are the same, and are not repeated here.
  • FIG. 15 is a schematic structural diagram of an embodiment of the electronic device provided by the present application, as shown in FIG. 15 ,
  • the electronic device 1500 may include a communication interface 1510 and a processor 1520 .
  • the electronic device 1500 may further include a memory 1530 .
  • the memory 1530 may be disposed inside the electronic device or outside the electronic device.
  • the processor 1520 sends data through the communication interface 1510 and is used to implement any method performed by the data processing apparatus in FIG. 14 .
  • the steps of the processing flow can be implemented by the hardware integrated logic circuit in the processor 1520 or the instructions in the form of software to complete the method executed by the first local server in FIGS. 4-10 .
  • the program codes executed by the processor 1520 for implementing the above method may be stored in the memory 1530 .
  • the memory 1530 is connected to the processor 1520, such as a coupling connection and the like.
  • Some features of the embodiments of the present application may be implemented/supported by the processor 1520 executing program instructions or software codes in the memory 1530 .
  • the software components loaded on the memory 1530 may be functionally or logically summarized, for example, the virtual machine shown in FIG. 7 , the second machine learning model, the fusion virtual machine shown in FIG. 9 , and the like.
  • Any communication interface involved in the embodiments of this application may be a circuit, a bus, a transceiver, or any other device that can be used for information interaction.
  • the communication interface 1510 in the electronic device 1500 for example, the other device may be a device connected to the electronic device, for example, the other device may be a sensor or an automatic driving module and the like.
  • the processors involved in the embodiments of the present application may be general-purpose processors, digital signal processors, application-specific integrated circuits, field programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and may implement or The methods, steps and logic block diagrams disclosed in the embodiments of this application are executed.
  • a general purpose processor may be a microprocessor or any conventional processor or the like.
  • the steps of the methods disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.
  • the coupling in the embodiments of the present application is an indirect coupling or communication connection between devices, modules or modules, which may be in electrical, mechanical or other forms, and is used for information exchange between devices, modules or modules.
  • the processor may cooperate with the memory.
  • the memory can be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), etc., or a volatile memory (volatile memory), such as random access memory (random-state drive, SSD), etc. access memory, RAM).
  • Memory is, but is not limited to, any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
  • connection medium among the above-mentioned communication interface, processor, and memory is not limited in the embodiments of the present application.
  • the memory, the processor and the communication interface can be connected by a bus.
  • the bus can be divided into an address bus, a data bus, a control bus, and the like.
  • the present application also provides a storage medium, where a computer program is stored in the storage medium, and when the computer program runs on a computer, the computer is made to execute any one of the executions of the data processing apparatus in the embodiments of the present application. Methods.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Automation & Control Theory (AREA)
  • Human Computer Interaction (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Traffic Control Systems (AREA)

Abstract

本申请提供一种数据处理方法、装置及智能车辆，通过在数据处理装置内设置多个虚拟机，并将机器学习模型中每个传感器组对应的第一神经网络模型分别放置在一个虚拟机中，使得机器学习模型在对对应的传感器组的检测数据进行计算时，都是在一个独立的虚拟机中进行，随后所有多个虚拟机中的第一神经网络模型将多个传感器组的检测数据的输出结果共同输出到第二神经网络模型中，由第二神经网络模型根据多个输出结果最终得到可用于指示行驶参数信息的融合输出结果。本申请提供的数据处理装置，能够在保证机器学习模型安全的同时还不会过多增加机器学习模型的计算开销和时延。

Description

数据处理方法、装置及智能车辆 技术领域
本申请涉及车联网技术领域，尤其涉及一种数据处理方法、装置及智能车辆。
背景技术
随着电子技术以及汽车技术的不断的发展，越来越多的车辆都具有了辅助驾驶功能，为了实现辅助驾驶功能，通常在车辆上安装摄像头、光达、雷达等各种类型的传感器，通过这些传感器采集车辆周围的环境信息的检测数据后，由设置在车辆内的车载终端对将些检测数据作为机器学习模型，由进行进一步分析与处理后输出能够控制车辆调整行驶参数实现辅助驾驶的信息。同时，用于车载终端分析传感器数据的机器学习模型通常由车载终端的供应商提供，而供应商为了得到机器学习模型，会耗费大量的人力物力，例如由实验人员驾驶车辆在道路上行驶、并采集实际道路数据作为样本，经过大量训练计算后才得到可应用于车载终端中的机器学习模型。
现有技术中，供应商为了保证车载终端中机器学习模型的安全，防止机器学习模型被盗用而造成经济损失，以及机器学习模型被仿造后给其他车辆带来严重的安全隐患，车载终端的供应商会通过一些技术保证机器学习模型的安全。例如，在一种技术中，供应商为设置在车载终端内的机器学习模型设置加密参数，将机器学习模型加密后再部署到车载终端内，但是这种技术会造成机器学习模型在进行每一步计算都需要进行加密和解密，从而提高了机器学习模型的计算开销，带来较大的处理时延；或者，在另一种技术中，机器学习模型在专门设置的可信执行环境(trusted execution environment，简称：TEE)内存储，但是这种技术受到TEE算力与空间的限制，每次只能将机器学习模型正在进行的部分计算从TEE中取出到一般环境中进行，在计算完成之后还要将计算结果返回到TEE中去，从而增加了机器学习模型在计算时TEE与一般环境之间的交互次数，同样带来了处理时延。
因此，如何保证车载终端内机器学习模型的安全的同时，还不会增加机器学习模型的计算开销和时延，是本领域需要解决的技术问题。
发明内容
本申请提供一种数据处理方法、装置及智能车辆，用于解决现有技术中车载终端不能在保证模型安全的情况下，还不会增加时延的技术问题。
本申请第一方面提供一种数据处理装置，通过在数据处理装置内设置多个虚拟机，并将机器学习模型中每个传感器组对应的第一神经网络模型分别放置在一个虚拟机中，这样整个机器学习模型在对对应的传感器组的检测数据进行计算时，都是在一个独立的虚拟机中进行，并且互不影响，所有多个虚拟机中的第一神经网络模型将多个传感器组的检测数据的输出结果共同输出到第二神经网络模型中，由第二神经网络模型根据多个输出结果最终得到可用于指示行驶参数信息的融合输出结果。
可见，本实施例提供的数据处理装置内，通过数据处理装置内设置的多个虚拟机，将 机器学习模型拆分为不同的部分分别存入多个虚拟机中，从而增加了数据处理装置被攻破、获得其中机器学习模型的难度，作为攻击者需要将所有虚拟机破解之后，才可以反向得到数据处理装置内的机器学习模型，因此一定程度上实现了对数据处理装置中机器学习模型的保护，提高了机器学习模型的安全性；同时，本申请为了保护机器学习模型所引入的时延仅仅是由虚拟机和数据处理装置之间的交互带来，而虚拟机和数据处理装置之间交互的技术较为成熟，时延相对较小，使得本申请与机器学习模型加密技术相比不会增加过多的计算开销、与机器学习模型在TEE中运行相比不会增加过多的交互，从而使得本申请的机器学习模型在提升安全的基础上，减少了所引入的计算时延。综上，本申请提供的数据处理装置，能够在保证机器学习模型安全的同时还不会过多增加机器学习模型的计算开销和时延。
在本申请第一方面一实施例中，数据处理装置除了通过多个虚拟机对传感器进行计算的第一机器学习模型进行保护，还通过融合虚拟机对用于对多个虚拟机的输出结果进行融合得到融合输出结果的第二机器学习模型进行保护。
使得本实施例中提供的数据处理装置虽然引入了虚拟机和融合虚拟机之间交互所带来的时延，但是其时延与现有的技术相比依然较小，从而通过将机器学习模型整体所划分的第一机器学习模型和第二机器学习模型都各自放入不同的虚拟机中，进一步在保证了机器学习模型整体的安全，与现有技术相比也能够减少机器学习模型计算时的时延，同时实现了数据处理装置更高安全性能与一定的计算效率。
在本申请第一方面一实施例中，进一步将数据处理装置中的整个机器学习模型拆分为多个分支神经网络以及一个融合神经网络，其中，多个分支神经网络相互独立地设置在多个虚拟机中，用于对虚拟机所连接的传感器组的检测数据进行处理得到输出结果，融合神经网络用于将所有分支神经网络计算得到的输出结果进行特征融合，得到融合输出结果。
因此，本实施例基于分支网络以及后期融合的融合网络，能够将所有传感器组的检测数据通过分支神经网络分别进行处理后，将输出结果进行融合，由于结合了多个传感器组的检测数据，从而在所有分支神经网络计算出输出结果后，通过特征融合能够得到更加准确的融合输出结果，而不是依赖于某一个传感器组对应的输出结果。同时，机器学习模型在相当于整体被拆分成了多个分支神经网络模型以及一个融合神经网络模型之后，使得攻击者想要窃取供应商存储在数据处理装置内的机器学习模型时，除了要攻破数据处理装置本身的操作系统获取融合神经网络模型，还需要进一步对数据处理装置中的多个虚拟机进行一一攻击，并只有在攻破所有虚拟机获取所有虚拟机中的分支神经网络模型之后，才可以实现盗取整个机器学习模型，当传感器数量较多时，需要攻破的虚拟机数量也将增加，无疑增加了盗取机器学习模型的难度，使得攻击者更不容易从数据装置中盗取机器学习模型，从而提高了数据处理装置中存储的机器学习模型的安全性能。
在本申请第一方面一实施例中，基于将机器学习模型拆分为多个分支神经网络以及一个融合神经网络，而每个分支神经网络仅需要处理一个传感器的检测数据，因此分支神经网络可以由传感器组的供应商提供，而融合神经网络需要将多个传感器的输出结果融合得到融合结果，因此融合神经网络可以由数据处理装置的供应商，也就是本实施例中数据处理装置的供应商训练得到。
综上，本实施例提供的数据处理装置中，机器学习模型所拆分的多个分支神经网络以 及一个融合神经网络分别可以由不同的供应商提供，减少了一家供应商计算所有机器学习模型的难度，通过不同供应商的共同计算提高得到机器学习模型的效率。并且每家供应商还可以使用其传感器对应的私有训练数据进行计算，还能够保证每家供应商数据的安全。
在本申请第一方面一实施例中，在传感器的供应商和数据处理装置的供应商训练各自的网络时，可以对训练数据进行共享，例如，每个传感器组的供应商所采集的训练数据都上传至互联网中的服务器等存储空间进行存储。
综上，根据本实施例提供的训练数据，使得所有传感器组的供应商可以使用相同的数据对各自的网络进行训练，使得整个机器学习模型虽然被拆分为不同的网络，但是每个网络训练时采用的数据相同，最终保持整个训练过程的数据一致，使得最终得到的机器学习模型整体更加准确。并且由不同的供应商共享训练数据，能够丰富机器学习模型训练时的数据的数量，减少每个供应商在获取训练数据时的人力物力投入，同时，不同传感器组对应的机器学习模型的厂商之间也能够保证其机器学习模型的私密性。
本申请第二方面提供一种数据处理方法，可应用于如本申请第一方面提供的数据处理装置执行，所述方法包括：通过多个虚拟机，获取每个虚拟机对应的一组传感器的检测数据；其中，每个虚拟机包括一个第一机器学习模型；以所述每个虚拟机对应的一组传感器的检测数据为所述虚拟机中第一机器学习模型的输入，得到多个输出结果；以所述多个虚拟机中的第一机器学习模型的多个输出结果，作为第二机器学习模型的输入，得到融合输出结果；所述融合输出结果用于指示所述智能车辆的行驶参数信息。
在本申请第二方面一实施例中，所述第二机器学习模型设置在融合虚拟机内；所述以所述多个虚拟机中的第一机器学习模型的多个输出结果，作为第二机器学习模型的输入，得到融合输出结果之前，所述方法还包括：通过所述融合虚拟机获取所述多个虚拟机中的第一机器学习模型的多个输出结果。
在本申请第二方面一实施例中，所述第一机器学习模型包括：分支神经网络模型；所述第二机器学习模型包括：融合神经网络模型。
在本申请第二方面一实施例中，每个所述虚拟机包括的分支神经网络模型，由所述虚拟机对应的一组传感器的供应商根据训练数据集训练得到；所述训练数据集包括多个对应关系，在每个所述对应关系中，所述智能车辆的一组传感器的检测数据与一个行驶参数信息相对应。
在本申请第二方面一实施例中，所述融合神经网络模型，由所述数据处理装置的供应商根据所述训练数据集训练得到。
在本申请第二方面一实施例中，所述训练数据集中包括的对应关系，由所述数据处理装置的供应商和所述多组传感器的供应商提供。
本申请第三方面提供一种智能车辆，包括如本申请第一方面任一项的数据处理装置。
本申请第四方面提供一种电子设备，包括：处理器和存储器；所述存储器中存储有指令，当所述处理器运行所述指令时，所述处理器执行如本申请第二方面任一项所述的方法。
本申请第五方面提供一种存储介质，所述存储介质存储有计算机程序，当所述计算机程序在计算机上运行时，使所述计算机执行如本申请第二方面任一项所述的方法。
附图说明
图1为本申请应用场景的示意图；
图2为一种机器学习模型的结构示意图；
图3为一种早期融合的分支网络的结构示意图；
图4为一种后期融合的分支网络的结构示意图；
图5为一种中期深度融合的分支网络的结构示意图；
图6为机器学习模型生成及使用的相关场景示意图；
图7为本申请提供的数据处理装置一实施例的结构示意图；
图8为本申请提供的虚拟机的结构示意图；
图9为本申请提供的数据处理装置一实施例的结构示意图；
图10为本申请提供的数据处理装置一实施例的结构示意图；
图11为光达传感器的检测数据；
图12为前摄像机的检测数据；
图13为后置摄像机的检测数据；
图14为本申请提供的数据处理方法一实施例的流程示意图；
图15为本申请提供的电子设备一实施例的结构示意图。
具体实施方式
在正式介绍本申请之前，先结合附图，对本申请所应用的场景以及所存在的问题进行说明。
图1为本申请应用场景的示意图，具体地，在如图1所示的场景中，智能车辆1具有辅助驾驶功能，在智能车辆1上设置有多个传感器2，例如：摄像头等成像传感器、雷达传感器(radar)、光达传感器(lidar)以及红外传感器等，这些传感器2设置在智能车辆1上的不同位置处，可用于对智能车辆1周围的环境进行检测，生成检测数据。同时，智能车辆1上还设置有车载终端3，可用于通过多个传感器2的检测数据进行分析，并进一步根据分析结果，对智能车辆1进行相应的控制，从而实现对智能车辆1的驾驶员进行辅助驾驶的多种功能，例如：自动刹车、自动泊车、盲点检测、自动巡航以及车道偏移警示等。
同时，随着人工智能(artificial intelligence，AI)技术的不断发展，机器学习(machine learning，ML)模型被广泛应用于如图1所示的辅助驾驶场景中，智能车辆1中的车载终端3可以通过机器学习模型31对传感器2的检测数据进行处理。例如，图2为一种机器学习模型的结构示意图，其中，以机器学习模型中的深度学习(deep learning，DL)模型作为示例，由于其使用神经网络，又可被称为深度神经网络模型，该深度学习模型的输入层包括两个输入值X1和X2，可以将智能车辆1上设置两个不同的传感器2所获取的检测数据作为机器学习模型的输入，中间层有三个神经元h1-h3，用于对输入层的输入值进行向量内积以及非线性函数等计算后，通过输出层的两个输出值表示类型1和类型2的概率值作为机器学习模型的输出，所述概率值可用于后续对对智能车辆1进行相应的辅助驾驶控制方案的选择。需要说明的是，如图2所示的机器学习模型仅用于描述其结构，在具体实现时中间层的数量、以及每一层所包括的节点的数量可以是多个，不做限定。
更为具体地，由于智能车辆1设置了多个传感器2，每个传感器所输入的数据格式以 及数据处理方式可能不同，机器学习模型31可以采用将不同的分支网络进行融合的方式，对多个传感器2的检测数据进行处理，并最终根据多个传感器2的检测数据得到输出层的概率值，用于后续对对智能车辆1进行相应的辅助驾驶控制方案的选择。本申请提供的机器学习模型31所采用的融合方式包括：早期融合、后期融合或者深度融合。
例如，图3为一种早期融合的分支网络的结构示意图，其中，机器学习模型将多个不同传感器的检测数据作为输入，首先通过融合网络对输入的所有多个传感器的检测数据进行融合，再将融合后的检测数据经过多个中间层的处理，并最终输出一个结合了所有输入的检测数据融合后经过中间层处理的输出结果。图4为一种后期融合的分支网络的结构示意图，其中，机器学习模型将接收到不同传感器的检测数据分别作为多个输入层的输入，并分别通过中间层进行处理，随后将所有检测数据的中间层输出结果通过融合网络进行融合，最终输出一个所有检测数据经过中间层处理后融合得到的输出结果。图5为一种中期深度融合的分支网络的结构示意图，其中，机器学习模型将多个不同传感器的检测数据作为输入，并将多个输入层的数据经过多次循环的中间层计算以及融合处理后，最终输出一个结合了所有输入的检测数据经过多次中间层处理以及融合后的输出结果。
在如图1所示的场景中，机器学习模型31可以由车载终端3的供应商提供，例如，图6为机器学习模型生成及使用的相关场景示意图，其中，在步骤①中，为了得到更加真实有效的训练数据，供应商的工作人员通过驾驶智能车辆在道路上行驶采集测试数据，例如采集不同路况下传感器的检测数据作为机器学习模型31的输入，并采集或者人工标注在上述不同路况下对应需要进行的辅助驾驶策略的行驶参数信息作为机器学习模型31的输出。随后，通过已知的检测数据作为输入数据和已知的行驶参数信息作为输出数据，在步骤②中，对机器学习模型31的中间层进行学习训练，得到可以使用的机器学习模型31。最终，在步骤③中，供应商可以将机器学习模型31存入智能车辆1的车载终端3中，使得存储有机器学习模型31的智能车辆可以使用该模型，将智能车辆的传感器实时获取的检测数据作为机器学习模型的输入，输出实时的检测数据对应的行驶参数信息，从而实现如图1所示场景中的辅助驾驶功能。
供应商为了提高机器学习模型31在提供辅助驾驶功能时的性能，会投入大量的人力物力，在步骤①采集大量的测试数据，以覆盖更多可能的驾驶场景，采集更加丰富的驾驶场景下的路况，并在步骤②投入大量的时间和计算成本，对步骤①采集的大量测试数据进行反复的分析与计算，最终才能得到能够部署在智能车辆1内的机器学习模型31。然而，在如图6所示的场景中，存储在智能车辆1的车载终端3内的机器学习模型31给了一些非法的攻击者盗取机器学习模型31的可能，通过车载终端3提供的各种软件系统以及硬件接口，攻击者有可能在攻破车载终端3之后，盗取其中存储的机器学习模型31，之后将盗取的机器学习模型31仿造、并复制在其他车辆上进行牟利。这种攻击者可能进行的攻击，除了盗用供应商提供的机器学习模型31给供应商造成经济利益上的损失，对于攻击者复制到其他车辆上的机器学习模型31由于缺少后续的支持与更新等服务，还可能会对车辆的正常行驶带来极大的安全隐患。因此，为了保证部署在车载终端3内的机器学习模型31不会轻易被攻击者盗用，供应商可以采用加密、设置安全环境等方式，对机器学习模型31进保护。
例如，在第一种现有技术中，供应商为设置在车载终端内的机器学习模型设置专门的 加密参数，将机器学习模型整体进行加密后再部署到车载终端内，使得攻击者即使攻破了车载终端，也无法直接确定其中存储的机器学习模型的具体内容。但是，在这第一种技术中对机器学习模型所采取的加密方式通常为同态加密算法，这种算法不仅计算开销大，增大了机器学习模型本身的数据量，还影响到机器学习模型的计算精度，进而影响机器学习模型实现辅助驾驶功能的性能。
在第二种现有技术中，机器学习模型可以存储在车载终端内的可信执行环境(trusted execution environment，简称：TEE)内，从而基于成熟的TEE技术，使得攻击者即使攻破了车载终端，也无法进一步获取TEE内存储的机器学习模型。然而，由于机器学习模型的数据量较大，而同时TEE算力与存储空间又非常有限，使得机器学习模型在实际进行计算时需要在TEE之外的一般执行环境(rich execution environment，REE)中进行。在第一种计算方式中，例如，对于如图2所示机器学习模型31在进行图2中箭头对应的每一次计算时，分别将每次箭头对应的计算需要使用的数据发送到一般执行环境中，并在一般执行环境中计算出结果后返回TEE内，使得机器学习模型31得到计算结果。在第二种计算方式中，同样对于图2所示的机器学习模型，按照一定的方式，例如节点之间的前后顺序，每次将机器学习模型31中部分节点的计算发送到一般执行环境中，并在一般执行环境中计算出部分节点的结果后返回TEE内。但是，在上述第二种现有技术中，两种计算方式都会给TEE和一般执行环境之间，增加了大量的数据交互，从而增加了机器学习模型在进行计算时的时间开销，进而增大了机器学习模型在实现辅助驾驶功能时的时延。
综上，在现有的对车载终端内机器学习模型进行保护的方法中，在实现保护的基础之上，都会给机器学习模型本身的计算带来额外的时间开销，进而给机器学习模型实现辅助驾驶功能时时延，最终影响车载终端实现辅助驾驶功能的性能。因此，如何在对车载终端内机器学习模型进行保护的同时，还不增加机器学习模型本身的计算时间开销，是本领域需要解决的技术问题。
本申请提供一种数据处理方法、装置及智能车辆，通过将智能车辆的车载终端内设置的机器学习模型，拆分为多个部分，并分别存入不同的多个虚拟机中，使得不同虚拟机中存储的部分独立运行，并同时提供保护，以实现对机器学习模型进行保护的同时还不会增加机器学习模型的计算时间开销的技术效果。下面以具体地实施例对本申请的技术方案进行详细说明。下面这几个具体的实施例可以相互结合，对于相同或相似的概念或过程可能在某些实施例不再赘述。
图7为本申请提供的数据处理装置一实施例的结构示意图，如图7示出了如图1所示的场景中智能车辆1、以及智能车辆1内具体设置的数据处理装置3的结构，如图7所示的智能车辆1包括：多个传感器，数据处理装置3和自动驾驶模块4。
其中，每个传感器2可以设置在智能车辆1的车体上，用于对智能车辆1当前所在位置周围的环境信息进行检测，并生成检测数据。例如，传感器包括：摄像头等成像传感器、雷达传感器、光达传感器以及红外传感器等。本申请实施例中，将智能车辆1上设置的多个传感器可以划分为不同的传感器组，例如图7中以将智能车辆1上设置的传感器划分为N个不同的传感器组作为示例，N＞1，每个传感器组中可以包括一个或多个传感器。本申请实施例对划分不同传感器组的具体依据不做限定，例如，可以将传感器按照采集数据的格式，将智能车辆1上设置的摄像头、激光雷达等用于采集图像数据的传感器划分为传感 器组1，将GPS、红外雷达等用于测距得到距离数据的传感器划分为传感器组2等；或者，还可以按照传感器设置的位置，将设置在智能车辆1前方的传感器划分为传感器组1，将设置在智能车辆1后方的传感器划分为传感器组2等。
数据处理装置3可以是如图1所示的车载终端、也可以是车载终端中的装置，数据处理装置用于对N组传感器2输入的检测数据进行处理后，向自动驾驶模块4输出信息，所述信息可用于指示智能车辆的行驶参数信息。
自动驾驶模块4可用于对智能车辆1的速度、方向等行驶参数进行调整，实现辅助智能车辆1驾驶员驾驶的辅助驾驶功能，所述自动驾驶模块4可以是独立的装置、或者可以设置在数据处理装置3中的模块。最终当接收到融合结果，自动驾驶模块4可以根据接收到车载终端3发送的信息，控制智能车辆1调整行驶参数，实现辅助驾驶功能。
具体地，本实施例提供的数据处理装置3中，设置有多个虚拟机31，每个所述虚拟机31与智能车辆1中设置的多个传感器组2一一对应。例如在如图7所示的示例中，智能车辆1上设置有标号为1-N共N个传感器组2，则数据处理装置3中也设置标号为1-N共N个虚拟机31，N个传感器与N个虚拟机一一对应连接。可选地，多个虚拟机31可以运行在数据处理装置3的操作系统上，这多个虚拟机的类型、具体实现方式可以相同或者不同，本申请对虚拟机的原理及实现不做限定。每个虚拟机可用于实现不同的功能，并且这多个虚拟机之间是相互独立的，在实现不同功能时也互不影响。
对于图7所示实施例中，每个与传感器组2连接的虚拟机31，均可用于对各自所连接的传感器组2所获取的检测数据进行进一步分析与处理，并得到各自传感器2对应的输出结果。在一种具体的实现方式中，每个虚拟机31中包括一个第一机器学习模型，则虚拟机31在获取对应的一组传感器的检测数据后，具体可以将检测数据作为第一机器学习模型的输入，由第一机器学习模型对检测数据进行分析后输出检测数据，得到的检测数据作为输出结果。
例如，图8为本申请提供的虚拟机的结构示意图，如图8所示的示例在如图7所示的基础上，以数据传输装置中设置三个虚拟机31为例，这三个虚拟机31分别与标号为1-3的传感器组2连接，每个虚拟机31中包括一个第一机器学习模型，该第一机器学习模型可以是分支神经网络模型，其结构可以是图2所示的机器学习模型的基本结构，则虚拟机31获取对应的传感器组2的检测数据后，将检测数据作为分支神经网络模型的输入，经过中间层的计算后，从输出层输出的概率值可作为输出结果，对应于传感器组2的检测数据的输出结果。例如，当传感器组是摄像头时，该传感器组所获取的检测数据是摄像头拍摄的智能车辆前方的图像，则将图像作为检测数据输入到虚拟机中的分支神经网络模型后，经过中间层的计算，机器学习模型通过其输出层输出用于表示该智能车辆前方图像中存在障碍物的概率值，或者用于表示智能车辆前障碍物位置的概率值。
同时，由于智能车辆上设置了多个传感器组2，每个虚拟机31中的分支神经网络模型会根据各自连接对应的传感器组2的检测数据输出一个输出结果，为了对不同传感器组2对应的输出结果进行融合处理，本申请提供的数据处理装置3中还包括：第二机器学习模型32，用于将多个虚拟机31所计算得到的输出结果进行融合，得到可用于指示智能车辆行驶参数信息的融合输出结果，并将融合输出结果发送至自动驾驶模块4。可选地，本申请实施例中以如图3所示的早期融合的网络结构作为示例，设置特征融合模块32，可以理 解的是，特征融合模块也可选用如图4或者如图5所示的中期或者后期融合网络结构。
例如，同样在如图8所示的示例中，所述第二机器学习模型32可以是融合神经网络模型，其结构也可以是如图2所示的机器学习模型的基本结构。则第二机器学习模型32在接收到同样运行在操作系统上的三个虚拟机31发送的对应标号为1-3的传感器组的输出结果之后，将这三个输出结果作为融合神经网络模型的输入，经过中间层的计算后，从输出层输出的融合输出结果可以为一个概率值，这个概率值可用于指示智能这辆的行驶参数信息。示例性地，当传感器组1、传感器组2和传感器3组分别根据车辆前方的环境信息生成检测数据，每个传感器组的检测数据经过对应的虚拟机31的特征分析后，得到三个不同的输出结果，这三个输出结果虽然都用于指示前方其他车辆的位置，但是其可能存在表达方式、准确程度之间的不同，因此通过融合神经网络模型32，可以将三个输出结果进行融合，最终得到的融合输出结果是根据三个输出结果共同得到的，因此可以较为准确地指示智能车辆的行驶参数信息。
最终，自动驾驶模块4接收到数据处理装置3发送的融合输出结果后，可以直接调整或者向驾驶员指示智能车辆的行驶参数信息，实现数据处理装置辅助智能车辆驾驶的功能。例如，自动驾驶模块4可以直接控制智能车辆的行驶方向或行驶速度等；或者，自动驾驶模块4可以通过车载终端等设备提供的可视化界面，向智能车辆的驾驶员提示根据融合输出结果所指示的行驶方向获取行驶速度等。
可选地，在如图7和图8所示的实施例中，第二机器学习模型32可以运行在数据处理装置3的操作系统内，即，数据处理装置3的操作系统内同时运行了多个虚拟机31和第二机器学习模型32，每个虚拟机与第二机器学习模型在逻辑上是平等的关系，虚拟机31与第二机器学习模型32之间的交互可以理解为虚拟机31与其所在的主机之间的交互，基于虚拟机技术中虚拟机与主机之间交互时使用平行计算结构，各虚拟机可以并行且无需交互，故没有额外效能开销，不增加计算时延，虚拟机与主机之间交互带来的时延较低甚至可以忽略不计，使得虚拟机31将输出结果发送至第二机器学习模型32模块的时延也较低。
当如图7或8所提供的数据处理装置中设置的第一机器学习模型和第二机器学习模型这样的结构，应用与如图6所示的实施例中，能够数据处理装置中所存储的机器学习模型可以相当于整体被拆分成了多个分支神经网络模型(第一机器学习模型)以及一个融合神经网络模型(第二机器学习模型)。并且，在数据处理装置中，还通过设置不同的多个虚拟机分别存储多个分支神经网络模型，使得每个分支神经网络模型只在所存储的虚拟机内进行计算。而每个虚拟机对外可见的部分只有输入的检测数据和输出的输出结果，可以理解为将数据处理装置中整个机器学习模型拆分为不同的部分(分支神经网络模型)，不同的部分分别在不同的虚拟机中运行。
可见，本申请提供的数据处理装置，能够在通过机器学习模型根据传感器的检测数据得到用于指示智能车辆的行驶参数信息，从而实现辅助驾驶功能的基础上，为了进一步保障数据处理装置中机器学习模型的安全，设置了多个相互独立的虚拟机存储多个分支神经网络模型，并通过多个虚拟机分别对对应的不同传感器组的检测数据进行特征分析并输出结果，随后多个虚拟机将得到的输出结果发送给融合神经网络模型，由融合神经网络模型将所有多个虚拟机的输出结果作为输入，并进行特征融合之后得到融合输出结果，可用于指示智能车辆的行驶参数信息。
因此，当攻击者想要窃取供应商存储在数据处理装置内的机器学习模型时，除了要攻破数据处理装置本身的操作系统获取到的融合神经网络模型，还需要进一步对数据处理装置中的多个虚拟机进行一一攻击，并只有在攻破所有虚拟机获取所有虚拟机中的分支神经网络模型之后，才可以实现盗取数据处理装置中的整个机器学习模型，当传感器组数量较多时，数据处理装置中设置的虚拟机的数量较多，攻击者需要攻破的虚拟机数量也将增加，无疑增加了盗取整个机器学习模型的难度，使得攻击者更不容易从数据装置中盗取机器学习模型，从而提高了数据处理装置中存储的机器学习模型的安全性能。
同时，本实施例提供的数据处理装置中，通过增加虚拟机的方式，对机器学习模型提供保护，由于虚拟机只是运行环境的不同，因此与现有的第一种技术中对机器学习模型进行加密保护的方式相比，没有增加机器学习模型本身的计算量，也不会影响到机器学习模型的计算精度；与现有的第二种技术中将机器学习模型整体存在TEE中进行保护的方式相比，由于虚拟机的算力和存储空间与TEE相比更大，不需要在机器学习模型计算时反复从TEE中提取数据并返回结果的过程，没有增加机器学习模型计算时的数据交互。
因此，本实施例提供的数据处理装置，能够通过将数据数据处理装置中的机器学习模型拆分成多个部分，并分别存入不同的多个虚拟机中，使得不同的多个虚拟机分别独立运行并提供保护，并且在多个虚拟机实现对机器学习模型进行保护的基础上，不会给机器学习模型本身的计算带来额外的计算开销、时间开销以及数据交互，减少了机器学习模型实现辅助驾驶功能时的时延，最终同时保证了数据处理装置的安全性能与计算效率。
进一步地，在如图7或图8所示的实施例提供的数据处理装置的基础上，第二机器学习模型也可以设置在操作系统的虚拟机中，用于存储第二机器学习模型的虚拟机可记为融合虚拟机，例如，图9为本申请提供的数据处理装置一实施例的结构示意图，在如图9所示的实施例中，数据处理装置3除了包括多个虚拟机31，还包括融合虚拟机33，在融合虚拟机33内存储有第二机器学习模型32。此时，多个虚拟机31和融合虚拟机33可以同时运行在数据处理装置3的操作系统中，相互之间逻辑上属于并列关系，并且各自独立运行，互不影响。
因此，本实施里提供的数据处理装置，除了通过多个虚拟机对分支神经网络模型进行保护，还通过融合虚拟机对融合神经网络模型进行保护，虽然引入了多个虚拟机和融合虚拟机之间交互所带来的时延，但是其时延与现有的技术相比依然较小，从而通过将机器学习模型整体所划分的分支神经网络模型和融合神经网络模型都各自放入不同的虚拟机中存储以及计算，进一步在保证了机器学习模型整体的安全，与现有技术相比也能够减少机器学习模型计算时的时延，同时实现了数据处理装置更高安全性能与一定的计算效率。
可选地，在上述如图7-9所示的数据处理装置中，机器学习模型的一种生成方式是机器学习模型的整体都由数据处理装置的供应商提供；另一种生成方式是虚拟机中的分支神经网络模型可以由不同的传感器组的供应商提供，而融合神经网络模型可以由数据处理装置的供应商结合不同传感器组的供应商提供的分支神经网络模型共同训练得到。此时，传感器的供应商以及数据处理装置的供应商可以共享用于各自训练机器学习模型的训练数据。
例如，以图8所示的机器学习模型为例，传感器组1的供应商可以驾驶智能车辆在道路上行驶，采集不同路况下传感器组中所有传感器的的检测数据，并采集或者人工标注在 上述不同路况下对应需要进行的辅助驾驶策略对应的行驶参数信息，例如行驶方向和行驶速度等，将传感器1的供应商采集的数据记为训练数据集1，在训练数据集1中，包括了在不同时刻，传感器组的检测数据，以及行驶参数信息之间的对应关系。同样地，传感器组2的供应商采集的数据记为训练数据集2，传感器组3的供应商采集的数据记为训练数据集3，将数据处理装置的供应商采集的数据记为训练数据集4。上述训练数据集1-4可以通过例如上传服务器的方式，存储在同一个网络服务器中，后续每个供应商对各自的机器学习模型进行训练时，可以使用服务器中存储的所有供应商采集的训练数据集，例如，传感器组1-3的供应商均可以通过训练数据集1-4训练各自传感器组对应的虚拟机中的分支神经网络模型，数据处理装置的供应商也可以根据训练数据集1-4训练融合神经网络模型。
因此，本实施例能够丰富机器学习模型训练时的数据的数量，减少每个供应商在获取训练数据时的人力物力投入。同时，每个传感器组的供应商各自训练对应的机器学习模型，机器学习模型即使存储在一个数据处理装置的多个虚拟机中，由于不同虚拟机之间的相互独立设置，不同虚拟机之间也不能互相获取其他传感器组对应的机器学习模型，对每个传感器组对应的机器学习模型的厂商之间也能够保证其机器学习模型的私密性。
示例性地，下面结合具体的实施例，对本申请提供的数据处理装置实现辅助驾驶功能的过程进行说明，图10为本申请提供的数据处理装置一实施例的结构示意图，其中，智能车辆1上设置有三个传感器组，这三个传感器组分别包括一个传感器：光达、前摄像机和后摄像机，数据处理装置3中设置有三个虚拟机，分别连接三个传感器组，并各自通过虚拟机中的分支神经网络模型，对传感器组的检测数据作为输出，得到输出结果。例如，图11为光达传感器的检测数据，经过标号为1的虚拟机的特征分析，能够得到的输出结果为图11中存在标号为①-⑤的共五辆车以及各车辆的位置，同时，图12为前摄像机的检测数据，经过标号为2的虚拟机的特征分析，能够得到的输出结果为图12中存在标号为①-⑤的共五辆车以及各车辆的位置。图13为后置摄像机的检测数据，经过标号为3的虚拟机的特征分析，能够得到输出结果为图13也就是智能车辆后方没有其他车辆。如果仅仅依靠这三个传感器组中任一个传感器组对应的检测数据所得到的输出结果确定融合输出结果，可能会因为视线遮挡、分辨率等异常情况，影响行驶参数的准确性，进而降低辅助驾驶的性能，因此，在通过三个虚拟机分别得到三个传感器组对应的输出结果后，标号为1-3的虚拟机分别将上述图11-图13的输出结果输出到第二机器学习模型32中，由第二机器学习模型32综合三个输出结果，共同得到融合输出结果，并最终能够根据三个传感器组的输出结果融合后的融合输出结果指示智能车辆的行驶参数信息，使得数据处理装置能够更加完善地，依赖所有传感器组的融合输出结果来确定行驶参数，防止某一个传感器出现被遮挡等异常情况时，依然能够由多个传感器的输出结果融合后的融合输出结果指示更加准确的行驶参数信息。例如，第二机器学习模型32可以根据获取的如图11-图13的多个传感器组对应的输出结果，得到融合输出结果并发送至自动驾驶模块4，所述融合输出结果可用于智能车辆保持当前行驶方向、减速到40km/h的行驶参数信息。
本申请还提供一种数据处理方法，可应用于本申请前述实施例中的数据处理装置执行，具体地，图14为本申请提供的数据处理方法一实施例的流程示意图，该方法包括：
S101：通过多个虚拟机，获取每个虚拟机对应的一组传感器的检测数据；其中，每个 虚拟机包括一个第一机器学习模型。
具体地，本申请提供的数据处理装置通过数据处理装置内设置的多个虚拟机，并将机器学习模型拆分为不同的部分分别存入多个虚拟机中。每个所述虚拟机与智能车辆中设置的多个传感器组一一对应，则在S101中，作为执行主体的数据处理装置首先通过多个虚拟机，分别获取对应的多组传感器的检测数据。
S102：以每个虚拟机对应的一组传感器的检测数据为虚拟机中第一机器学习模型的输入，得到多个输出结果。
随后，多个虚拟机在获取对应的多组传感器的检测数据后，对于其中的每一个虚拟机，可以将检测数据作为第一机器学习模型的输入，由第一机器学习模型对检测数据进行分析后输出检测数据，得到的检测数据作为输出结果。
S103：以多个虚拟机中的第一机器学习模型的多个输出结果，作为第二机器学习模型的输入，得到融合输出结果；融合输出结果用于指示智能车辆的行驶参数信息。
具体地，由于每个虚拟机中的第一机器学习模型会根据各自连接对应的传感器组的检测数据输出一个输出结果，为了对不同传感器组对应的输出结果进行融合处理，本申请提供的数据处理装置中还包括：第二机器学习模型，则在S103中，作为执行主体的数据处理装置，可用于将多个虚拟机所计算得到的输出结果进行融合，得到可用于指示智能车辆行驶参数信息的融合输出结果，并将融合输出结果发送至自动驾驶模块。
综上，本实施例提供的数据处理方法，由于在数据处理装置内设置了多个虚拟机，将机器学习模型拆分为不同的部分分别存入多个虚拟机中，从而增加了数据处理装置被攻破、获得其中机器学习模型的难度，作为攻击者需要将所有虚拟机破解之后，才可以反向得到数据处理装置内的机器学习模型，因此一定程度上实现了对数据处理装置中机器学习模型的保护，提高了机器学习模型的安全性；同时，本申请提供的数据处理方法，能够在保证机器学习模型安全的同时还不会过多增加机器学习模型的计算开销和时延。
可选地，第二机器学习模型可以设置在融合虚拟机内；则在如图14所示的S103之前，还包括：通过融合虚拟机获取多个虚拟机中的第一机器学习模型的多个输出结果。
因此，本实施例通过融合虚拟机对用于对多个虚拟机的输出结果进行融合得到融合输出结果的第二机器学习模型进行保护。使得本实施例中提供的数据处理方法，虽然引入了虚拟机和融合虚拟机之间交互所带来的时延，但是其时延与现有的技术相比依然较小，从而通过将机器学习模型整体所划分的第一机器学习模型和第二机器学习模型都各自放入不同的虚拟机中，进一步在保证了机器学习模型整体的安全，与现有技术相比也能够减少机器学习模型计算时的时延，同时实现了数据处理装置更高安全性能与一定的计算效率。
可选地，第一机器学习模型包括：分支神经网络模型；第二机器学习模型包括：融合神经网络模型。
可选地，在本实施例中，每个虚拟机包括的分支神经网络模型，由虚拟机对应的一组传感器的供应商根据训练数据集训练得到；训练数据集包括多个对应关系，在每个对应关系中，智能车辆的一组传感器的检测数据与一个行驶参数信息相对应。
可选地，融合神经网络模型，由数据处理装置的供应商根据训练数据集训练得到。训练数据集中包括的对应关系，由数据处理装置的供应商和多组传感器的供应商提供。
本实施例提供的数据处理方法，可以由本申请前述实施例中提供的对应的数据处理装 置执行，其实现方式与原理相同，不再赘述。
此外，本申请实施例还提供另外一种可应用于实现本申请提供的数据处理装置的电子设备的结构，图15为本申请提供的电子设备一实施例的结构示意图，如图15所示，电子设备1500中可以包括通信接口1510、处理器1520。可选的，电子设备1500中还可以包括存储器1530。其中，存储器1530可以设置于电子设备内部，还可以设置于电子设备外部。
示例性地，上述图14中由数据处理装置所执行的动作均可以由处理器1520实现。处理器1520通过通信接口1510发送数据，并用于实现图14中数据处理装置执行的任一方法。在实现过程中，处理流程的各步骤可以通过处理器1520中的硬件的集成逻辑电路或者软件形式的指令完成图4-图10中所述第一本地服务器所执行的方法。为了简洁，在此不再赘述。处理器1520用于实现上述方法所执行的程序代码可以存储在存储器1530中。存储器1530和处理器1520连接，如耦合连接等。
本申请实施例的一些特征可以由处理器1520执行存储器1530中的程序指令或者软件代码来完成/支持。存储器1530上在加载的软件组件可以从功能或者逻辑上进行概括，例如，图7所示的虚拟机、第二机器学习模型，以及图9所示的融合虚拟机等。
本申请实施例中涉及到的任一通信接口可以是电路、总线、收发器或者其它任意可以用于进行信息交互的装置。比如电子设备1500中的通信接口1510，示例性地，该其它装置可以是与该电子设备相连的设备，比如，其他装置可以是传感器或者自动驾驶模块等。
本申请实施例中涉及的处理器可以是通用处理器、数字信号处理器、专用集成电路、现场可编程门阵列或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件，可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件处理器执行完成，或者用处理器中的硬件及软件模块组合执行完成。
本申请实施例中的耦合是装置、模块或模块之间的间接耦合或通信连接，可以是电性，机械或其它的形式，用于装置、模块或模块之间的信息交互。
处理器可能和存储器协同操作。存储器可以是非易失性存储器，比如硬盘(hard disk drive，HDD)或固态硬盘(solid-state drive，SSD)等，还可以是易失性存储器(volatile memory)，例如随机存取存储器(random-access memory，RAM)。存储器是能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质，但不限于此。
本申请实施例中不限定上述通信接口、处理器以及存储器之间的具体连接介质。比如存储器、处理器以及通信接口之间可以通过总线连接。所述总线可以分为地址总线、数据总线、控制总线等。
本申请还提供一种存储介质，所述存储介质存储有计算机程序，当所述计算机程序在计算机上运行时，使所述计算机执行如本申请实施例中数据处理装置执行的任一项所述的方法。
在本申请实施例中，“至少一个”是指一个或者多个，“多个”是指两个或两个 以上。可以理解的是，在本申请实施例中涉及的各种数字编号仅为描述方便进行的区分，并不用来限制本申请实施例的范围。可以理解的是，在本申请的实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
最后应说明的是：以上各实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述各实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的范围。

Claims (15)

  1. 一种数据处理装置，设置在智能车辆上，所述智能车辆设置多组用于检测周围环境信息的传感器；其特征在于，所述数据处理装置，包括：
    多个虚拟机，每个虚拟机包括一个第一机器学习模型；每个所述虚拟机用于获取所述多组传感器中对应的一组传感器的检测数据，并以所述检测数据为所述第一机器学习模型的输入，得到输出结果；
    第二机器学习模型，用于获取所述多个虚拟机的输出结果，并以所述多个虚拟机的输出结果为所述第二机器学习模型的输入，得到融合输出结果，所述融合输出结果用于指示所述智能车辆的行驶参数信息。
  2. 根据权利要求1所述的装置，其特征在于，还包括：
    融合虚拟机，所述多个虚拟机分别连接所述融合虚拟机，所述融合虚拟机包括所述第二机器学习模型。
  3. 根据权利要求2所述的装置，其特征在于，
    所述第一机器学习模型包括：分支神经网络模型；
    所述第二机器学习模型包括：融合神经网络模型。
  4. 根据权利要求3所述的装置，其特征在于，
    每个所述虚拟机包括的分支神经网络模型，由所述虚拟机对应的一组传感器的供应商根据训练数据集训练得到；
    所述训练数据集包括多个对应关系，在每个所述对应关系中，所述智能车辆的一组传感器的检测数据与一个行驶参数信息相对应。
  5. 根据权利要求4所述的装置，其特征在于，
    所述融合神经网络模型，由所述数据处理装置的供应商根据所述训练数据集训练得到。
  6. 根据权利要求4或5所述的装置，其特征在于，
    所述训练数据集中包括的对应关系，由所述数据处理装置的供应商和所述多组传感器的供应商提供。
  7. 一种数据处理方法，应用于设置在智能车辆上的数据处理装置，所述智能车辆设置有多组用于检测周围环境信息的传感器，其特征在于，包括：
    通过多个虚拟机，获取每个虚拟机对应的一组传感器的检测数据；其中，每个虚拟机包括一个第一机器学习模型；
    以所述每个虚拟机对应的一组传感器的检测数据为所述虚拟机中第一机器学习模型的输入，得到多个输出结果；
    以所述多个虚拟机中的第一机器学习模型的多个输出结果，作为第二机器学习模型的输入，得到融合输出结果；所述融合输出结果用于指示所述智能车辆的行驶参数信息。
  8. 根据权利要求7所述的方法，其特征在于，
    所述第二机器学习模型设置在融合虚拟机内；
    所述以所述多个虚拟机中的第一机器学习模型的多个输出结果，作为第二机器学习模型的输入，得到融合输出结果之前，所述方法还包括：
    通过所述融合虚拟机获取所述多个虚拟机中的第一机器学习模型的多个输出结果。
  9. 根据权利要求8所述的方法，其特征在于，
    所述第一机器学习模型包括：分支神经网络模型；
    所述第二机器学习模型包括：融合神经网络模型。
  10. 根据权利要求9所述的方法，其特征在于，
    每个所述虚拟机包括的分支神经网络模型，由所述虚拟机对应的一组传感器的供应商根据训练数据集训练得到；
    所述训练数据集包括多个对应关系，在每个所述对应关系中，所述智能车辆的一组传感器的检测数据与一个行驶参数信息相对应。
  11. 根据权利要求10所述的方法，其特征在于，
    所述融合神经网络模型，由所述数据处理装置的供应商根据所述训练数据集训练得到。
  12. 根据权利要求10或11所述的方法，其特征在于，
    所述训练数据集中包括的对应关系，由所述数据处理装置的供应商和所述多组传感器的供应商提供。
  13. 一种智能车辆，其特征在于，包括：如本申请权利要求1-6任一项所述的数据处理装置。
  14. 一种电子设备，包括：处理器和存储器；所述存储器中存储有指令，当所述处理器运行所述指令时，所述处理器执行如权利要求7-12任一项所述的方法。
  15. 一种存储介质，其特征在于，所述存储介质存储有计算机程序，当所述计算机程序在计算机上运行时，使所述计算机执行如权利要求7-12任一项所述的方法。
PCT/CN2020/102868 2020-07-17 2020-07-17 数据处理方法、装置及智能车辆 WO2022011722A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
CN202080004591.3A CN112585583B (zh) 2020-07-17 2020-07-17 数据处理方法、装置及智能车辆
PCT/CN2020/102868 WO2022011722A1 (zh) 2020-07-17 2020-07-17 数据处理方法、装置及智能车辆
EP20945053.5A EP4141663A4 (en) 2020-07-17 2020-07-17 DATA PROCESSING METHOD AND APPARATUS, AND INTELLIGENT VEHICLE
US18/154,148 US20230169343A1 (en) 2020-07-17 2023-01-13 Data processing method and apparatus, and intelligent vehicle

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/102868 WO2022011722A1 (zh) 2020-07-17 2020-07-17 数据处理方法、装置及智能车辆

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/154,148 Continuation US20230169343A1 (en) 2020-07-17 2023-01-13 Data processing method and apparatus, and intelligent vehicle

Publications (1)

Publication Number Publication Date
WO2022011722A1 true WO2022011722A1 (zh) 2022-01-20

Family

ID=75145413

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/102868 WO2022011722A1 (zh) 2020-07-17 2020-07-17 数据处理方法、装置及智能车辆

Country Status (4)

Country Link
US (1) US20230169343A1 (zh)
EP (1) EP4141663A4 (zh)
CN (1) CN112585583B (zh)
WO (1) WO2022011722A1 (zh)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170372201A1 (en) * 2016-06-22 2017-12-28 Massachusetts Institute Of Technology Secure Training of Multi-Party Deep Neural Network
CN108108766A (zh) * 2017-12-28 2018-06-01 东南大学 基于多传感器数据融合的驾驶行为识别方法及系统
CN108196535A (zh) * 2017-12-12 2018-06-22 清华大学苏州汽车研究院（吴江） 基于增强学习和多传感器融合的自动驾驶系统
US10503174B1 (en) * 2019-01-31 2019-12-10 StradVision, Inc. Method and device for optimized resource allocation in autonomous driving on the basis of reinforcement learning using data from lidar, radar, and camera sensor
CN110619229A (zh) * 2019-09-25 2019-12-27 南开大学 基于神经网络拆分的数据加密传输方法及系统
CN110674941A (zh) * 2019-09-25 2020-01-10 南开大学 基于神经网络的数据加密传输方法及系统

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107783818B (zh) * 2017-10-13 2021-12-24 北京百度网讯科技有限公司 深度学习任务处理方法、装置、设备及存储介质
EP3707572B1 (en) * 2017-11-10 2023-08-23 Nvidia Corporation Systems and methods for safe and reliable autonomous vehicles
US10831519B2 (en) * 2017-11-22 2020-11-10 Amazon Technologies, Inc. Packaging and deploying algorithms for flexible machine learning
US10324467B1 (en) * 2017-12-29 2019-06-18 Apex Artificial Intelligence Industries, Inc. Controller systems and methods of limiting the operation of neural networks to be within one or more conditions
US10546393B2 (en) * 2017-12-30 2020-01-28 Intel Corporation Compression in machine learning and deep learning processing
US11315013B2 (en) * 2018-04-23 2022-04-26 EMC IP Holding Company LLC Implementing parameter server in networking infrastructure for high-performance computing
CN108594819A (zh) * 2018-05-03 2018-09-28 济南浪潮高新科技投资发展有限公司 自动驾驶车载计算资源管理系统和方法
CN111079475A (zh) * 2018-10-19 2020-04-28 上海商汤智能科技有限公司 驾驶状态检测方法和装置、驾驶员监控系统、车辆
CN111105006B (zh) * 2018-10-26 2023-08-04 杭州海康威视数字技术股份有限公司 一种深度学习网络训练系统及方法
CN111399853B (zh) * 2020-02-20 2023-06-06 四川新网银行股份有限公司 机器学习模型与自定义算子的模板化部署方法
CN111382868B (zh) * 2020-02-21 2024-06-18 华为技术有限公司 神经网络结构搜索方法和神经网络结构搜索装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170372201A1 (en) * 2016-06-22 2017-12-28 Massachusetts Institute Of Technology Secure Training of Multi-Party Deep Neural Network
CN108196535A (zh) * 2017-12-12 2018-06-22 清华大学苏州汽车研究院（吴江） 基于增强学习和多传感器融合的自动驾驶系统
CN108108766A (zh) * 2017-12-28 2018-06-01 东南大学 基于多传感器数据融合的驾驶行为识别方法及系统
US10503174B1 (en) * 2019-01-31 2019-12-10 StradVision, Inc. Method and device for optimized resource allocation in autonomous driving on the basis of reinforcement learning using data from lidar, radar, and camera sensor
CN110619229A (zh) * 2019-09-25 2019-12-27 南开大学 基于神经网络拆分的数据加密传输方法及系统
CN110674941A (zh) * 2019-09-25 2020-01-10 南开大学 基于神经网络的数据加密传输方法及系统

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
LU, FENG ET AL.: "Obstacle Detection Method based on Fusion Information", JOURNAL OF COMPUTER APPLICATIONS, vol. 37, no. S2, 20 December 2017 (2017-12-20), pages 115 - 119, XP055887569 *
See also references of EP4141663A4 *

Also Published As

Publication number Publication date
CN112585583B (zh) 2021-12-03
EP4141663A1 (en) 2023-03-01
EP4141663A4 (en) 2023-05-31
CN112585583A (zh) 2021-03-30
US20230169343A1 (en) 2023-06-01

Similar Documents

Publication Publication Date Title
Liu et al. Edge computing for autonomous driving: Opportunities and challenges
Sun et al. Towards robust {LiDAR-based} perception in autonomous driving: General black-box adversarial sensor attack and countermeasures
CN109215170B (zh) 行车数据的存储方法、装置和设备
WO2019237068A1 (en) Protecting vehicle buses from cyber-attacks
CN111127931A (zh) 一种智能网联汽车的车路云协同方法、装置及系统
Jha et al. Ml-driven malware that targets av safety
CN109895786A (zh) 一种自动驾驶辅助方法、装置、设备及可读存储介质
Narayanan et al. Security in smart cyber-physical systems: a case study on smart grids and smart cars
WO2019165614A1 (zh) 一种车辆周边环境分析方法、装置及车载设备
WO2024131678A1 (zh) 一种融合真实道路场景的智能驾驶在环测试方法、环境及存储介质
Girdhar et al. Post-accident cyberattack event analysis for connected and automated vehicles
Ghosh et al. An integrated approach of threat analysis for autonomous vehicles perception system
Yao et al. Accident responsibility identification model for Internet of Vehicles based on lightweight blockchain
Hoque et al. Avguard: A forensic investigation framework for autonomous vehicles
CN112600839A (zh) 基于车联网平台构建安全威胁关联视图的方法及装置
WO2022011722A1 (zh) 数据处理方法、装置及智能车辆
Finkenzeller et al. Simutack-an attack simulation framework for connected and autonomous vehicles
US20230185919A1 (en) System and process using homomorphic encryption to secure neural network parameters for a motor vehicle
Wang et al. Systematic assessment of cyber-physical security of lane keeping control system for autonomous vehicles
Sharma et al. Towards an AI-based after-collision forensic analysis protocol for autonomous vehicles
US20240073037A1 (en) Internal certificate authority for electronic control unit
US20230367872A1 (en) Threat detection for a processing system of a motor vehicle
Gruyer et al. PerSEE: A central sensors fusion electronic control unit for the development of perception-based ADAS
Zhou et al. Runtime Stealthy Perception Attacks against DNN-based Adaptive Cruise Control Systems
Akca et al. Privacy, Security and Legal Aspects of Autonomous Vehicles

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20945053

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020945053

Country of ref document: EP

Effective date: 20221121

NENP Non-entry into the national phase

Ref country code: DE