CN117612129B

CN117612129B - Vehicle dynamic perception method, system and dynamic perception model training method

Info

Publication number: CN117612129B
Application number: CN202410096311.6A
Authority: CN
Inventors: 李仁刚; 张腾飞; 赵雅倩; 李茹杨; 张恒; 邓琪
Original assignee: Suzhou Metabrain Intelligent Technology Co Ltd
Current assignee: Suzhou Metabrain Intelligent Technology Co Ltd
Priority date: 2024-01-24
Filing date: 2024-01-24
Publication date: 2024-04-16
Anticipated expiration: 2044-01-24
Also published as: CN117612129A

Abstract

The invention relates to the technical field of automatic driving, in particular to a vehicle dynamic perception method, a system and a dynamic perception model training method. The vehicle dynamic sensing method comprises the following steps: collecting dynamic perception associated information of a current vehicle, wherein the dynamic perception associated information comprises the following steps: vehicle external environment information, vehicle running state information and processor load information of a sensing system; based on the dynamic perception associated information and a preset dynamic perception parameter set matching rule, a dynamic perception parameter set is obtained, the dynamic perception associated information corresponds to the dynamic perception parameter set, and the dynamic perception parameter set comprises: a perception range parameter, a perception resolution parameter, a depth estimation range parameter, and a depth estimation point interval parameter; based on the dynamic perception parameter set, the data preprocessing strategy and the dynamic perception model of the perception system are adjusted to obtain a perception result. The method can well reduce the calculation load and ensure the instantaneity and the robustness of the perception system and the unmanned system.

Description

Vehicle dynamic perception method, system and dynamic perception model training method

Technical Field

The invention relates to the technical field of automatic driving, in particular to a vehicle dynamic perception method, a system and a dynamic perception model training method.

Background

Vehicle driving systems, such as unmanned driving systems, have extremely high requirements for real-time performance. The unmanned system is generally composed of a perception system, a decision system and an execution system, wherein each event of each system in the three systems is prevented from occupying computing resources outside a specified range, and response is completed as quickly as possible within a specified time, so that the real-time performance and the robustness of the whole unmanned system are improved.

However, the existing sensing system generally has a calculation burden, that is, the sensing system cannot accurately and real-timely adjust the sensing range, so that a large amount of unnecessary calculation burden is generated, and the real-time performance and the robustness of the sensing system and even the unmanned system are affected.

Disclosure of Invention

The invention provides a vehicle dynamic perception method, a system and a dynamic perception model training method, which are used for solving the problems that a perception system of a vehicle in the prior art cannot accurately and real-timely adjust a perception range, so that a large amount of unnecessary calculation burden is generated, and the real-time performance and the robustness of the perception system and even an unmanned system are influenced.

The invention provides a vehicle dynamic sensing method, which comprises the following steps:

collecting dynamic perception associated information of a current vehicle, wherein the dynamic perception associated information comprises the following steps: vehicle external environment information, vehicle running state information and processor load information of a sensing system;

Based on the dynamic perception associated information and a preset dynamic perception parameter set matching rule, a dynamic perception parameter set is obtained, the dynamic perception associated information corresponds to the dynamic perception parameter set, and the dynamic perception parameter set comprises: a perception range parameter, a perception resolution parameter, a depth estimation range parameter, and a depth estimation point interval parameter;

based on the dynamic sensing parameter set, a data preprocessing strategy of the sensing system and a dynamic sensing model are adjusted to obtain a sensing result, wherein the data preprocessing strategy is used for preprocessing sensing data acquired by the sensing system, and the dynamic sensing model is used for performing sensing task processing based on the preprocessed sensing data to obtain the sensing result.

According to the vehicle dynamic sensing method provided by the invention, based on the dynamic sensing parameter set, the step of adjusting the data preprocessing strategy of the sensing system comprises the following steps:

Based on the sensing range parameters in the dynamic sensing parameter set, adjusting a point cloud screening strategy in the data preprocessing strategy to be: clearing original point cloud data, which are located outside a sensing range corresponding to the sensing range parameter, in the sensing data to obtain target point cloud data; the sensing data includes: the original point cloud data are data acquired by a laser radar or a millimeter wave radar of the current vehicle.

Based on the perceived resolution parameters in the dynamic perceived parameter set, adjusting an image resolution adjustment strategy in the data preprocessing strategy to: the original resolution of the original image data in the sensing data is adjusted to the sensing resolution parameter, and target image data is obtained; the original image data are data acquired by the camera device of the current vehicle.

According to the vehicle dynamic sensing method provided by the invention, based on the dynamic sensing parameter set, the step of adjusting the dynamic sensing model of the sensing system comprises the following steps:

Based on the dynamic perception parameter set, adjusting an image feature view conversion network and a point cloud feature extraction network of the dynamic perception model; the image feature view conversion network is used for converting the two-dimensional image features output by the image feature extraction network of the dynamic perception model into three-dimensional camera view cone point cloud features; the input data of the image feature extraction network is target image data, and the target image data is preprocessed original image data; the point cloud feature extraction network is used for carrying out feature extraction on target point cloud data to obtain target point cloud features, the target point cloud data are preprocessed original point cloud data, the original image data and the original point cloud data are both the sensing data, the original image data are data acquired by a camera device of a current vehicle, and the original point cloud data are data acquired by a laser radar or a millimeter wave radar of the current vehicle.

According to the vehicle dynamic sensing method provided by the invention, based on the dynamic sensing parameter set, the step of adjusting the image feature view conversion network of the dynamic sensing model comprises the following steps:

adjusting a depth estimation sub-network of the image feature view conversion network based on the depth estimation range parameter and the depth estimation point interval parameter in the dynamic perception parameter set;

Adjusting a camera view cone point cloud feature extraction sub-network of the image feature view conversion network based on the perceived resolution parameter, the depth estimation range parameter and the depth estimation point interval parameter in the dynamic perceived parameter set;

and adjusting a camera view cone point cloud feature pooling sub-network of the image feature view conversion network based on the perception range parameters in the dynamic perception parameter set.

According to the vehicle dynamic sensing method provided by the invention, the step of adjusting the depth estimation sub-network of the image feature view conversion network based on the depth estimation range parameter and the depth estimation point interval parameter in the dynamic sensing parameter set comprises the following steps:

Determining a difference value between a maximum value and a minimum value in the depth estimation range parameter as an intermediate value;

determining the ratio between the intermediate value and the depth estimation point interval parameter as the number of target cone points in the depth direction;

determining the sum value of the number of the target view cone points and the number of the original channels of a target network layer as the number of the target channels, wherein the target network layer refers to the last layer of the depth estimation sub-network;

and adjusting the channel number of the target network layer to be the target channel number.

According to the vehicle dynamic sensing method provided by the invention, based on the sensing resolution parameter in the dynamic sensing parameter set, the step of adjusting the camera view cone point cloud feature extraction sub-network of the image feature view conversion network comprises the following steps:

obtaining a target height and a target width of the image feature based on the perceived resolution parameter;

obtaining the number of target view cone points in the depth direction based on the depth estimation range parameter and the depth estimation point interval parameter;

And determining the target height, the target width and the target view cone point quantity as target characteristic shapes, and determining the target characteristic shapes as characteristic shape extraction basis of the camera view cone point cloud characteristic extraction sub-network.

According to the vehicle dynamic sensing method provided by the invention, the sensing resolution parameters comprise: a first image resolution and a second image resolution, the first image resolution being a resolution in an image height direction, the second image resolution being a resolution in an image width direction;

based on the perceived resolution parameter, the step of obtaining the target height and the target width of the image feature comprises:

Determining the ratio of the first image resolution to a target downsampling multiple which is the downsampling multiple of the camera view cone point cloud feature extraction sub-network as the target height;

and determining the ratio between the second image resolution and the target downsampling multiple as the target width.

According to the vehicle dynamic sensing method provided by the invention, the step of obtaining the number of target view cone points in the depth direction based on the depth estimation range parameter and the depth estimation point interval parameter comprises the following steps:

And determining the ratio between the intermediate value and the depth estimation point interval parameter as the number of target cone points in the depth direction.

According to the vehicle dynamic sensing method provided by the invention, based on the sensing range parameter in the dynamic sensing parameter set, the step of adjusting the camera view cone point cloud feature pooling sub-network of the image feature view conversion network comprises the following steps:

And adjusting the target pooling shape of the camera view cone point cloud characteristic pooling sub-network based on the perception range parameter, wherein the target pooling shape refers to the shape of the pooled camera view cone point cloud characteristic.

According to the vehicle dynamic sensing method provided by the invention, based on the sensing range parameter, the step of adjusting the target pooling shape of the camera view cone point cloud characteristic pooling sub-network comprises the following steps:

dividing the sensing distance in each sensing direction in the sensing range parameter by a preset characteristic grid size to obtain a sensing pooling shape;

and adjusting the target pooling shape based on the perception pooling shape.

According to the vehicle dynamic sensing method provided by the invention, the sensing resolution parameters comprise the point cloud resolution, and the point cloud resolution points to the cloud voxelized size; based on the dynamic perception parameter set, the step of adjusting the point cloud feature extraction network of the dynamic perception model comprises the following steps:

And adjusting the target voxel size of the point cloud feature extraction network based on the point cloud resolution, wherein the target voxel size is a voxel size adjustment basis when the point cloud feature extraction network performs voxelization operation.

According to the vehicle dynamic sensing method provided by the invention, based on the dynamic sensing parameter set, the steps of adjusting the data preprocessing strategy and the dynamic sensing model of the sensing system to obtain the sensing result comprise the following steps:

preprocessing the sensing data by utilizing the adjusted data preprocessing strategy to obtain preprocessed sensing data;

And inputting the preprocessed sensing data into the adjusted dynamic sensing model, and performing sensing task processing to obtain the sensing result.

According to the vehicle dynamic sensing method provided by the invention, the sensing data comprises the following steps: the system comprises original point cloud data and original image data, wherein the original point cloud data are data acquired by a laser radar or a millimeter wave radar of a current vehicle, and the original image data are data acquired by a camera device of the current vehicle; the preprocessed sensor data includes: target point cloud data and target image data, wherein the target point cloud data refers to preprocessed original point cloud data, and the target image data refers to preprocessed original image data;

Inputting the preprocessed sensing data into the adjusted dynamic sensing model, and performing sensing task processing, wherein the step of obtaining the sensing result comprises the following steps:

inputting the target point cloud data into a point cloud feature extraction network of the adjusted dynamic perception model, and performing voxelization operation and point cloud feature extraction to obtain target point cloud features;

Inputting the target image data into an image feature extraction network of a dynamic perception model, and extracting image features to obtain two-dimensional image features;

performing image feature conversion based on the two-dimensional image features and the image feature view conversion network of the adjusted dynamic perception model to obtain three-dimensional camera view cone point cloud features;

Inputting the target point cloud characteristics and the camera view cone point cloud characteristics into a characteristic fusion network of a dynamic perception model to obtain fusion characteristics;

And inputting the fusion characteristics into a perception task processing layer of a dynamic perception model, and performing perception task processing to obtain the perception result.

According to the vehicle dynamic sensing method provided by the invention, the target image data is input into the image feature extraction network of the dynamic sensing model, the image feature extraction is carried out, and the step of obtaining the two-dimensional image features comprises the following steps:

inputting the target image data into the image feature extraction network of the adjusted dynamic perception model, and extracting image features to obtain initial image features;

grouping all the initial image features according to the resolution of each target image data to obtain a plurality of initial image feature groups, wherein each initial image feature group comprises a plurality of initial image features with the same resolution;

Acquiring the average value of the initial image features in each initial image feature group;

Acquiring variances of the initial image features in each initial image feature group based on the mean value;

Based on the variance of each initial image feature group and a preset normalization formula, respectively carrying out normalization processing on each initial image feature to obtain normalized image features;

Determining the product between the normalized image feature and a target scaling parameter as a scaled image feature, the target scaling parameter being a scaling parameter corresponding to the resolution of the current normalized image feature;

Determining a product between the scaled image feature and a target translation parameter as the two-dimensional image feature, the target translation parameter being a translation parameter corresponding to a resolution of a current scaled image feature; the target scaling parameters and the target translation parameters corresponding to each resolution are determined by the dynamic perception model after repeated iterative learning;

The mathematical expression of the normalization formula is:

Where denotes the normalized image feature,/> denotes the variance,/> is the adjustment term,/> denotes the j-th initial image feature at the i-th resolution, and/> denotes the mean of the initial image features at the i-th resolution.

According to the vehicle dynamic sensing method provided by the invention, the step of converting the image characteristics based on the two-dimensional image characteristics and the image characteristic view conversion network of the adjusted dynamic sensing model to obtain the three-dimensional camera view cone point cloud characteristics comprises the following steps:

Inputting the two-dimensional image features into a feature optimization network of a dynamic perception model to perform feature optimization to obtain optimized two-dimensional image features;

and inputting the optimized two-dimensional image features into the image feature view conversion network of the adjusted dynamic perception model, and converting the image features to obtain the camera view cone point cloud features.

The invention also provides a vehicle dynamic sensing system, comprising:

The acquisition module is used for acquiring dynamic perception associated information of the current vehicle, and the dynamic perception associated information comprises: vehicle external environment information, vehicle running state information and processor load information of a sensing system;

the dynamic perception parameter set determining module is configured to obtain a dynamic perception parameter set based on the dynamic perception association information and a preset dynamic perception parameter set matching rule, where the dynamic perception association information corresponds to the dynamic perception parameter set, and the dynamic perception parameter set includes: a perception range parameter, a perception resolution parameter, a depth estimation range parameter, and a depth estimation point interval parameter;

The sensing module is used for adjusting a data preprocessing strategy of the sensing system and a dynamic sensing model based on the dynamic sensing parameter set to obtain a sensing result, wherein the data preprocessing strategy is used for preprocessing sensing data acquired by the sensing system, and the dynamic sensing model is used for performing sensing task processing based on the preprocessed sensing data to obtain the sensing result.

The invention also provides a dynamic perception model training method, which comprises the following steps:

Obtaining a training set, the training set comprising: a plurality of sets of training data, the training data comprising: a dynamic perception associated information sample, a sensing data sample and a perception real result; the dynamic perception associated information sample comprises: a vehicle external environment information sample, a vehicle running state information sample, and a processor load information sample of a vehicle sensing system;

obtaining a dynamic perception parameter set sample based on the dynamic perception associated information sample and a preset dynamic perception parameter set matching rule, wherein the dynamic perception associated information sample corresponds to the dynamic perception parameter set sample, and the dynamic perception parameter set sample comprises: a perception range parameter, a perception resolution parameter, a depth estimation range parameter, and a depth estimation point interval parameter;

based on the dynamic perception parameter set sample, adjusting a data preprocessing strategy and a dynamic perception model of the perception system;

Preprocessing the sensing data sample by utilizing the adjusted data preprocessing strategy to obtain a preprocessed sensing data sample;

Inputting the preprocessed sensing data sample into an adjusted dynamic sensing model, and performing sensing task processing to obtain a sensing prediction result;

And training the dynamic perception model based on the difference between the perception prediction result and the corresponding perception real result to obtain a trained dynamic perception model.

According to the dynamic perception model training method provided by the invention, the sensing data sample comprises the following steps: the system comprises an original point cloud data sample and an original image data sample, wherein the original point cloud data sample is data acquired by a laser radar or a millimeter wave radar of a vehicle, and the original image data sample is data acquired by a camera device of the vehicle; the preprocessed sensor data sample comprises: the target point cloud data sample and the target image data sample, wherein the target point cloud data sample refers to a preprocessed original point cloud data sample, and the target image data sample refers to a preprocessed original image data sample;

Inputting the preprocessed sensing data sample into an adjusted dynamic perception model, and carrying out perception task processing, wherein the step of obtaining a perception prediction result comprises the following steps of:

inputting the target point cloud data sample into a point cloud feature extraction network of the adjusted dynamic perception model, and performing voxelization operation and point cloud feature extraction to obtain a target point cloud feature sample;

Inputting the target image data sample into an image feature extraction network of a dynamic perception model, and extracting image features to obtain a two-dimensional image feature sample;

Performing image feature conversion based on the two-dimensional image feature sample and the image feature view conversion network of the adjusted dynamic perception model to obtain a three-dimensional camera view cone point cloud feature sample;

Inputting the target point cloud characteristic sample and the camera view cone point cloud characteristic sample into a characteristic fusion network of a dynamic perception model to obtain a fusion characteristic sample;

And inputting the fusion characteristic sample into a perception task processing layer of a dynamic perception model, and performing perception task processing to obtain the perception prediction result.

According to the method for training the dynamic perception model provided by the invention, based on the two-dimensional image feature sample and the image feature view conversion network of the adjusted dynamic perception model, the image feature conversion is carried out, and the step of obtaining the three-dimensional camera view cone point cloud feature sample comprises the following steps:

inputting the two-dimensional image feature sample into a feature optimization network of a dynamic perception model to perform feature optimization to obtain an optimized two-dimensional image feature sample;

And inputting the optimized two-dimensional image characteristic sample into the image characteristic view conversion network of the adjusted dynamic perception model to perform image characteristic conversion so as to obtain the camera view cone point cloud characteristic sample.

According to the dynamic perception model training method provided by the invention, after the step of obtaining the optimized two-dimensional image characteristic sample, the method further comprises the following steps:

acquiring an image feature gradient of the optimized two-dimensional image feature sample;

Determining the product of the image characteristic gradient and the corresponding gradient balance factor as a target characteristic gradient, wherein the gradient balance factor is obtained in the following way: obtaining the sensing resolution parameters corresponding to the image feature gradients at present, wherein the sensing resolution parameters comprise: a first image resolution and a second image resolution, the first image resolution being a resolution in an image height direction, the second image resolution being a resolution in an image width direction; determining a product between the first image resolution and the second image resolution as a to-be-determined value; determining the ratio between 1 and the value to be determined as the gradient balance factor;

And training the dynamic perception model based on the difference between the target feature gradient and a preset standard feature gradient.

The invention also provides a dynamic perception model training system, which comprises:

The training set acquisition module is used for acquiring a training set, and the training set comprises: a plurality of sets of training data, the training data comprising: a dynamic perception associated information sample, a sensing data sample and a perception real result; the dynamic perception associated information sample comprises: a vehicle external environment information sample, a vehicle running state information sample, and a processor load information sample of a vehicle sensing system;

The dynamic perception parameter set sample acquisition module is configured to obtain a dynamic perception parameter set sample based on the dynamic perception correlation information sample and a preset dynamic perception parameter set matching rule, where the dynamic perception correlation information sample corresponds to the dynamic perception parameter set sample, and the dynamic perception parameter set sample includes: a perception range parameter, a perception resolution parameter, a depth estimation range parameter, and a depth estimation point interval parameter;

The adjusting module is used for adjusting the data preprocessing strategy and the dynamic perception model of the perception system based on the dynamic perception parameter set sample;

The preprocessing module is used for preprocessing the sensing data sample by utilizing the adjusted data preprocessing strategy to obtain a preprocessed sensing data sample;

The task processing module is used for inputting the preprocessed sensing data sample into the adjusted dynamic perception model, and carrying out perception task processing to obtain a perception prediction result;

and the training module is used for training the dynamic perception model based on the difference between the perception prediction result and the corresponding perception real result to obtain a trained dynamic perception model.

The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the vehicle dynamic perception method according to any one of the above or the dynamic perception model training method according to any one of the above when executing the program.

The invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method of vehicle dynamic perception as defined in any one of the above, or a method of training a dynamic perception model as defined in any one of the above.

The invention has the beneficial effects that: the invention provides a vehicle dynamic perception method, a system and a dynamic perception model training method, wherein the dynamic perception associated information of a current vehicle is acquired, and the dynamic perception associated information comprises the following components: vehicle external environment information, vehicle running state information and processor load information of a sensing system; based on the dynamic perception associated information and a preset dynamic perception parameter set matching rule, a dynamic perception parameter set is obtained, the dynamic perception associated information corresponds to the dynamic perception parameter set, and the dynamic perception parameter set comprises: a perception range parameter, a perception resolution parameter, a depth estimation range parameter, and a depth estimation point interval parameter; based on the dynamic perception parameter set, the data preprocessing strategy and the dynamic perception model of the perception system are adjusted to obtain a perception result. Therefore, the calculation burden of the perception system is reduced as much as possible, the processing of the perception task is completed as soon as possible, and the instantaneity and the robustness of the perception system and the unmanned system are ensured.

Drawings

In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a vehicle dynamic sensing method provided by the invention;

FIG. 2 is a schematic flow chart of determining a dynamic sensing parameter set in the vehicle dynamic sensing method provided by the invention;

FIG. 3 is a schematic flow chart of determining a dynamic sensing parameter set based on a matching rule of the dynamic sensing parameter set in the vehicle dynamic sensing method provided by the invention;

FIG. 4 is a schematic diagram of the distribution of sensing ranges in the vehicle dynamic sensing method provided by the invention;

FIG. 5 is a schematic flow chart of dynamic sensing in the vehicle dynamic sensing method provided by the invention;

FIG. 6 is a schematic diagram of a preprocessing flow of sensing data in the vehicle dynamic sensing method provided by the invention;

FIG. 7 is a schematic diagram of a process flow of a dynamic sensing model for image data in the vehicle dynamic sensing method provided by the invention;

FIG. 8 is a schematic diagram of a process flow of a dynamic sensing model for point cloud data in the vehicle dynamic sensing method provided by the invention;

FIG. 9 is a schematic flow chart of normalizing image features in the vehicle dynamic sensing method provided by the invention;

FIG. 10 is a schematic flow chart of feature optimization of two-dimensional image features in the vehicle dynamic sensing method provided by the invention;

FIG. 11 is a schematic diagram of a vehicle dynamics awareness system according to the present invention;

FIG. 12 is a flow chart of a dynamic perception model training method provided by the invention;

FIG. 13 is a schematic diagram of a method for balancing multiple gradients in a vehicle dynamics sensing method provided by the present invention;

FIG. 14 is a schematic diagram of a dynamic perceptual model training system provided by the present invention;

Fig. 15 is a schematic structural diagram of an electronic device provided by the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

By way of example, the method and system for vehicle dynamic perception and the training method for dynamic perception model provided by the invention are described below with reference to fig. 1 to 15.

Referring to fig. 1, the vehicle dynamic sensing method provided in the present embodiment includes:

S110: collecting dynamic perception associated information of a current vehicle, wherein the dynamic perception associated information comprises the following steps: vehicle external environment information, vehicle operating state information, and processor load information of the perception system.

The vehicle external environment information includes a road type and a weather type. The road type includes: urban roads, highways, rural roads, campus roads, tunnel roads, high altitude roads, and the like. The weather types include: sunny days, rain, snow, fog, etc. The vehicle running state information includes: vehicle speed, acceleration, yaw rate, lateral speed, etc. The sensing system refers to a sensing system of a driving system or an unmanned system of a current vehicle, and the processor load information refers to the utilization rate of the sensing system, and specifically comprises the following steps: memory utilization, CPU (Central Processing Unit ) utilization, and GPU (Graphics Processing Unit, graphics processor) utilization, etc. The embodiment defines the above information types (i.e. the external environment information of the vehicle, the running state information of the vehicle and the processor load information of the sensing system), and defines the information items of each information type (such as urban roads, expressways, rural roads, etc.), so that the definition of the dynamic sensing related information is more comprehensive, and the method is beneficial to the subsequent acquisition of a better dynamic sensing parameter set with higher matching degree.

S120: based on the dynamic perception associated information and a preset dynamic perception parameter set matching rule, a dynamic perception parameter set is obtained, the dynamic perception associated information corresponds to the dynamic perception parameter set, and the dynamic perception parameter set comprises: a perceptual range parameter, a perceptual resolution parameter, a depth estimation range parameter, and a depth estimation point interval parameter.

It should be noted that the sensing range parameter includes sensing distances in a plurality of sensing directions, which is expressed in , where/> represents a sensing range parameter of an s-type sensor,/> represents a sensing distance of an i-th sensor of the s-type sensor, s represents a type of a sensor of a vehicle, such as a laser radar, a millimeter wave radar, and an image pickup device (camera), i represents a sensor number of the type, and/> represents a total number of the s-type sensors.

It should also be noted that the mathematical expression of the perceived resolution parameter is , where/() represents the perceived resolution parameter of the s-type sensor, and/() represents the perceived resolution parameter of the i-th sensor of the s-type sensor.

It should be mentioned that the depth estimation range parameter and the depth estimation point interval parameter are both set for an image capturing device, i.e. a camera sensor, and in the mainstream sensing method, by converting two-dimensional image features into three-dimensional space, depth estimation is required in the view conversion process, i.e. a series of discrete depth points (such as 1 meter, 2 meters,..10 meters, etc.) are set according to the depth estimation point interval parameter within a given depth estimation range, and probability values at each depth point are estimated. Therefore, the depth estimation range parameter and the depth estimation point interval parameter obtained in step S120 are both used to adjust the image feature view conversion network of the dynamic perceptual model.

Referring to fig. 2, in the present embodiment, a dynamic sensing parameter set corresponding to dynamic sensing association information (vehicle external environment information, vehicle running state information, and processor load information of a sensing system) is determined by using a dynamic sensing parameter set matching rule, and multiple sets of dynamic sensing parameter sets and dynamic sensing association information corresponding to each other are defined in the dynamic sensing parameter set matching rule. The dynamic sensing parameter sets corresponding to different dynamic sensing associated information are different, for example: when the vehicle is positioned in a tunnel road or a park road, the perceived distance of the vehicle in the left-right direction can be shortened, and when the weather type is rainy, the safety distance, namely the perceived distance of the vehicle in the forward direction and the backward direction, and the like, should be increased. According to the embodiment, the dynamic perception parameter set is obtained, so that the data preprocessing strategy and the dynamic perception model of the perception system can be conveniently adjusted based on the dynamic perception parameter set, and the calculation load of the system is reduced.

It should be noted that fig. 3 illustrates a flow of determining a dynamic sensing parameter set based on a dynamic sensing parameter set matching rule in this embodiment, as shown in fig. 3, in a case of obtaining dynamic sensing related information, vehicle external environment information is sequentially determined (i.e., a current road type and a weather type are sequentially determined, specifically, whether the vehicle external environment is an urban road, an expressway, a park road, or the like is first determined, whether the weather type is a sunny day or a rainy day, or the like) vehicle running state information (i.e., a section where a vehicle speed, an acceleration, a yaw rate, a lateral speed, or the like is determined, e.g., a section where the vehicle speed is located in a speed section 1 or a speed section 2, or the like) and processor load information of a sensing system (i.e., a load section where a processor of the sensing system is located is determined, e., a load section 1 or a load section 2, or the like, and it should be noted that the load section 1 includes a memory load section 1, a CPU load section 2, and a GPU load section 2, and the like).

In addition, it should be mentioned that, in this embodiment, on the premise of ensuring the driving safety of the vehicle, the calculation load and delay of the sensing system are reduced, and the external environment information of the vehicle, the running state information of the vehicle and the load information of the processor of the sensing system are comprehensively utilized to obtain a corresponding dynamic sensing parameter set. Compared with a fixed sensing range parameter, a sensing resolution parameter, a depth estimation range parameter and a depth estimation point interval parameter, the method for determining the dynamic sensing parameter set in the embodiment is more beneficial to reducing the calculation load of a sensing system, reducing delay and considering safety, instantaneity and robustness.

Fig. 4 is a schematic diagram showing the distribution of the sensing range, and the triangle symbol in fig. 4 is a simplified vehicle, and the rectangle on the periphery of the vehicle represents the sensing range, and the sensing range is determined by the sensing range parameter. The different dynamic sensing related information corresponds to different sensing ranges, such as (a) a sensing range of 100 m100 m, (b) a sensing range of 50m50m, (c) a sensing range of 100 m50m, and (d) a sensing range of 200m50 m. Fig. 4 illustrates the sensing ranges of four sensing directions, namely, the forward direction, the backward direction, the left direction and the right direction, but the present invention is not limited to these four sensing directions in the specific implementation process. The perceived direction may be increased or decreased as appropriate based on different application scenarios. The sensing resolution parameters are the same, it can be understood that the sensing resolution parameters corresponding to the sensors located in different sensing directions are different, and in a specific application process, the number of sensing resolution parameters to be acquired can be adjusted, for example, the forward sensing resolution parameters, the backward sensing resolution parameters and the like are acquired, which are not described herein.

S130: based on the dynamic sensing parameter set, a data preprocessing strategy of the sensing system and a dynamic sensing model are adjusted to obtain a sensing result, wherein the data preprocessing strategy is used for preprocessing sensing data acquired by the sensing system, and the dynamic sensing model is used for performing sensing task processing based on the preprocessed sensing data to obtain the sensing result.

Specifically, as shown in fig. 5, the data preprocessing strategy and the dynamic perception model of the perception system are adjusted based on the dynamic perception parameter set. And processing the sensing data acquired by the sensing system, namely the sensing data acquired by the sensor of the vehicle, by utilizing the adjusted data preprocessing strategy and the dynamic sensing model to obtain a sensing result. In the embodiment, the data preprocessing strategy and the dynamic sensing model of the sensing system are adjusted based on the dynamic sensing parameter set, so that the calculation load of the sensing system can be well reduced, and the instantaneity and the robustness of the sensing system and the unmanned system are ensured.

In some embodiments, the step of adjusting the data preprocessing strategy of the perception system based on the set of dynamic perception parameters comprises:

When the data preprocessing strategy and the dynamic perception model are adjusted, the required parameters are required to be read from the dynamic perception parameter set, and the data preprocessing strategy is adjusted based on the read parameters. In the embodiment, the original point cloud data which is positioned outside the sensing range corresponding to the sensing range parameter in the sensing data is cleared by reading the sensing range parameter in the dynamic sensing parameter set, so that the calculation load of the sensing system can be effectively reduced. For example: let the range parameters of lidar or millimeter wave radar be , where/() represents the forward perceived distance,/> represents the backward perceived distance,/> represents the left perceived distance, and/() represents the right perceived distance. Let's assume that the set of original point cloud data is/> , where/> represents the original point cloud data acquired by the ith laser radar or millimeter wave radar, and k represents the number of laser radars or millimeter wave radars. The mathematical expression of the screening flow of the original point cloud data is: Wherein,/> denotes the target point cloud data, and/> denotes a point cloud data screening function, which is used to check whether each original point cloud data is located in a sensing range corresponding to the sensing range parameter, and clear the original point cloud data located outside the sensing range to obtain the target point cloud data.

Specifically, a first image resolution and a second image resolution in the perceived resolution parameters are read, wherein the first image resolution is a resolution in the image height direction, and the second image resolution is a resolution in the image width direction. The original resolution of the original image data is adjusted or updated based on the first image resolution and the second image resolution. It should be mentioned that the perceived resolution parameters include a resolution for the image data (e.g. the first image resolution and the second image resolution described above), and a resolution for the point cloud data (i.e. the point cloud resolution). In this embodiment, the original resolution of the original image data is adjusted, so that the resolutions of different images can be unified, and the difficulty and burden of subsequent data processing are reduced.

It should be noted that the sensing system of the vehicle generally includes a plurality of data preprocessing strategies, and the adjustment of the data preprocessing strategies in the above embodiments is not directed to all the data preprocessing strategies. And for the data preprocessing strategy which is irrelevant to the dynamic perception parameter set, namely irrelevant to the perception range parameter, the perception resolution parameter, the depth estimation range parameter and the depth estimation point interval parameter, the original strategy or the original execution flow is kept, and the dynamic perception parameter set is not required to be adjusted. As shown in fig. 6, for the s-type sensor (such as an image capturing device, a laser radar or a millimeter wave radar, etc.), the number of data preprocessing strategies corresponding to the s-type sensor is plural, and in fig. 6 refers to the number of data preprocessing strategies corresponding to the s-type sensor. For the adjustment of the data preprocessing strategy, in this embodiment, the data preprocessing strategy related to the dynamic sensing parameter set is only adjusted based on the dynamic sensing parameter set, such as the data preprocessing strategy 2 in fig. 6. As for data preprocessing strategies that are independent of the dynamic perception parameter set, such as data preprocessing strategy 1, data preprocessing strategy/> , etc., no adjustment is required. It should be noted that n in fig. 6 represents the number of the s-th type of sensors, and each of the s-th type of sensors (e.g., sensor 1, sensor 2 , sensor n) is respectively corresponding to a corresponding data preprocessing policy (e.g., data preprocessing policy 1, data preprocessing policy 2 , data preprocessing policy/> ).

In some embodiments, the step of adjusting the dynamic perception model of the perception system based on the set of dynamic perception parameters comprises:

And adjusting an image feature view conversion network and a point cloud feature extraction network of the dynamic perception model based on the dynamic perception parameter set. The image feature view conversion network is used for converting the two-dimensional image features output by the image feature extraction network of the dynamic perception model into three-dimensional camera view cone point cloud features; the input data of the image feature extraction network is target image data, and the target image data is preprocessed original image data; the point cloud feature extraction network is used for carrying out feature extraction on target point cloud data to obtain target point cloud features, the target point cloud data are preprocessed original point cloud data, the original image data and the original point cloud data are both the sensing data, the original image data are data acquired by a camera device of a current vehicle, and the original point cloud data are data acquired by a laser radar or a millimeter wave radar of the current vehicle.

The image feature view conversion network and the point cloud feature extraction network of the dynamic perception model are adjusted based on the dynamic perception parameter set, so that the calculation load of the dynamic perception model can be reduced, and dynamic perception is realized.

In some embodiments, the step of transforming the network of image feature views of the dynamic perceptual model comprises:

1. And adjusting a depth estimation sub-network of the image feature view conversion network based on the depth estimation range parameter and the depth estimation point interval parameter in the dynamic perception parameter set.

2. And adjusting a camera view cone point cloud feature extraction sub-network of the image feature view conversion network based on the perceived resolution parameter, the depth estimation range parameter and the depth estimation point interval parameter in the dynamic perceived parameter set.

3. And adjusting a camera view cone point cloud feature pooling sub-network of the image feature view conversion network based on the perception range parameters in the dynamic perception parameter set.

The image feature view conversion network comprises a depth estimation sub-network, a camera view cone point cloud feature extraction sub-network and a camera view cone point cloud feature pooling sub-network which are sequentially connected. The depth estimation sub-network is used for estimating the depth of the input two-dimensional image features, namely estimating the depth values of the two-dimensional image features. The camera view cone point cloud feature extraction sub-network is used for extracting features based on the space position of the two-dimensional image features and the estimated depth values. The camera view cone point cloud feature pooling sub-network is used for feature pooling. By performing the adjustment operation, the processing difficulty and complexity of the image feature view conversion network can be effectively reduced, the calculated amount of the image feature view conversion network is reduced, and the calculation speed is increased.

In some embodiments, the step of adjusting the depth estimation sub-network of the image feature view conversion network based on the depth estimation range parameter and the depth estimation point spacing parameter in the dynamic perceptual parameter set comprises:

first, a difference between a maximum value and a minimum value in the depth estimation range parameter is determined as an intermediate value.

And secondly, determining the ratio between the intermediate value and the depth estimation point interval parameter as the target number of cone points in the depth direction.

And then, determining the sum value of the target cone point number and the original channel number of a target network layer as the target channel number, wherein the target network layer refers to the last layer of the depth estimation sub-network.

And finally, adjusting the channel number of the target network layer to the target channel number.

Specifically, assume that the depth estimation range parameter is , where/> is the minimum value in the depth estimation range parameter and/> is the maximum value in the depth estimation range parameter. The target cone point number in the depth direction, where/() represents a depth estimation point spacing parameter. The number of target channels is , where/() represents the number of target channels and/() represents the number of original channels of the target network layer.

In some embodiments, the step of adjusting the camera cone point cloud feature extraction sub-network of the image feature view conversion network based on the perceived resolution parameters in the set of dynamic perceived parameters comprises:

1. And obtaining the target height and the target width of the image characteristic based on the perceived resolution parameter.

2. And obtaining the number of target cone points in the depth direction based on the depth estimation range parameter and the depth estimation point interval parameter.

3. And determining the target height, the target width and the target view cone point quantity as target characteristic shapes, and determining the target characteristic shapes as characteristic shape extraction basis of the camera view cone point cloud characteristic extraction sub-network.

By adopting the steps to adjust the camera view cone point cloud feature extraction sub-network, the calculation pressure of the camera view cone point cloud feature extraction sub-network can be effectively reduced, and the calculation speed is increased.

In some embodiments, the perceived resolution parameters include: a first image resolution, which is a resolution in the image height direction, and a second image resolution, which is a resolution in the image width direction.

Further, the step of obtaining the target height and the target width of the image feature based on the perceived resolution parameter includes:

1. And determining the ratio of the first image resolution to a target downsampling multiple as the target height, wherein the target downsampling multiple is the downsampling multiple of the camera view cone point cloud feature extraction sub-network.

2. And determining the ratio between the second image resolution and the target downsampling multiple as the target width.

Note that, the resolution of the image in the perceived resolution parameter is denoted by , where/() denotes the first image resolution and/() denotes the second image resolution. The target height is/> , the target width is/> , and the feature map size is/> , where/> represents the target downsampling multiple.

In some embodiments, the step of obtaining the number of target cone points in the depth direction based on the depth estimation range parameter and the depth estimation point interval parameter includes:

1. And determining the difference value between the maximum value and the minimum value in the depth estimation range parameter as an intermediate value.

2. And determining the ratio between the intermediate value and the depth estimation point interval parameter as the number of target cone points in the depth direction.

In some embodiments, the step of adjusting the camera cone point cloud feature pooling sub-network of the image feature view conversion network based on the perception range parameters in the dynamic perception parameter set comprises:

And adjusting the target pooling shape of the camera view cone point cloud characteristic pooling sub-network based on the perception range parameter, wherein the target pooling shape refers to the shape of the pooled camera view cone point cloud characteristic. The target pooling shape of the camera view cone point cloud feature pooling sub-network is adjusted based on the perception range parameter, so that the calculated amount of the camera view cone point cloud feature pooling sub-network can be reduced.

In some embodiments, the step of adjusting the target pooling shape of the camera cone point cloud feature pooling sub-network based on the perception range parameter comprises:

1. And dividing the sensing distance in each sensing direction in the sensing range parameters by a preset characteristic grid size to obtain a sensing pooling shape.

2. And adjusting the target pooling shape based on the perception pooling shape.

Specifically, assuming the perceptual span parameter , is the feature mesh size, the perceptual pooling shape is/> .

In some embodiments, the perceived resolution parameters include a point cloud resolution that points to a cloud voxelized size, such as (0.075,0.075,0.2) representing the size of each voxel in three dimensions, length, width, and height. Further, the step of adjusting the point cloud feature extraction network of the dynamic perception model based on the dynamic perception parameter set includes:

And adjusting the target voxel size of the point cloud feature extraction network based on the point cloud resolution, wherein the target voxel size is a voxel size adjustment basis when the point cloud feature extraction network performs voxelization operation. Specifically, the target voxel size of the point cloud feature extraction network is adjusted to a size in the point cloud resolution, so that the calculation amount of the point cloud feature extraction network is reduced.

In some embodiments, the step of adjusting the data preprocessing strategy and the dynamic perception model of the perception system to obtain the perception result based on the dynamic perception parameter set includes:

1. and preprocessing the sensing data by utilizing the adjusted data preprocessing strategy to obtain preprocessed sensing data.

2. And inputting the preprocessed sensing data into the adjusted dynamic sensing model, and performing sensing task processing to obtain the sensing result.

In some embodiments, the sensing data comprises: the system comprises original point cloud data and original image data, wherein the original point cloud data are data acquired by a laser radar or a millimeter wave radar of a current vehicle, and the original image data are data acquired by a camera device of the current vehicle; the preprocessed sensor data includes: target point cloud data and target image data, wherein the target point cloud data refers to preprocessed original point cloud data, and the target image data refers to preprocessed original image data.

Further, the step of inputting the preprocessed sensing data into the adjusted dynamic sensing model to perform sensing task processing to obtain the sensing result includes:

firstly, inputting the target point cloud data into a point cloud feature extraction network of the adjusted dynamic perception model, and carrying out voxelization operation and point cloud feature extraction to obtain target point cloud features.

And secondly, inputting the target image data into an image feature extraction network of the dynamic perception model, and extracting the image features to obtain two-dimensional image features.

And then, performing image feature conversion based on the two-dimensional image features and the image feature view conversion network of the adjusted dynamic perception model to obtain three-dimensional camera view cone point cloud features.

And inputting the target point cloud characteristics and the camera view cone point cloud characteristics into a characteristic fusion network of a dynamic perception model to obtain fusion characteristics.

And finally, inputting the fusion characteristics into a perception task processing layer of a dynamic perception model, and performing perception task processing to obtain the perception result. By executing the above operation, the calculation burden of the sensing system can be effectively reduced, and the instantaneity and the robustness of the sensing system and the unmanned system can be improved.

Fig. 7 shows a dynamic perceptual model process flow for image data. As shown in fig. 7, first, the image feature view conversion network is adjusted based on the set of dynamic sensing parameters by using a preset dynamic sensing adjustment module. Secondly, inputting target image data (i.e. preprocessed original image data) into a corresponding image feature extraction network (the resolution and setting positions of different image pickup devices may be different, so that each image pickup device corresponds to one image feature extraction network respectively, and after preprocessing, the original image data collected by each image pickup device is input into the corresponding image feature extraction network) for image feature conversion, so as to obtain two-dimensional image features. And then inputting the target image data into an image feature extraction network to extract the image features, so as to obtain two-dimensional image features. And performing image feature conversion based on the two-dimensional image features and the image feature view conversion network of the adjusted dynamic perception model to obtain three-dimensional camera view cone point cloud features. And finally, inputting the camera view cone point cloud characteristics into a characteristic fusion network to perform characteristic fusion.

Fig. 8 shows a dynamic perception model processing flow for point cloud data. As shown in fig. 8, first, the point cloud feature extraction network is adjusted based on the dynamic sensing parameter set by using a preset dynamic sensing adjustment module. Secondly, inputting target point cloud data (i.e. preprocessed original point cloud data) into an adjusted point cloud feature extraction network, and carrying out voxelization operation and point cloud feature extraction to obtain target point cloud features. And finally, inputting the cloud characteristics of the target point into a characteristic fusion network to perform characteristic fusion.

The feature fusion network fuses all the target point cloud features and the camera view cone point cloud features to obtain fusion features. And then inputting the fusion characteristics into a perception task processing layer of the dynamic perception model to perform perception task processing so as to obtain a perception result. The perception task processing layer includes a plurality of perception tasks.

In the dynamic perception model, when different image resolutions are set for different image pickup devices (cameras), the processing difficulty of the dynamic perception model is increased, and the difference of image feature distribution of the different resolutions is large, so that the optimization difficulty of batch normalization layers in the dynamic perception model is large. Based on the above, the invention better solves the problem of larger image characteristic distribution difference of different resolutions by respectively setting corresponding normalization parameters (namely the target scaling parameters and the target translation parameters in the lower embodiment) for each image resolution in the image characteristic extraction network of the dynamic perception model.

In some embodiments, the step of inputting the target image data into an image feature extraction network of a dynamic perception model to perform image feature extraction, and obtaining two-dimensional image features includes:

1. Inputting the target image data into the image feature extraction network of the adjusted dynamic perception model, and extracting image features to obtain initial image features.

2. And grouping all the initial image features according to the resolution of each target image data to obtain a plurality of initial image feature groups, wherein each initial image feature group comprises a plurality of initial image features with the same resolution. For example: all initial image features are divided into r initial image feature groups , where/() represents the initial image features and r represents the number of initial image feature groups.

3. And acquiring the average value of the initial image features in each initial image feature group.

Specifically, the mathematical expression of the mean value of the initial image features is:

Where denotes the mean value of the initial image features,/> denotes the number of initial image features of the current resolution and denotes the initial image features of the current resolution.

4. And acquiring variances of the initial image features in each initial image feature group based on the mean value.

Specifically, the mathematical expression of the variance of the initial image features is:

Wherein represents the variance of the initial image features.

5. And respectively carrying out normalization processing on each initial image feature based on the variance of each initial image feature group and a preset normalization formula to obtain normalized image features.

The mathematical expression of the normalization formula is:

Where denotes the normalized image feature,/> denotes the variance,/> is the adjustment term (typically 10 ^-7 or 10 ^-8, etc.), denotes the j-th initial image feature at the i-th resolution, and/> denotes the mean of the initial image features at the i-th resolution.

6. And determining the product between the normalized image feature and a target scaling parameter as a scaled image feature, wherein the target scaling parameter refers to a scaling parameter corresponding to the resolution of the current normalized image feature.

7. Determining a product between the scaled image feature and a target translation parameter as the two-dimensional image feature, the target translation parameter being a translation parameter corresponding to a resolution of a current scaled image feature; the target scaling parameters and the target translation parameters corresponding to each resolution are determined by the dynamic perception model after repeated iterative learning. It should be noted that, through multiple iterative learning, a better target scaling parameter and a better target translation parameter are determined for each resolution, and the normalized image features are scaled and translated by using the target scaling parameter and the target translation parameter, so that the distribution difference between the normalized image features with different resolutions can be reduced better.

Specifically, is scaled and translated to obtain two-dimensional image features. The mathematical expression of scaling and translation of/> is: /(I)

Where denotes a two-dimensional image feature,/> denotes a target zoom parameter, and/> denotes a target pan parameter.

Fig. 9 shows a flow of normalizing the image features, where, as shown in fig. 9, in the case of obtaining the initial image features, multiple data normalization processes are performed based on the initial image features and corresponding normalization parameters (i.e., the target zoom parameter and the target pan parameter), so as to obtain normalized image features, i.e., two-dimensional image features. The initial image feature (resolution 1) in fig. 9 corresponds to the normalized parameter (resolution 1), and a two-dimensional image feature (resolution 1) is obtained. The initial image feature (resolution 2) corresponds to the normalized parameter (resolution 2) and results in a two-dimensional image feature (resolution 2). The initial image feature (resolution r) corresponds to the normalized parameter (resolution r) and results in a two-dimensional image feature (resolution r), and so on.

In order to further reduce the distribution difference of the image features with different resolutions, the embodiment provides a feature optimization mode to perform feature optimization on the two-dimensional image features. Specifically, the step of performing image feature conversion to obtain three-dimensional camera view cone point cloud features based on the two-dimensional image features and the image feature view conversion network of the adjusted dynamic perception model includes:

1. And inputting the two-dimensional image features into a feature optimization network of the dynamic perception model to perform feature optimization, so as to obtain optimized two-dimensional image features.

It should be noted that the feature optimization network is a trained convolutional neural network or a trained convolutional neural network, or may be one or more fully-connected layers, or may be other network layers. The feature optimization network has feature optimization capability through repeated learning. By inputting the two-dimensional image features into the feature optimization network, the distribution difference between the two-dimensional image features with different resolutions can be further reduced, and the optimization difficulty of the subsequent feature fusion network is reduced. It should also be noted that each resolution two-dimensional image feature corresponds to a feature optimization network.

2. And inputting the optimized two-dimensional image features into the image feature view conversion network of the adjusted dynamic perception model, and converting the image features to obtain the camera view cone point cloud features.

Fig. 10 shows a flow of feature optimization of two-dimensional image features. As shown in fig. 10, first, target image data with different resolutions is input into an image feature extraction network of a dynamic perception model, and image feature extraction is performed to obtain two-dimensional image features. And secondly, respectively inputting the two-dimensional image features with different resolutions into corresponding feature optimization networks to perform feature optimization, and obtaining the optimized two-dimensional image features. The target image data in fig. 10 includes: target image data (resolution 1), target image data (resolution 2), target image data (resolution r), and the like, the corresponding feature optimization network includes: a feature optimization network (resolution 1), a feature optimization network (resolution 2), a feature optimization network (resolution r), and the like. The optimized two-dimensional image features include: an optimized two-dimensional image feature (resolution 1), an optimized two-dimensional image feature (resolution 2), an optimized two-dimensional image feature (resolution r), and the like.

The vehicle dynamic sensing system provided by the invention is described below, and the vehicle dynamic sensing system described below and the vehicle dynamic sensing method described above can be referred to correspondingly.

Referring to fig. 11, the vehicle dynamic sensing system provided in the present embodiment includes:

the acquisition module 1110 is configured to acquire dynamic perception association information of a current vehicle, where the dynamic perception association information includes: vehicle external environment information, vehicle running state information and processor load information of a sensing system;

The dynamic sensing parameter set determining module 1120 is configured to obtain a dynamic sensing parameter set based on the dynamic sensing association information and a preset dynamic sensing parameter set matching rule, where the dynamic sensing association information corresponds to the dynamic sensing parameter set, and the dynamic sensing parameter set includes: a perception range parameter, a perception resolution parameter, a depth estimation range parameter, and a depth estimation point interval parameter;

The sensing module 1130 is configured to adjust a data preprocessing policy of the sensing system and a dynamic sensing model based on the dynamic sensing parameter set to obtain a sensing result, where the data preprocessing policy is used to preprocess sensing data acquired by the sensing system, and the dynamic sensing model is used to perform sensing task processing based on the preprocessed sensing data to obtain the sensing result. The vehicle dynamic sensing system in the embodiment can reduce the calculation burden of the sensing system as much as possible, complete the processing of the sensing task as soon as possible, and ensure the instantaneity and the robustness of the sensing system and the unmanned system.

In some embodiments, the perception module 1130 includes: the data preprocessing strategy adjustment module is used for adjusting a point cloud screening strategy in the data preprocessing strategy to be based on the perception range parameters in the dynamic perception parameter set: clearing original point cloud data, which are located outside a sensing range corresponding to the sensing range parameter, in the sensing data to obtain target point cloud data; the sensing data includes: the original point cloud data are data acquired by a laser radar or a millimeter wave radar of the current vehicle.

In some embodiments, the data preprocessing policy adjustment module is further configured to adjust an image resolution adjustment policy in the data preprocessing policy to: the original resolution of the original image data in the sensing data is adjusted to the sensing resolution parameter, and target image data is obtained; the original image data are data acquired by the camera device of the current vehicle.

In some embodiments, the perception module 1130 further includes: the dynamic perception adjusting module is used for adjusting an image feature view conversion network and a point cloud feature extraction network of the dynamic perception model based on the dynamic perception parameter set; the image feature view conversion network is used for converting the two-dimensional image features output by the image feature extraction network of the dynamic perception model into three-dimensional camera view cone point cloud features; the input data of the image feature extraction network is target image data, and the target image data is preprocessed original image data; the point cloud feature extraction network is used for carrying out feature extraction on target point cloud data to obtain target point cloud features, the target point cloud data are preprocessed original point cloud data, the original image data and the original point cloud data are both the sensing data, the original image data are data acquired by a camera device of a current vehicle, and the original point cloud data are data acquired by a laser radar or a millimeter wave radar of the current vehicle.

In some embodiments, the dynamic perception adjustment module is specifically configured to adjust a depth estimation sub-network of the image feature view conversion network based on the depth estimation range parameter and the depth estimation point interval parameter in the dynamic perception parameter set;

In some embodiments, the dynamic sensing adjustment module is specifically configured to determine a difference between a maximum value and a minimum value in the depth estimation range parameter as an intermediate value;

In some embodiments, the dynamic sensing adjustment module is specifically configured to obtain a target height and a target width of the image feature based on the sensing resolution parameter;

In some embodiments, the dynamic sensing adjustment module is specifically configured to determine a ratio between the first image resolution and a target downsampling multiple as the target height, where the target downsampling multiple is a downsampling multiple of the camera cone point cloud feature extraction sub-network;

In some embodiments, the dynamic sensing adjustment module is specifically configured to adjust a target pooling shape of the camera view cone point cloud feature pooling sub-network based on the sensing range parameter, where the target pooling shape refers to a shape of the pooled camera view cone point cloud feature.

In some embodiments, the dynamic sensing adjustment module is specifically configured to divide a sensing distance in each sensing direction in the sensing range parameter by a preset feature grid size to obtain a sensing pooling shape;

and adjusting the target pooling shape based on the perception pooling shape.

In some embodiments, the dynamic sensing adjustment module is specifically configured to adjust a target voxel size of the point cloud feature extraction network based on the point cloud resolution, where the target voxel size is a voxel size adjustment basis when the point cloud feature extraction network performs a voxelization operation.

In some embodiments, the perception module 1130 includes: the execution module is used for preprocessing the sensing data by utilizing the adjusted data preprocessing strategy to obtain preprocessed sensing data;

In some embodiments, the execution module is specifically configured to input the target point cloud data into a point cloud feature extraction network of the adjusted dynamic perception model, and perform voxelization operation and point cloud feature extraction to obtain a target point cloud feature;

In some embodiments, the execution module is specifically configured to input the target image data into the image feature extraction network of the adjusted dynamic perception model, and perform image feature extraction to obtain an initial image feature;

Determining a product between the scaled image feature and a target translation parameter as the two-dimensional image feature, the target translation parameter being a translation parameter corresponding to a resolution of a current scaled image feature; the target scaling parameters and the target translation parameters corresponding to each resolution are determined by the dynamic perception model after repeated iterative learning.

In some embodiments, the execution module is specifically configured to input the two-dimensional image feature into a feature optimization network of a dynamic perception model, perform feature optimization, and obtain an optimized two-dimensional image feature;

Referring to fig. 12, the present embodiment further provides a dynamic perception model training method, which includes:

S1210: obtaining a training set, the training set comprising: a plurality of sets of training data, the training data comprising: a dynamic perception associated information sample, a sensing data sample and a perception real result; the dynamic perception associated information sample comprises: a vehicle exterior environment information sample, a vehicle operating state information sample, and a processor load information sample of a vehicle perception system.

S1220: obtaining a dynamic perception parameter set sample based on the dynamic perception associated information sample and a preset dynamic perception parameter set matching rule, wherein the dynamic perception associated information sample corresponds to the dynamic perception parameter set sample, and the dynamic perception parameter set sample comprises: a perceptual range parameter, a perceptual resolution parameter, a depth estimation range parameter, and a depth estimation point interval parameter.

S1230: and adjusting a data preprocessing strategy and a dynamic perception model of the perception system based on the dynamic perception parameter set sample.

S1240: and preprocessing the sensing data sample by utilizing the adjusted data preprocessing strategy to obtain a preprocessed sensing data sample.

S1250: and inputting the preprocessed sensing data sample into the adjusted dynamic sensing model, and performing sensing task processing to obtain a sensing prediction result.

S1260: and training the dynamic perception model based on the difference between the perception prediction result and the corresponding perception real result to obtain a trained dynamic perception model. The dynamic perception model training method in the embodiment can better improve the dynamic perception capability of the dynamic perception model. In addition, the trained dynamic perception model has the capability of reducing the calculation burden of the perception system, and the instantaneity and the robustness of the perception system and the unmanned system can be well ensured.

In some embodiments, the sensing data sample comprises: the system comprises an original point cloud data sample and an original image data sample, wherein the original point cloud data sample is data acquired by a laser radar or a millimeter wave radar of a vehicle, and the original image data sample is data acquired by a camera device of the vehicle; the preprocessed sensor data sample comprises: the target point cloud data sample refers to a preprocessed original point cloud data sample, and the target image data sample refers to a preprocessed original image data sample.

Further, the step of inputting the preprocessed sensing data sample into the adjusted dynamic sensing model to perform sensing task processing to obtain a sensing prediction result includes:

S12501: and inputting the target point cloud data sample into a point cloud feature extraction network of the adjusted dynamic perception model, and performing voxelization operation and point cloud feature extraction to obtain a target point cloud feature sample.

S12502: and inputting the target image data sample into an image feature extraction network of the dynamic perception model, and extracting image features to obtain a two-dimensional image feature sample.

S12503: and performing image feature conversion based on the two-dimensional image feature sample and the image feature view conversion network of the adjusted dynamic perception model to obtain a three-dimensional camera view cone point cloud feature sample.

S12504: and inputting the target point cloud characteristic sample and the camera cone point cloud characteristic sample into a characteristic fusion network of a dynamic perception model to obtain a fusion characteristic sample.

S12505: and inputting the fusion characteristic sample into a perception task processing layer of a dynamic perception model, and performing perception task processing to obtain the perception prediction result.

In some embodiments, the step of obtaining the three-dimensional camera cone point cloud feature sample based on the two-dimensional image feature sample and the image feature view conversion network of the adjusted dynamic perception model includes:

s12503a: inputting the two-dimensional image feature sample into a feature optimization network of a dynamic perception model to perform feature optimization, and obtaining an optimized two-dimensional image feature sample.

Under the condition that the optimized two-dimensional image feature sample is obtained, the feature optimization network is obtained based on the difference between the optimized two-dimensional image feature sample and a preset real optimization sample, so that a better feature optimization network is obtained.

S12503b: and inputting the optimized two-dimensional image characteristic sample into the image characteristic view conversion network of the adjusted dynamic perception model to perform image characteristic conversion so as to obtain the camera view cone point cloud characteristic sample.

In some embodiments, after the step of obtaining the optimized two-dimensional image feature sample, the method further comprises:

firstly, obtaining the image feature gradient of the optimized two-dimensional image feature sample.

Secondly, determining the product between the image characteristic gradient and the corresponding gradient balance factor as a target characteristic gradient, wherein the gradient balance factor is obtained in the following way: obtaining the sensing resolution parameters corresponding to the image feature gradients at present, wherein the sensing resolution parameters comprise: a first image resolution and a second image resolution, the first image resolution being a resolution in an image height direction, the second image resolution being a resolution in an image width direction; determining a product between the first image resolution and the second image resolution as a to-be-determined value; and determining the ratio between 1 and the to-be-determined value as the gradient balance factor.

Specifically, denotes a gradient balance factor set, and/() denotes a gradient balance factor of the i-th resolution. Where/> denotes the first image resolution, i.e. the number of pixels in the image height direction or image height dimension. And/> denotes the second image resolution, i.e. the number of pixels in the image width direction or image width dimension.

And finally, training the dynamic perception model based on the difference between the target feature gradient and a preset standard feature gradient. It should be noted that, because there are different image resolutions, the training difficulty of the dynamic perception model is increased, and the gradient values of the two-dimensional image feature samples with different resolutions are different. In order to solve the problem, the embodiment sets corresponding gradient balance factors for the two-dimensional image feature samples with different resolutions by adding a multi-path gradient balance layer, so that the image feature gradients of the two-dimensional image feature samples are re-weighted in the back propagation process, and a better training effect is realized.

1 fig. 13 is a schematic diagram of a method of multi-path gradient balancing, as shown in fig. 13, in which image feature gradients (such as image feature gradient (resolution 1) and image feature gradient (resolution 2) image feature gradient (resolution r)) of two-dimensional image feature samples with different resolutions are multiplied by corresponding gradient balancing factors (such as gradient balancing factor (resolution 1) and gradient balancing factor (resolution 2) gradient balancing factor (resolution r)) respectively, so as to obtain corresponding target feature gradients (such as target feature gradient (resolution 1) and target feature gradient (resolution 2) target feature gradient (resolution r)).

In some embodiments, further comprising: acquiring a plurality of groups of initial image feature sample sets and real normalized image features corresponding to the initial image feature samples in the initial image feature sample sets one by one, wherein the initial image feature sample sets correspond to the resolutions of the initial image feature samples, and the initial image feature sample sets comprise a plurality of initial image feature samples with the same resolution;

Acquiring the average value of the initial image feature samples in each initial image feature sample set;

Acquiring variances of the initial image feature samples in each initial image feature sample set based on the mean value of the initial image feature samples;

Based on the variance of each initial image characteristic sample set and a preset normalization formula, respectively carrying out normalization processing on each initial image characteristic sample to obtain normalized image characteristic samples;

Determining a product between the normalized image feature sample and a target scaling parameter sample as a scaled image feature sample, the target scaling parameter sample referring to a scaling parameter corresponding to the resolution of the current normalized image feature sample;

determining a product between the scaled image feature sample and a target translation parameter sample as the two-dimensional image feature sample, wherein the target translation parameter sample refers to a translation parameter corresponding to the resolution of the current scaled image feature sample;

And based on the difference between the two-dimensional image characteristic sample and the corresponding real normalized image characteristic, iteratively updating the target scaling parameter sample and the target translation parameter sample until the target scaling parameter sample and the target translation parameter sample with better quality are obtained.

The dynamic perception model training system provided by the invention is described below, and the dynamic perception model training system described below and the dynamic perception model training method described above can be referred to correspondingly.

Referring to fig. 14, the dynamic perception model training system provided in this embodiment includes:

A training set obtaining module 1410, configured to obtain a training set, where the training set includes: a plurality of sets of training data, the training data comprising: a dynamic perception associated information sample, a sensing data sample and a perception real result; the dynamic perception associated information sample comprises: a vehicle external environment information sample, a vehicle running state information sample, and a processor load information sample of a vehicle sensing system;

the dynamic sensing parameter set sample obtaining module 1420 is configured to obtain a dynamic sensing parameter set sample based on the dynamic sensing correlation information sample and a preset dynamic sensing parameter set matching rule, where the dynamic sensing correlation information sample corresponds to the dynamic sensing parameter set sample, and the dynamic sensing parameter set sample includes: a perception range parameter, a perception resolution parameter, a depth estimation range parameter, and a depth estimation point interval parameter;

an adjustment module 1430, configured to adjust a data preprocessing strategy and a dynamic perception model of the perception system based on the dynamic perception parameter set sample;

The preprocessing module 1440 is configured to preprocess the sensing data sample by using the adjusted data preprocessing policy, so as to obtain a preprocessed sensing data sample;

The task processing module 1450 is configured to input the preprocessed sensor data sample into the adjusted dynamic sensing model, perform sensing task processing, and obtain a sensing prediction result;

the training module 1460 is configured to train the dynamic sensing model based on a difference between the sensing prediction result and the corresponding sensing real result, to obtain a trained dynamic sensing model.

In some embodiments, the task processing module 1450 is specifically configured to input the target point cloud data sample into a point cloud feature extraction network of the adjusted dynamic perception model, perform voxelization operation and point cloud feature extraction, and obtain a target point cloud feature sample;

In some embodiments, the task processing module 1450 is further specifically configured to input the two-dimensional image feature sample into a feature optimization network of a dynamic perception model, perform feature optimization, and obtain an optimized two-dimensional image feature sample;

In some embodiments, further comprising: the gradient balancing module is used for acquiring the image characteristic gradient of the optimized two-dimensional image characteristic sample;

Fig. 15 illustrates a physical structure diagram of an electronic device, as shown in fig. 15, which may include: processor 1510, communication interface (Communications Interface) 1520, memory 1530, and communication bus 1540, wherein processor 1510, communication interface 1520, memory 1530 communicate with each other via communication bus 1540. The processor 1510 may invoke logic instructions in the memory 1530 to perform a vehicle dynamics awareness method or a dynamic awareness model training method, the vehicle dynamics awareness method comprising: collecting dynamic perception associated information of a current vehicle, wherein the dynamic perception associated information comprises the following steps: vehicle external environment information, vehicle running state information and processor load information of a sensing system; based on the dynamic perception associated information and a preset dynamic perception parameter set matching rule, a dynamic perception parameter set is obtained, the dynamic perception associated information corresponds to the dynamic perception parameter set, and the dynamic perception parameter set comprises: a perception range parameter, a perception resolution parameter, a depth estimation range parameter, and a depth estimation point interval parameter; based on the dynamic sensing parameter set, a data preprocessing strategy of the sensing system and a dynamic sensing model are adjusted to obtain a sensing result, the data preprocessing strategy is used for preprocessing sensing data acquired by the sensing system, and the dynamic sensing model is used for processing sensing tasks based on the preprocessed sensing data to obtain the sensing result. The dynamic perception model training method comprises the following steps: obtaining a training set, wherein the training set comprises: a plurality of sets of training data, the training data comprising: a dynamic perception associated information sample, a sensing data sample and a perception real result; the dynamic perception association information sample comprises: a vehicle external environment information sample, a vehicle running state information sample, and a processor load information sample of a vehicle sensing system; based on the dynamic perception associated information sample and a preset dynamic perception parameter set matching rule, a dynamic perception parameter set sample is obtained, the dynamic perception associated information sample corresponds to the dynamic perception parameter set sample, and the dynamic perception parameter set sample comprises: a perception range parameter, a perception resolution parameter, a depth estimation range parameter, and a depth estimation point interval parameter; based on the dynamic perception parameter set sample, adjusting a data preprocessing strategy and a dynamic perception model of a perception system; preprocessing the sensing data sample by utilizing the adjusted data preprocessing strategy to obtain a preprocessed sensing data sample; inputting the preprocessed sensing data sample into the adjusted dynamic sensing model, and performing sensing task processing to obtain a sensing prediction result; based on the difference between the perception prediction result and the corresponding perception real result, training the dynamic perception model to obtain a trained dynamic perception model.

Further, the logic instructions in the memory 1530 described above may be implemented in the form of software functional units and may be stored on a computer readable storage medium when sold or used as a stand alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a read-only memory (ROM), a random access memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the vehicle dynamic perception method or the dynamic perception model training method provided by the above methods.

The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of vehicle dynamics awareness, comprising:

Based on the dynamic sensing parameter set, a data preprocessing strategy of the sensing system and a dynamic sensing model are adjusted to obtain a sensing result, wherein the data preprocessing strategy is used for preprocessing sensing data acquired by the sensing system, and the dynamic sensing model is used for performing sensing task processing based on the preprocessed sensing data to obtain the sensing result;

The step of adjusting the data preprocessing strategy of the sensing system based on the dynamic sensing parameter set comprises:

based on the sensing range parameters in the dynamic sensing parameter set, adjusting a point cloud screening strategy in the data preprocessing strategy to be: clearing original point cloud data, which are located outside a sensing range corresponding to the sensing range parameter, in the sensing data to obtain target point cloud data;

based on the perceived resolution parameters in the dynamic perceived parameter set, adjusting an image resolution adjustment strategy in the data preprocessing strategy to: the original resolution of the original image data in the sensing data is adjusted to the sensing resolution parameter, and target image data is obtained;

the step of adjusting the dynamic perception model of the perception system based on the set of dynamic perception parameters comprises:

based on the dynamic perception parameter set, adjusting an image feature view conversion network and a point cloud feature extraction network of the dynamic perception model;

Based on the dynamic perception parameter set, the step of adjusting the image feature view conversion network of the dynamic perception model comprises the following steps:

2. The vehicle dynamic sensing method according to claim 1, wherein the sensing data includes: the original point cloud data are data acquired by a laser radar or a millimeter wave radar of the current vehicle.

3. The vehicle dynamic sensing method according to claim 2, wherein the raw image data is data acquired by a camera device of a current vehicle.

4. The vehicle dynamic sensing method according to claim 1, wherein the image feature view conversion network is configured to convert two-dimensional image features output by the image feature extraction network of the dynamic sensing model into three-dimensional camera cone point cloud features; the input data of the image feature extraction network is target image data, and the target image data is preprocessed original image data; the point cloud feature extraction network is used for carrying out feature extraction on target point cloud data to obtain target point cloud features, the target point cloud data are preprocessed original point cloud data, the original image data and the original point cloud data are both the sensing data, the original image data are data acquired by a camera device of a current vehicle, and the original point cloud data are data acquired by a laser radar or a millimeter wave radar of the current vehicle.

5. The method of claim 1, wherein the step of adjusting the depth estimation sub-network of the image feature view conversion network based on the depth estimation range parameter and the depth estimation point interval parameter in the dynamic sensing parameter set comprises:

6. The vehicle dynamic sensing method according to claim 1, wherein the step of adjusting the camera cone point cloud feature extraction sub-network of the image feature view conversion network based on the sensing resolution parameters in the dynamic sensing parameter set comprises:

7. The vehicle dynamic sensing method according to claim 6, wherein the sensing resolution parameter includes: a first image resolution and a second image resolution, the first image resolution being a resolution in an image height direction, the second image resolution being a resolution in an image width direction;

8. The vehicle dynamic sensing method according to claim 6, wherein the step of obtaining the number of target cone points in the depth direction based on the depth estimation range parameter and the depth estimation point interval parameter comprises:

9. The method of claim 1, wherein the step of adjusting the camera view cone point cloud feature pooling sub-network of the image feature view conversion network based on the perception range parameters in the dynamic perception parameter set comprises:

10. The method of claim 9, wherein the step of adjusting the target pooling shape of the camera cone point cloud feature pooling sub-network based on the perception range parameter comprises:

and adjusting the target pooling shape based on the perception pooling shape.

11. The vehicle dynamic sensing method of claim 4, wherein the sensing resolution parameter comprises a point cloud resolution, the point cloud resolution pointing to a cloud voxelized size; based on the dynamic perception parameter set, the step of adjusting the point cloud feature extraction network of the dynamic perception model comprises the following steps:

12. The method of claim 1, wherein the step of adjusting the data preprocessing strategy and the dynamic perception model of the perception system based on the set of dynamic perception parameters to obtain a perception result comprises:

13. The vehicle dynamic sensing method according to claim 12, wherein the sensing data includes: the system comprises original point cloud data and original image data, wherein the original point cloud data are data acquired by a laser radar or a millimeter wave radar of a current vehicle, and the original image data are data acquired by a camera device of the current vehicle; the preprocessed sensor data includes: target point cloud data and target image data, wherein the target point cloud data refers to preprocessed original point cloud data, and the target image data refers to preprocessed original image data;

14. The method of claim 13, wherein the step of inputting the target image data into an image feature extraction network of a dynamic perception model to perform image feature extraction, and obtaining two-dimensional image features comprises:

The mathematical expression of the normalization formula is:

15. The method of claim 13, wherein the step of performing image feature conversion based on the two-dimensional image feature and the image feature view conversion network of the adjusted dynamic perception model to obtain three-dimensional camera cone point cloud features comprises:

16. A vehicle dynamics awareness system, comprising:

The sensing module is used for adjusting a data preprocessing strategy of the sensing system and a dynamic sensing model based on the dynamic sensing parameter set to obtain a sensing result, wherein the data preprocessing strategy is used for preprocessing sensing data acquired by the sensing system, and the dynamic sensing model is used for performing sensing task processing based on the preprocessed sensing data to obtain the sensing result;

the perception module further comprises:

The data preprocessing strategy adjustment module is used for adjusting a point cloud screening strategy in the data preprocessing strategy to be based on the perception range parameters in the dynamic perception parameter set: clearing original point cloud data, which are located outside a sensing range corresponding to the sensing range parameter, in the sensing data to obtain target point cloud data;

the data preprocessing strategy adjustment module is further configured to adjust an image resolution adjustment strategy in the data preprocessing strategy based on the perceived resolution parameters in the dynamic perceived parameter set to: the original resolution of the original image data in the sensing data is adjusted to the sensing resolution parameter, and target image data is obtained;

the perception module further comprises:

the dynamic perception adjusting module is used for adjusting an image feature view conversion network and a point cloud feature extraction network of the dynamic perception model based on the dynamic perception parameter set;

the dynamic perception adjusting module is specifically configured to adjust a depth estimation sub-network of the image feature view conversion network based on the depth estimation range parameter and the depth estimation point interval parameter in the dynamic perception parameter set;

17. A method for training a dynamic perception model, comprising:

Training the dynamic perception model based on the difference between the perception prediction result and the corresponding perception real result to obtain a trained dynamic perception model;

based on the dynamic perception parameter set samples, the step of adjusting the data preprocessing strategy of the perception system comprises the following steps:

Based on the sensing range parameters in the dynamic sensing parameter set sample, adjusting a point cloud screening strategy in the data preprocessing strategy to be: clearing original point cloud data, which are located outside a sensing range corresponding to the sensing range parameter, in the sensing data to obtain target point cloud data;

Based on the perceived resolution parameters in the dynamic perceived parameter set samples, adjusting an image resolution adjustment strategy in the data preprocessing strategy to: the original resolution of the original image data in the sensing data is adjusted to the sensing resolution parameter, and target image data is obtained;

Based on the dynamic perception parameter set samples, the step of adjusting the dynamic perception model of the perception system comprises:

Based on the dynamic perception parameter set sample, adjusting an image feature view conversion network and a point cloud feature extraction network of the dynamic perception model;

Based on the dynamic perception parameter set sample, the step of adjusting the image feature view conversion network of the dynamic perception model comprises the following steps:

adjusting a depth estimation sub-network of the image feature view conversion network based on the depth estimation range parameter and the depth estimation point interval parameter in the dynamic perception parameter set sample;

Adjusting a camera view cone point cloud feature extraction sub-network of the image feature view conversion network based on the perceived resolution parameter, the depth estimation range parameter and the depth estimation point interval parameter in the dynamic perceived parameter set sample;

And adjusting a camera view cone point cloud feature pooling sub-network of the image feature view conversion network based on the perception range parameters in the dynamic perception parameter set sample.

18. The method of claim 17, wherein the sensing data samples comprise: the system comprises an original point cloud data sample and an original image data sample, wherein the original point cloud data sample is data acquired by a laser radar or a millimeter wave radar of a vehicle, and the original image data sample is data acquired by a camera device of the vehicle; the preprocessed sensor data sample comprises: the target point cloud data sample and the target image data sample, wherein the target point cloud data sample refers to a preprocessed original point cloud data sample, and the target image data sample refers to a preprocessed original image data sample;

19. The method for training a dynamic perception model according to claim 18, wherein the step of performing image feature conversion based on the two-dimensional image feature sample and the image feature view conversion network of the adjusted dynamic perception model to obtain a three-dimensional camera cone point cloud feature sample comprises:

20. The method of claim 19, further comprising, after the step of obtaining the optimized two-dimensional image feature samples:

21. A dynamic perception model training system, comprising:

The training module is used for training the dynamic perception model based on the difference between the perception prediction result and the corresponding perception real result to obtain a trained dynamic perception model;

22. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the vehicle dynamic perception method of any one of claims 1 to 15 or the dynamic perception model training method of any one of claims 17 to 20 when the program is executed by the processor.

23. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the vehicle dynamic perception method of any of claims 1 to 15, or the dynamic perception model training method of any of claims 17 to 20.