CN114140765B - Obstacle sensing method and device and storage medium - Google Patents

Abstract

The present application discloses an obstacle sensing method, device and storage medium, which are used to reduce the false detection and false detection rate of obstacles and improve the detection accuracy. The obstacle perception method disclosed in the present application includes: obtaining the original point cloud and camera picture at the same time; obtaining the calibration internal and external parameters of projection transformation; performing semantic segmentation on the original point cloud to obtain a second point cloud; Semantic segmentation is performed on the camera picture to obtain a second picture; according to the internal parameters and the external parameters, the original point cloud is projected to the second picture to obtain a third point cloud. Each point includes second semantic category information corresponding to the second picture; for the second semantic category information in the third point cloud, the first semantic category information in the second point cloud and the original point After the feature information of the cloud is voxelized, it is input into the adaptive attention mechanism network for learning, and the weighted semantic information is obtained; according to the weighted semantic information, the obstacle target is detected. The present application also provides an obstacle sensing device and a storage medium.

Description

Obstacle sensing method, device and storage medium

technical field

The present application relates to the field of automatic driving, and in particular, to an obstacle sensing method, device and storage medium.

Background technique

With the continuous development of autonomous driving technology, various sensors are an important part of the autonomous driving system. The environmental perception part of the automatic driving system usually needs to obtain a large amount of surrounding environment information to ensure the correct understanding and corresponding decision-making of the automatic vehicle's surrounding environment. However, there are limitations in using a single sensor for sensing. On the one hand, a single sensing device may have a detection blind spot due to the limitation of the sensor installation location; on the other hand, each sensor has its own unique defects.

It can be seen that using a single sensor for obstacle perception has the problem of low recognition accuracy.

SUMMARY OF THE INVENTION

In view of the above technical problems, embodiments of the present application provide an obstacle sensing method, device, and storage medium, so as to improve the accuracy of obstacle sensing.

In a first aspect, an obstacle sensing method provided by an embodiment of the present application includes:

Get the original point cloud and camera image at the same moment;

Obtain the calibration internal and external parameters of the projection transformation;

Semantically segment the original point cloud to obtain a second point cloud;

Semantically segment the camera picture to obtain a second picture;

According to the internal reference and the external reference, the original point cloud is projected to the second picture to obtain a third point cloud, where each point in the third point cloud includes the corresponding second semantic category information;

After voxelizing the second semantic category information in the third point cloud, the first semantic category information in the second point cloud and the feature information of the original point cloud, input the adaptive attention mechanism network learning in the process to obtain weighted semantic information;

Detecting an obstacle target according to the weighted semantic information;

Wherein, the second point cloud and the third point cloud include obstacle category information.

Preferably, the learning in the input adaptive attention mechanism network includes:

Learning the local features in the adaptive attention mechanism network to obtain the learned local features V _i ;

The global feature is learned in the adaptive attention mechanism network, and the learned global feature V _global is obtained;

The learned global feature V _global is spliced onto each local feature V _i to obtain an enhanced feature V _gl .

Preferably, the detecting an obstacle target according to the first semantic information includes:

The first semantic information is input into the target detector to detect the obstacle target.

Preferably, the obtaining the original point cloud and the camera picture at the same moment includes:

Software synchronization or hardware synchronization of point cloud and camera;

Get the original point cloud and camera image at the same moment.

Preferably, the original point cloud is semantically segmented, and obtaining the second point cloud includes:

The original point cloud is input into the point cloud semantic segmentation network to obtain a second point cloud.

Preferably, the semantic segmentation of the camera picture to obtain the second picture includes:

The camera picture is input into the picture semantic segmentation network to obtain the second picture.

Before performing voxelization on the second semantic category information in the third point cloud, the first semantic category information in the second point cloud, and the feature information of the original point cloud, the method further includes:

Convert the first semantic category information and the second semantic category information into One-Hot encoding format.

Preferably, performing the projection of the original point cloud to the second picture includes:

Projection is performed according to the following formula:

P′=Proj(K,M,P),

Among them, Proj is the projection matrix processing process;

K is the internal parameter matrix of the camera;

M is the extrinsic parameter matrix from camera to lidar;

P is the lidar point cloud collection;

P' is the lidar point cloud projected to the camera coordinate system.

The local features are learned in the adaptive attention mechanism network, and the learned local features V _i include:

The learning of local features is carried out according to the following formula:

V _i =max _{i =1,2,...,N} {MLP _l (pi )},

Among them, V _i is the feature in the i-th three-dimensional voxel lattice obtained by learning;

p _i is the ith point in the spatial point cloud;

MLP _l ( _pi ) is a local feature multilayer perceptron;

max is the maximum pooling operation for all points in a voxel;

C ₁ is the number of channels of the local feature map;

N is the number of voxel grids.

Preferably, the global feature is learned in an adaptive attention mechanism network, and the learned global feature V _global includes:

The learning of global features is carried out according to the following formula:

V _global =max _i=1,2,...,N {MLP _g (V _i )}

MLP _g (V _i ) is a global feature multilayer perceptron;

max is the maximum pooling operation for all voxels;

C ₂ is the number of channels in the entire feature map;

N is the number of voxels;

V _i is the learned feature in the i-th voxel grid.

The weighted semantic information includes:

The weighted semantic information is obtained according to the following formula:

Among them, P _a,s and P _a,t are the weighted semantic information;

P _2D is the second semantic information;

P _3D is the first semantic information;

MLP _att is a multi-layer perceptron;

σ is the sigmoid activation function.

Using the obstacle sensing method provided by the present invention, the lidar sensor and the camera sensor are integrated, the advantages of different sensors are used, and the shortcomings of the respective sensors are supplemented, and the perception and recognition accuracy of point cloud target detection is improved. At the same time, in this solution, the deep learning network combining the semantic segmentation information of 3D point cloud and the semantic segmentation information of 2D image is used, and the semantic information of different sensors is used to reduce the false detection and false detection rate of obstacles.

In a second aspect, an embodiment of the present application further provides an obstacle sensing device, including:

The image acquisition module is configured to acquire the camera image, and to acquire the calibration internal and external parameters of the projection transformation;

A point cloud acquisition module, configured to acquire the original point cloud;

The image semantic segmentation module is configured to perform semantic segmentation on the camera image to obtain a second image;

a point cloud semantic segmentation module, configured to perform semantic segmentation on the original point cloud to obtain a second point cloud;

The image semantic projection module is configured to perform the projection of the original point cloud to the second image according to the internal parameters and the external parameters to obtain a third point cloud, each of which is in the third point cloud. The point includes second semantic category information corresponding to the second picture;

a semantic fusion module, configured to perform voxelization on the second semantic category information in the third point cloud, the first semantic category information in the second point cloud and the feature information of the original point cloud , input into the adaptive attention mechanism network for learning, and obtain weighted semantic information;

The obstacle perception module is configured to detect the obstacle target according to the weighted semantic information;

Wherein, the second point cloud and the second picture include obstacle category information.

In a third aspect, an embodiment of the present application further provides an obstacle sensing device, including: a memory, a processor, and a user interface;

the memory for storing computer programs;

the user interface for interacting with the user;

The processor is configured to read the computer program in the memory, and when the processor executes the computer program, the obstacle sensing method provided by the present invention is implemented.

In a fourth aspect, an embodiment of the present application further provides a processor-readable storage medium, where the processor-readable storage medium stores a computer program, and when the processor executes the computer program, the obstacle perception provided by the present invention is implemented method.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a schematic diagram of obstacle perception provided by an embodiment of the present application;

FIG. 2 is a schematic flowchart of an adaptive attention mechanism provided by an embodiment of the present application;

FIG. 3 is a schematic structural diagram of an obstacle sensing device provided by an embodiment of the present application;

FIG. 4 is a schematic structural diagram of another obstacle sensing device provided by an embodiment of the present application.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. . Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The following is an explanation of some words that appear in the text:

1. The term "and/or" in the embodiment of the present invention describes the association relationship of the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist simultaneously, and a separate relationship exists. There are three cases of B. The character "/" generally indicates that the associated objects are an "or" relationship.

2. The term "plurality" in the embodiments of the present application refers to two or more than two, and other quantifiers are similar.

3. One-Hot encoding format, One-Hot encoding, also known as one-bit effective encoding, is often used in classification prediction and is usually presented in the form of a binary vector. First, map the category to which the object belongs to an integer value, and then convert it to a binary code, that is, the value of the category dimension is 1, and the value of the remaining dimension is 0.

4. Voxelization refers to dividing the three-dimensional point cloud according to the grid of the same resolution size (such as 0.75m*0.75m*0.75), and putting it into different points according to the different spatial positions of each point in the point cloud. within the voxel grid.

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

It should be noted that the display order of the embodiments of the present application only represents the sequence of the embodiments, and does not represent the advantages and disadvantages of the technical solutions provided by the embodiments.

Example 1

Referring to FIG. 1, a schematic diagram of an obstacle sensing method provided by an embodiment of the present application, as shown in FIG. 1, the method includes steps S101 to S107:

S101. Obtain the original point cloud and the camera picture at the same moment;

In the embodiment of the present invention, the original point cloud is a three-dimensional point cloud, which can be acquired by a laser radar. The camera picture is obtained through the camera. If multiple cameras are installed, the camera pictures of the multiple cameras will be obtained at the same time. The acquisition time of the original point cloud is the same as the acquisition time of the camera image, that is, the original point cloud and the camera image are acquired at the same moment.

As a preferred example, in this step, the original point cloud and the camera picture may not be at the same time, but the difference between the time when the original point cloud is obtained and the time when the camera picture is obtained is within a predetermined time difference range, and the predetermined time difference range is achieved. Make sure, for example, that there is a difference of 0.001 seconds between the time when the original point cloud was acquired and the time when the camera image was acquired.

As a preferred example, in order to obtain the original point cloud and camera internal parameters at the same moment, soft synchronization or hardware synchronization may be performed on the point cloud and the camera. Soft synchronization refers to providing the same time source to different sensors, and the sensors stamp the recorded data with time stamps; hardware synchronization refers to using hardware triggers to record time directly through physical signals, such as PPS timing, after triggering different sensors.

S102, obtaining the calibration internal parameters and external parameters of the projection transformation;

In this step, the calibration internal parameters and external parameters of the projection transformation corresponding to the camera and the lidar are obtained.

In the embodiment of the present invention, the internal parameters are the internal parameters of the calibrated camera; the external parameters are the external parameters of the camera and the laser radar; the internal parameters and the external parameters are used for the projection transformation of the point cloud.

Preferably, the internal parameters include but are not limited to: distortion coefficient, focal length, pixel size, etc.; external parameters include but are not limited to: rotation, translation matrix, and the like.

It should be noted that, in the embodiment of the present invention, the internal reference and the external reference are pre-calibrated and stored.

S103, performing semantic segmentation on the original point cloud to obtain a second point cloud;

As a preferred example, the original point cloud is input into the point cloud semantic segmentation network.

In this step, a frame of point cloud is sent to the point cloud semantic segmentation network, and a second point cloud containing fine-grained semantic information will be obtained, that is, the second point cloud includes the first semantic information.

It should be noted that the fine-grained semantic information refers to that the category information of each point is clear and is not disturbed by other external conditions, such as the influence of internal and external parameters.

S104, performing semantic segmentation on the camera picture to obtain a second picture;

In this step, the camera picture is input into the picture semantic segmentation network to obtain the second picture.

It should be noted that, as a preferred example, in the above steps S103 and S104, the semantic segmentation network may be a Cylinder3D network or the like.

S105. Perform the projection of the original point cloud to the second picture according to the internal reference and the external reference to obtain a third point cloud, where each point in the third point cloud includes the second picture the corresponding second semantic category information;

In this step, the projection method of the original point cloud to the second picture may be:

Projection is performed according to the following formula:

P′=Proj(K,M,P),

Among them, Proj is the projection matrix processing process;

K is the internal parameter matrix of the camera;

M is the extrinsic parameter matrix from camera to lidar;

P is the lidar point cloud collection;

P' is the lidar point cloud projected to the camera coordinate system.

For example, the original point cloud is P, and the second image is K. After projection according to the above formula 1, the third point cloud P' is obtained.

After the above steps S101 to S105, each point in the point cloud has two kinds of semantic information obtained, namely the first semantic information from the original point cloud and the second semantic information from the projected image of the point cloud. It should be noted that, as a preferred example, the obtained first semantic information and second semantic information may also be in One-Hot format.

S106. After voxelizing the second semantic category information in the third point cloud, the first semantic category information in the second point cloud, and the feature information of the original point cloud, input adaptive attention Learning in the mechanism network to obtain weighted semantic information;

In this step, learning in the input adaptive attention mechanism network includes:

Learning the local features in the adaptive attention mechanism network to obtain the learned local features V _i ;

The global feature is learned in the adaptive attention mechanism network, and the learned global feature V _global is obtained;

The learned global feature V _global is spliced onto each local feature V _i to obtain an enhanced feature V _gl .

As a preferred example, the local features are learned in the attention mechanism network, and the learned local features _Vi include:

The learning of local features is carried out according to the following formula:

V _i =max _{i =1,2,...,N} {MLP _l (pi )},

Among them, V _i is the feature in the i-th three-dimensional voxel lattice obtained by learning;

p _i is the ith point in the spatial point cloud;

MLP _l ( _pi ) is a local feature multilayer perceptron;

max is the maximum pooling operation for all points in a voxel;

C ₁ is the number of channels of the local feature map;

N is the number of voxel grids.

The global feature is learned in the adaptive attention mechanism network, and the learned global feature V _global includes:

The learning of global features is carried out according to the following formula:

V _global =max _i=1,2,...,N {MLP _g (V _i )}

MLP _g (V _i ) is a global feature multilayer perceptron;

max is the maximum pooling operation for all voxels;

C ₂ is the number of channels in the entire feature map;

N is the number of voxels;

V _i is the learned feature in the i-th voxel grid.

As a preferred example, the weighted semantic information is obtained according to the following formula:

Among them, P _a,s and P _a,t are the weighted semantic information;

P _2D is the second semantic information;

P _3D is the first semantic information;

MLP _att is a multi-layer perceptron;

σ is the sigmoid activation function.

As a preferred example, the processing process of this step is shown in FIG. 2 . In FIG. 2 , the input point cloud is the original point cloud, the 2D semantic information is the data after the second semantic information is converted into the One-Hot format, and the 3D semantic information is the data after the first semantic information is converted into the One-Hot format. The process is as follows:

After splicing the 2D and 3D semantic information converted into One-Hot to the original point cloud feature information, if the number of predicted categories required is m, then each point contains 2m of semantic category information divided by 3D and 2D respectively. It is spliced with the original data information of the point cloud, such as XYZ, to obtain an N×(2m+3) dimensional feature vector. After voxelization, it is input into the adaptive attention mechanism network that combines local features and global features to obtain weighted weights. Then, each voxel grid belongs to the category feature, and finally it is input into the target detection network.

S107, detecting an obstacle target according to the weighted semantic information;

In this step, the weighted semantic information is input into the target detector to detect the obstacle target.

Using the obstacle sensing method provided by the present invention, the lidar sensor and the camera sensor are integrated, the advantages of different sensors are used, and the shortcomings of the respective sensors are supplemented, and the perception and recognition accuracy of point cloud target detection is improved. At the same time, in this solution, the deep learning network combining the semantic segmentation information of 3D point cloud and the semantic segmentation information of 2D image is used, and the semantic information of different sensors is used to reduce the false detection and false detection rate of obstacles.

Embodiment 2

Based on the same inventive concept, an embodiment of the present invention also provides an obstacle sensing device, as shown in FIG. 3 , the device includes:

The picture obtaining module 303 is configured to obtain the camera picture, and obtain the calibration internal parameters and external parameters of the projection transformation;

a point cloud acquisition module 301, configured to acquire an original point cloud;

The picture semantic segmentation module 304 is configured to perform semantic segmentation on the camera picture to obtain a second picture;

The point cloud semantic segmentation module 302 is configured to perform semantic segmentation on the original point cloud to obtain a second point cloud;

The picture semantic projection module 305 is configured to perform the projection of the original point cloud to the second picture according to the internal parameters and the external parameters to obtain a third point cloud, each of which is in the third point cloud. Each point includes the second semantic category information corresponding to the second picture;

The semantic fusion module 306 is configured to voxelize the second semantic category information in the third point cloud, the first semantic category information in the second point cloud and the feature information of the original point cloud Then, input it into the adaptive attention mechanism network for learning, and obtain the weighted semantic information;

The obstacle perception module 307 is configured to detect the obstacle target according to the weighted semantic information;

Wherein, the second point cloud and the second picture include obstacle category information.

As a preferred example, the picture acquisition module 303 is further configured to:

Obtain a camera image at the same moment as the original point cloud.

Specifically, software synchronization or hardware synchronization may be performed on the point cloud and the camera first, and then the camera picture at the same moment as the original point cloud may be obtained.

As a preferred example, the image semantic segmentation module 304 is further configured to:

The camera picture is input into the picture semantic segmentation network to obtain the second picture.

As a preferred example, the point cloud semantic segmentation module 302 is further configured to:

The original point cloud is input into the point cloud semantic segmentation network to obtain a second point cloud.

As a preferred example, the image semantic projection module 305 is further configured to perform the projection of the original point cloud to the second image in the following manner:

Projection is performed according to the following formula:

P′=Proj(K,M,P),

Among them, Proj is the projection matrix processing process;

K is the internal parameter matrix of the camera;

M is the extrinsic parameter matrix from camera to lidar;

P is the lidar point cloud collection;

P' is the lidar point cloud projected to the camera coordinate system.

As a preferred example, the learning in the input adaptive attention mechanism network includes:

Learning the local features in the adaptive attention mechanism network to obtain the learned local features V _i ;

The global feature is learned in the adaptive attention mechanism network, and the learned global feature V _global is obtained;

The learned global feature V _global is spliced onto each local feature V _i to obtain an enhanced feature V _gl .

The learning in the input adaptive attention mechanism network includes:

Learning the local features in the adaptive attention mechanism network to obtain the learned local features V _i ;

The global feature is learned in the adaptive attention mechanism network, and the learned global feature V _global is obtained;

The learned global feature V _global is spliced onto each local feature V _i to obtain an enhanced feature V _gl .

The global feature is learned in the adaptive attention mechanism network, and the learned global feature V _global includes:

The learning of global features is carried out according to the following formula:

V _global =max _i=1,2,...,N {MLP _g (V _i )}

MLP _g (V _i ) is a global feature multilayer perceptron;

max is the maximum pooling operation for all voxels;

C ₂ is the number of channels in the entire feature map;

N is the number of voxels;

V _i is the learned feature in the i-th voxel grid.

As a preferred example, the semantic fusion module 306 is further configured to obtain weighted semantic information according to the following formula:

Among them, P _a,s and P _a,t are the weighted semantic information;

P _2D is the second semantic information;

P _3D is the first semantic information;

MLP _att is a multi-layer perceptron;

σ is the sigmoid activation function.

It should be noted that the device provided in the second embodiment and the method provided in the first embodiment belong to the same inventive concept, solve the same technical problems, and achieve the same technical effect, and the device provided in the second embodiment can implement all the methods in the first embodiment. , the similarities will not be repeated.

Embodiment 3

Based on the same inventive concept, an embodiment of the present invention also provides an obstacle sensing device, as shown in FIG. 4 , the device includes:

including memory 402, processor 401 and user interface 403;

The memory 402 is used to store computer programs;

The user interface 403 is used to interact with the user;

The processor 401 is configured to read the computer program in the memory 402, and when the processor 401 executes the computer program, it realizes:

Get the original point cloud and camera image at the same moment;

Obtain the calibration internal and external parameters of the projection transformation;

Semantically segment the original point cloud to obtain a second point cloud;

Semantically segment the camera picture to obtain a second picture;

According to the internal reference and the external reference, the original point cloud is projected to the second picture to obtain a third point cloud, where each point in the third point cloud includes the corresponding second semantic category information;

After voxelizing the second semantic category information in the third point cloud, the first semantic category information in the second point cloud and the feature information of the original point cloud, input the adaptive attention mechanism network learning in the process to obtain weighted semantic information;

Detecting an obstacle target according to the weighted semantic information;

Wherein, the second point cloud and the third point cloud include obstacle category information.

Wherein, in FIG. 4 , the bus architecture may include any number of interconnected buses and bridges, specifically, one or more processors represented by processor 401 and various circuits of memory represented by memory 402 are linked together. The bus architecture may also link together various other circuits, such as peripherals, voltage regulators, and power management circuits, which are well known in the art and, therefore, will not be described further herein. The bus interface provides the interface. The processor 401 is responsible for managing the bus architecture and general processing, and the memory 402 may store data used by the processor 501 in performing operations.

The processor 401 may be a CPU, an ASIC, an FPGA or a CPLD, and the processor 401 may also adopt a multi-core architecture.

When the processor 401 executes the computer program stored in the memory 402, any obstacle sensing method in the first embodiment is implemented.

It should be noted that the device provided in the third embodiment and the method provided in the first embodiment belong to the same inventive concept, solve the same technical problems, and achieve the same technical effect, and the device provided in the third embodiment can realize all the methods of the first embodiment. , the similarities will not be repeated.

The present application also proposes a processor-readable storage medium. The processor-readable storage medium stores a computer program, and when the processor executes the computer program, any obstacle sensing method in Embodiment 1 is implemented.

It should be noted that the division of units in the embodiments of the present application is schematic, and is only a logical function division, and other division methods may be used in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein, including but not limited to disk storage, optical storage, and the like.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims (10)

1. An obstacle sensing method, comprising:

acquiring an original point cloud and a camera picture at the same time;

acquiring a calibration internal parameter and a calibration external parameter of projection conversion;

performing semantic segmentation on the original point cloud to obtain a second point cloud;

performing semantic segmentation on the camera picture to obtain a second picture;

according to the internal parameters and the external parameters, projecting the original point cloud to the second picture to obtain a third point cloud, wherein each point in the third point cloud comprises second semantic category information corresponding to the second picture;

after the second semantic category information in the third point cloud, the first semantic category information in the second point cloud and the feature information of the original point cloud are subjected to voxelization, the second semantic category information and the feature information are input into a self-adaptive attention mechanism network for learning, and the weighted semantic information is obtained;

detecting an obstacle target according to the weighted semantic information;

wherein the second point cloud and the third point cloud comprise obstacle category information;

the obtaining of the weighted semantic information includes:

obtaining weighted semantic information according to the following formula:

wherein, P_a，sAnd P_a，tThe weighted semantic information is obtained;

P_2Dthe second semantic category information;

P_3Dthe first semantic category information;

MLP_atta multilayer perceptron;

sigma is a Sigmoid activation function,

the learning in the input adaptive attention mechanism network comprises:

the local features are learned in the adaptive attention mechanism network to obtain learned local features V_i；

The global feature is learned in the adaptive attention mechanism network to obtain a learned global feature V_global；

The global feature V after learning is processed_globalSpliced to each local feature V_iTo obtain an enhanced feature V_gl。

2. The method of claim 1, wherein detecting an obstacle target based on the weighted semantic information comprises:

and inputting the weighted semantic information into a target detector to detect the obstacle target.

3. The method of claim 1, wherein the obtaining the original point cloud and the camera picture at the same time comprises:

carrying out software synchronization or hardware synchronization on the point cloud and the camera;

and obtaining the original point cloud and the camera picture at the same moment.

4. The method of claim 1, wherein semantically segmenting the original point cloud to obtain a second point cloud comprises:

and inputting the original point cloud into a point cloud semantic segmentation network to obtain a second point cloud.

5. The method of claim 1, wherein the semantically segmenting the camera picture to obtain a second picture comprises:

and inputting the camera picture into a picture semantic segmentation network to obtain a second picture.

6. The method of claim 1, wherein before the voxelizing the second semantic category information in the third point cloud, the first semantic category information in the second point cloud, and the feature information of the original point cloud, further comprising:

and converting the first semantic category information and the second semantic category information into a One-Hot coding format.

7. The method of claim 1, wherein the projecting the original point cloud to the second picture comprises:

projection is performed according to the following formula:

P′＝Proj(K，M，P)，

wherein, Proj is a projection matrix processing process;

k is an internal reference matrix of the camera;

m is an external parameter matrix from the camera to the laser radar;

p is a laser radar point cloud set;

and P' is the laser radar point cloud projected to the camera coordinate system.

8. An obstacle sensing device, comprising:

the image acquisition module is configured for acquiring a camera image and acquiring calibration internal parameters and external parameters of projection conversion;

a point cloud acquisition module configured to acquire an original point cloud;

the picture semantic segmentation module is configured for performing semantic segmentation on the camera picture to obtain a second picture;

a point cloud semantic segmentation module configured to perform semantic segmentation on the original point cloud to obtain a second point cloud:

the image semantic projection module is configured to perform projection from the original point cloud to the second image according to the internal parameters and the external parameters to obtain a third point cloud, wherein each point in the third point cloud comprises second semantic category information corresponding to the second image;

the semantic fusion module is configured to perform voxelization on second semantic category information in the third point cloud, first semantic category information in the second point cloud and feature information of the original point cloud, and input the voxelization information into an adaptive attention mechanism network for learning to obtain weighted semantic information;

an obstacle sensing module configured to detect an obstacle target according to the weighted semantic information;

wherein the second point cloud and the second picture comprise obstacle category information;

the semantic fusion module is further configured to obtain weighted semantic information according to the following formula:

wherein, P_a，sAnd P_a，tThe weighted semantic information is obtained;

P_2Dthe second semantic category information;

P_3Dthe first semantic category information;

MLP_attis a multilayer perceptron;

sigma is a Sigmoid activation function,

the learning in the input adaptive attention mechanism network comprises:

the local features are learned in the adaptive attention mechanism network to obtain learned local features V_i；

The global feature is learned in the adaptive attention mechanism network to obtain a learned global feature V_global；

The global feature V after learning is processed_globalSpliced to each local feature V_iTo obtain an enhanced feature V_gl。

9. An obstacle sensing apparatus comprising a memory, a processor and a user interface;

the memory for storing a computer program;

the user interface is used for realizing interaction with a user;

the processor for reading the computer program in the memory, the processor implementing the obstacle sensing method according to one of claims 1 to 7 when executing the computer program.

10. A processor-readable storage medium, characterized in that the processor-readable storage medium stores a computer program which, when executed by a processor, implements an obstacle sensing method according to one of claims 1 to 8.

Images