CN114299131A - Method, device and terminal equipment for detecting low and small obstacles based on three cameras - Google Patents

Abstract

The embodiment of the invention provides a method, a device and terminal equipment for detecting short and small obstacles based on three cameras, wherein the method comprises the following steps: acquiring a first image, a second image and a third image which are synchronously acquired by a first camera, a second camera and a third camera; calculating the difference between the first image and the second image to obtain a first difference characteristic diagram; calculating the difference between the third image and the second image to obtain a second difference characteristic diagram; fusing the first difference feature map and the second difference feature map to obtain a fused feature map; and determining the area where the obstacle is located based on the fused feature map. The generated difference characteristics in two different directions are fused to form a complete obstacle difference characteristic, so that short and short obstacles are effectively identified, the influence of interference factors such as marked lines and shadows on the identification accuracy is reduced, and the sensing capability of the obstacles is enhanced.

Description

Method, device and terminal equipment for detecting low and small obstacles based on three cameras

technical field

Embodiments of the present invention relate to the technical field of automatic driving visual perception, and in particular, to a method, device, and terminal device for detecting low-profile obstacles based on three cameras.

Background technique

As vehicles based on assisted driving and autonomous driving are applied to various scenarios, the demand for obstacle detection is getting higher and higher. Accurate detection of obstacles can improve the safety of vehicles in driving and ensure the normal driving of vehicles.

At present, there are two main methods to identify obstacles in autonomous driving perception technology, visual perception recognition and laser perception recognition. Visual perception includes monocular vision combined with lidar, TOF (Time of flight; time of flight method), and binocular vision obstacle avoidance method. The obstacles recognized by monocular vision combined with lidar and TOF mainly include limited categories such as people and vehicles, and the ability to recognize obstacles outside the limited categories or relatively small obstacles on the ground is relatively poor; while binocular vision is not limited to categories. , but it is very sensitive to the requirements of lighting, bright or dark will lead to poor recognition effect. As for laser perception recognition, although the recognition of obstacles is not limited to the obstacle category, because the laser point cloud is relatively sparse, the effect is not good when detecting low and small obstacles.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a three-camera-based low-profile obstacle detection method, device, robot, and storage medium, so as to solve the problem of improving the detection accuracy of low-profile obstacles.

In a first aspect, an embodiment of the present invention provides a three-camera-based low-profile obstacle detection method, including:

acquiring a first image, a second image and a third image synchronously collected by the first camera, the second camera and the third camera;

calculating the difference between the first image and the second image to obtain a first difference feature map;

calculating the difference between the third image and the second image to obtain a second difference feature map;

fusing the first difference feature map and the second difference feature map to obtain a fusion feature map;

The region where the obstacle is located is determined based on the fusion feature map.

In a second aspect, an embodiment of the present invention also provides a low-profile obstacle detection device, including:

an image acquisition module, configured to acquire the first image, the second image and the third image synchronously collected by the first camera, the second camera and the third camera;

a first difference feature calculation module, configured to calculate the difference between the first image and the second image to obtain a first difference feature map;

a second difference feature calculation module, configured to calculate the difference between the third image and the second image to obtain a second difference feature map;

a difference fusion module, configured to fuse the first difference feature map and the second difference feature map to obtain a fusion feature map;

The post-processing module is used for determining the area where the obstacle is located based on the fusion feature map.

In a third aspect, an embodiment of the present invention also provides a low-profile obstacle detection device, including:

at least one processor; and at least one memory storing instructions executable by the at least one processor;

The instructions are executed by the at least one processor, so that the at least one processor implements the three-camera-based low-profile obstacle detection method according to the first aspect.

In a fourth aspect, an embodiment of the present invention also provides a robot, including:

a first camera for collecting a first image;

a second camera for collecting a second image;

a third camera for collecting a third image;

And the low-profile obstacle detection device according to the second aspect or the third aspect.

In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the third aspect based on the third aspect is implemented. Camera-based low-profile obstacle detection method.

In this embodiment, a first image, a second image, and a third image that are synchronously collected by the first camera, the second camera, and the third camera are acquired; the difference between the first image and the second image is calculated to obtain a first difference feature map Calculate the difference between the third image and the second image to obtain a second difference feature map; fuse the first difference feature map and the second difference feature map to obtain a fusion feature map; determine the area where the obstacle is located based on the fusion feature map. Through the combination of three cameras and two networks, the generated difference features in two different directions are fused to form a complete obstacle difference feature, which can effectively identify low and small obstacles on indoor and outdoor roads and reduce the number of obstacles on the road. The influence of interference factors such as marking lines and shadows on the recognition accuracy can be complementary to general visual perception and laser perception, and enhance the ability to perceive obstacles in the process of automatic driving.

Description of drawings

1A is a flowchart of a method for detecting low and small obstacles based on three cameras according to Embodiment 1 of the present invention;

FIG. 1B is an example flow chart of data processing of the three-camera-based low-profile obstacle detection method in Embodiment 1 of the present invention;

Fig. 1C is a fusion feature map obtained by using a three-camera-based low and small obstacle detection method for a conical roadblock;

2 is a schematic structural diagram of a low-profile obstacle detection device according to Embodiment 2 of the present invention;

3 is a schematic structural diagram of a low-profile obstacle detection device according to Embodiment 3 of the present invention;

FIG. 4 is a schematic structural diagram of a robot according to Embodiment 4 of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. In addition, it should be noted that, for the convenience of description, the drawings only show some but not all structures related to the present invention.

Example 1

1A is a flowchart of a method for detecting low and small obstacles based on three cameras according to Embodiment 1 of the present invention, and FIG. 1B is a data processing of the method for detecting low and small obstacles based on three cameras in Embodiment 1 of the present invention An example flow chart, this embodiment can be applied to improve the detection accuracy of low and small obstacles, the method can be performed by a three-camera-based low and small obstacle detection device, the three-camera-based low and small obstacles The detection device can be implemented by software and/or hardware, and can be configured in computer equipment, robots, such as intelligent robots, servers, personal computers, etc., and specifically includes the following steps:

Step 101: Acquire a first image, a second image, and a third image synchronously collected by a first camera, a second camera, and a third camera.

Exemplarily, the first camera, the second camera, and the third camera are all set on the same plane, and are all used to acquire the spatial scene in front of the current plane. The first camera and the second camera are aligned at a preset distance in the same horizontal direction, the second camera and the third camera are aligned at a preset distance in the same vertical direction, and the third camera is tilted downward by a preset angle , to obtain an image of a larger portion of the ground. A three-buffer queue is set up, and the images captured by the first camera, the second camera, and the third camera are input into each buffer queue, respectively, and the images captured at the same time are selected from the three buffer queues as the first image, the first image and the third buffer queue. Second image, third image.

Exemplarily, the first camera and the second camera are symmetrically arranged at a distance of 30 cm in the same horizontal direction, and the angles of the first camera and the second camera are also horizontal to the ground. The second camera and the third camera are arranged at an interval of 40cm in the same vertical direction, and the third camera is arranged above the second camera and is inclined downward by 20°. The first camera, the second camera, and the third camera all continue to photograph the scene in front of the cameras, so as to collect images. The first camera, the second camera, and the third camera transfer the collected data into different queues of the three buffer queues, respectively, and acquire images at the same time in the three buffer queues, as the first image, the second image, and the third image respectively. , if there are no three images whose times are completely matched, three images whose time difference does not exceed a preset interval threshold, for example, 10 ms, are selected as the first image, the second image, and the third image, respectively. The resolutions of the images collected by the three cameras are all 1028×720, and the frame rate is 30 frames per second.

It should be noted that the first camera and the second camera in the above embodiment are in the same horizontal direction, and the second camera and the third camera are in the same vertical direction are exemplary descriptions of the embodiments of the present invention, and other implementations of the present invention In an example, the first camera, the second camera, and the third camera may also have other different position setting methods, for example, the first camera and the second camera are in the same horizontal direction, and the third camera is located between the first camera and the second camera. Just above the midpoint, the present invention is not limited here.

In some embodiments of the present invention, step 101 includes:

Step 1011: Calculate a first homography matrix according to four pairs of key marker points in the first image and the second image.

The homography transformation is used to describe the positional mapping relationship of objects between two images. Through the coordinates of the four pairs of key marker points selected from the two images, the homography matrix H between the two images is calculated, and then The projective transformation function is called to transform one image into the data of another image. The transformation matrix used for the projective transformation is called the homography matrix. A homography matrix can be represented by a 3×3 nonsingular matrix H as follows:

The homography matrix H is a homogeneous matrix with 8 unknowns, with the last element normalized to 1.

Four pairs of points with the same feature in the first image and the second image are selected as key marker point pairs, and each pair of key marker point pairs is a pixel point pair representing the same feature in the first image and the second image.

The value of the first homography matrix H ₁ for enabling the first image to be aligned with the second image is obtained through the following formula using the coordinate relationship of the four pairs of key marker points in the first image and the second image.

In the following embodiments of the present invention, before calculating the first homography matrix, the internal parameter matrices of the first camera and the second camera can be used to correct the first image and the second image respectively, so as to facilitate the display of the picture and be Navigation provides standard location information. After correcting the first image and the second image by using the internal reference matrix, use the corrected first image and the corrected second image to mark key marker point pairs, so as to calculate the first homography matrix.

It should be noted that the process of obtaining the first homography matrix by using four pairs of key marker points in the above embodiment is an exemplary description of the embodiment of the present invention, and in other embodiments of the present invention, there may be other The process of obtaining the first homography matrix for key marker point pairs is not limited in the present invention.

Step 1012: Multiply the first homography matrix by a matrix consisting of pixel values of the first image to align the first image and the second image.

For a point in the first image plane, use w=1 to normalize the point value, and use the two coordinates x, y of the image coordinates to form a coordinate matrix [xy 1] ^-1 with it, and use this coordinate matrix to correspond to the first homography The coordinate matrix [x'y'w'] ^-1 obtained by multiplying the property matrix H ₁ , that is, the first image and the second image are aligned by the following formula.

In some embodiments of the present invention, step 101 includes:

Step 1013: Calculate a first homography matrix according to four pairs of key marker points in the second image and the third image.

Four pairs of points with the same feature in the second image and the third image are selected as key marker point pairs, and each pair of key marker point pairs is a pixel point pair representing the same feature in the second image and the third image.

The value of the second homography matrix H ₂ for enabling the third image to be aligned with the second image is obtained by the following formula using the coordinate relationship of the four pairs of key marker points in the second image and the third image.

In an embodiment of the present invention, before calculating the second homography matrix, the internal parameter matrices of the second camera and the third camera can be used to correct the second image and the third image respectively, so as to facilitate the display of the picture and the Navigation provides standard location information. After correcting the second image and the third image by using the internal reference matrix, use the corrected second image and the corrected third image to mark key marker point pairs, so as to calculate the second homography matrix.

It should be noted that the process of obtaining the second homography matrix by using four pairs of key marker points in the above embodiment is an exemplary description of the embodiment of the present invention, and in other embodiments of the present invention, there may be other The process of obtaining the second homography matrix for key marker point pairs is not limited in the present invention.

Step 1014: Multiply the second homography matrix by a matrix consisting of pixel values of the third image to align the third image and the second image.

For points in the second image plane, use w=1 to normalize the point values, and use the two coordinates x, y of the image coordinates to form a coordinate matrix [xy 1] ^-1 with them, and use this coordinate matrix to correspond to the second homography The coordinate matrix [x'y'w'] ^-1 obtained by multiplying the sex matrix H ₂ , that is, the second image and the third image are aligned by the following formula.

Step 102: Calculate the difference between the first image and the second image to obtain a first difference feature map.

The first image and the second image include a common part, and the first image and the second image are horizontally aligned. By calculating the difference between the first image and the second image, the first difference feature map is generated, and the current Features in the horizontal direction in the area in front of the camera. For objects with width, there will be more obvious features.

It should be noted that the horizontal direction in the above embodiment is an exemplary illustration of the embodiment of the present invention. In other embodiments of the present invention, the first image and the second camera are calculated according to the settings of the first camera and the second camera. The difference between the two images can also obtain features in other directions, which are not limited in the present invention.

In some embodiments of the present invention, step 102 includes:

Step 1021: Input the first image and the second image respectively into the MobileNetV2 network for processing, take the feature maps output by the first k convolution blocks in the MobileNetV2 network, and obtain k first feature maps and second images corresponding to the first image The corresponding k first feature maps.

The MobileNetV2 network is a network that can significantly reduce model parameters and computation while maintaining similar accuracy. The network now extends the input low-dimensional compressed representation to high dimensions, using lightweight depth volumes. The product is filtered; the features are then projected back to a low-dimensional compressed representation using a linear bottleneck. The depthwise convolution in the MobileNetV2 network uses a 3×3 depthwise separable convolution (Depthwise Separable Convolution). The MobileNetV2 network consists of multiple convolutional blocks connected in sequence, and the output of the previous convolutional block is used as the input of the subsequent convolutional block.

After adjusting the first image to the preset size and inputting it into the MobileNetV2 network, take the feature maps output by the first k convolution blocks in the MobileNetV2 network as the k first feature maps corresponding to the first image. The figure is a feature map containing image features output by the convolution block of the MobileNetV2 network.

After the second image is adjusted to a preset size and input into the MobileNetV2 network, the feature maps output by the first k convolution blocks in the MobileNetV2 network are taken as the k first feature maps corresponding to the second image. The number of first feature maps extracted from the first image and the second image must be the same.

Exemplarily, k is 3. After the first image and the second image are adjusted to 512×256, they are respectively input to the MobileNetV2 network, and the feature maps output by the first three convolution blocks are extracted respectively to obtain three corresponding to the first image. The first feature map and the three first feature maps corresponding to the second image. They include the features of the first image and the second image, respectively.

It should be noted that the value of k in the above embodiments is an exemplary description of the embodiments of the present invention, and in other embodiments of the present invention, k may also be other values, which are not limited in the present invention.

Step 1022: Calculate the difference between the ith first feature map corresponding to the first image and the ith first feature map corresponding to the second image, respectively, to obtain k first salient feature maps, where i is less than or equal to k .

The number of first feature maps extracted from the first image and the second image is the same, and the value of i changes from 1 to k to calculate the first feature map corresponding to the first image and the first feature corresponding to the second image layer by layer. The difference between the two images makes the features in the overlapping part of the first image and the second image more obvious, and the feature map obtained after the subtraction is called the first salient feature map.

Exemplarily, the value of k is 3, i is first set to 1, and the difference between the first first feature map corresponding to the first image and the first first feature map corresponding to the second image is calculated to obtain the first The first salient feature map; i is set to 2, calculate the difference between the second first feature map corresponding to the first image and the second first feature map corresponding to the second image, and obtain the second first salient feature map ; i is set to 3, calculate the difference between the third first feature map corresponding to the first image and the third first feature map corresponding to the second image, and obtain the third first salient feature map, so a total of k first salient feature maps.

Step 1023: Input the first image and the second image respectively into the mbv2_ca network for processing, take the feature maps output by the first m convolution blocks in the mbv2_ca network, and obtain m second feature maps and second images corresponding to the first image The corresponding m second feature maps.

The mbv2_ca network is a MobileNetV2 network with an attention block added. The attention mechanism can be considered as a resource allocation mechanism. For the resources that were originally allocated evenly, the resources are redistributed according to the importance of the attention object. The important units are divided a little more, and the unimportant or bad units are divided a little less. In the deep neural network In the structural design of , the resources to be allocated by attention are basically weights. By adding attention blocks, the ability to express convolutional features can be effectively improved. The mbv2_ca network consists of multiple convolutional blocks connected in sequence, and the output of the previous convolutional block is used as the input of the subsequent convolutional block.

After adjusting the first image to the preset size and inputting it into the mbv2_ca network, take the feature maps output by the first m convolution blocks in the mbv2_ca network as m second feature maps corresponding to the first image, the second feature The figure is a feature map containing image features output by the convolutional block of the mbv2_ca network.

After the second image is adjusted to a preset size and input into the mbv2_ca network, the feature maps output by the first m convolution blocks in the mbv2_ca network are taken as m second feature maps corresponding to the second image. The number of second feature maps extracted from the first image and the second image must be the same.

Use the mbv2_ca network to perform feature extraction on the first image and the second image to obtain m second feature maps, which can be combined with the k first feature maps obtained by using the MobileNetV2 network feature extraction in step 1021 to form (k+m) features , it is convenient to search for more different features in the first image and the second image.

Exemplarily, m is 2. After the first image and the second image are adjusted to 512×256, they are respectively input to the mbv2_ca network, and the feature maps output by the first two convolution blocks are extracted respectively to obtain two corresponding to the first image. The second feature map and two second feature maps corresponding to the second image. They include the features of the first image and the second image, respectively.

It should be noted that the value of m in the above-mentioned embodiment is an exemplary illustration of the embodiment of the present invention. In other embodiments of the present invention, m may also be other values, and the values of m and k may be equal or different. Equivalent, the present invention is not limited here.

Step 1024: Calculate the difference between the ith second feature map corresponding to the first image and the ith second feature map corresponding to the second image to obtain m second salient feature maps, where i is less than or equal to m, The k first salient feature maps and the m second salient feature maps form the first difference feature map.

The number of second feature maps extracted from the first image and the second image is the same, and the value of i varies from 1 to m. The second feature map corresponding to the first image and the second feature corresponding to the second image are calculated layer by layer. The difference between the two images is used to make the features in the overlapping part of the first image and the second image more obvious, and the feature map obtained after the subtraction is called the second salient feature map.

Exemplarily, the value of m is 2, i is set to 1 first, and the difference between the first second feature map corresponding to the first image and the first second feature map corresponding to the second image is calculated to obtain the first Second salient feature map; i is set to 2, calculate the difference between the second second feature map corresponding to the first image and the second second feature map corresponding to the second image, and obtain the second second salient feature map ; so a total of 2 second salient feature maps are obtained.

The k first salient feature maps extracted and calculated by the MobileNetV2 network and the m second salient feature maps extracted and calculated by the mbv2_ca network constitute the first difference feature map. There are two images captured by the first camera and the second camera with horizontal visual difference in direction, so the first difference feature map generated by the processing and calculation of the first image and the second image will have a more obvious level for objects with width. Differential features in direction.

The shallow layer of the network extracts the primary features of the image, such as texture, color, corners and other features, while the features extracted from the deep layer of the network are semantic features that cannot be recognized by the naked eye. The purpose of the present invention is to extract the saliency difference feature of the image without extracting the semantic feature deep in the network. In this embodiment, the feature map of the first k layers of the MobileNetV2 network and the feature map of the first m layers of the mbv2_ca network are extracted The primary features such as texture and color in each image can be extracted, which are sufficient to detect obstacles, thereby improving the efficiency in the process of detecting low and small obstacles.

Moreover, the MobileNetV2 network and the mbv2_ca network are used to extract the primary features of the image. The combination of the primary features extracted by the two networks can improve the detection accuracy.

It should be noted that the MobileNetV2 network and the mbv2_ca network used in the above embodiments are exemplary descriptions of the embodiments of the present invention. Feature extraction is not limited in the present invention.

Step 103: Calculate the difference between the third image and the second image to obtain a second difference feature map.

The second image and the third image include a common part. The second image and the third image are vertically aligned. By calculating the difference between the second image and the third image, a second difference feature map is generated, and the current camera can be obtained. Features in the vertical direction in the front area. For objects with height, there will be more obvious features.

In some embodiments of the present invention, step 103 includes:

Step 1031: Input the third image into the MobileNetV2 network for processing, and obtain the k first feature maps corresponding to the third image by taking the feature maps output by the first k convolution blocks in the MobileNetV2 network.

After the third image is adjusted to a preset size and input into the MobileNetV2 network, the feature maps output by the first k convolution blocks in the MobileNetV2 network are taken as the k first feature maps corresponding to the third image.

Exemplarily, k is 3. After the third image is adjusted to 512×256, it is input to the MobileNetV2 network, and the feature maps output by the first three convolution blocks are extracted to obtain three first feature maps corresponding to the third image. They include a third image.

It should be noted that the value of k in the above embodiments is an exemplary description of the embodiments of the present invention, and in other embodiments of the present invention, k may also be other values, which are not limited in the present invention.

Step 1032: Calculate the difference between the ith first feature map corresponding to the third image and the ith first feature map corresponding to the second image, respectively, to obtain k third salient feature maps, where i is less than or equal to k .

The number of first feature maps extracted from the second image and the third image is the same, there are k first feature maps, and the value of i changes from 1 to k to calculate the first feature map corresponding to the third image layer by layer The difference value of the first feature map corresponding to the second image, so that the features of the overlapping part of the third image and the second image are more clearly expressed.

Exemplarily, the value of k is 3, i is set to 1 first, and the difference between the first first feature map corresponding to the third image and the first first feature map corresponding to the second image is calculated to obtain the first The third salient feature map; i is set to 2, calculate the difference between the second first feature map corresponding to the third image and the second first feature map corresponding to the second image, and obtain the second third salient feature map ; i is set to 3, calculate the difference between the third first feature map corresponding to the third image and the third first feature map corresponding to the second image, and obtain the third third significant feature map, so a total of k third salient feature maps.

Step 1033: Input the third image into the mbv2_ca network for processing, take the feature maps output by the first m convolution blocks in the mbv2_ca network, and obtain m second feature maps corresponding to the third image.

After the third image is adjusted to a preset size and input into the mbv2_ca network, the feature maps output by the first m convolution blocks in the mbv2_ca network are taken as m second feature maps corresponding to the third image.

Exemplarily, m is 2, after adjusting the third image to 512×256, input it to the mbv2_ca network, and extract the feature maps output by the first two convolution blocks to obtain two second feature maps corresponding to the third image. They include the features of the first image and the second image, respectively.

It should be noted that the value of m in the above-mentioned embodiment is an exemplary illustration of the embodiment of the present invention. In other embodiments of the present invention, m may also be other values, and the values of m and k may be equal or different. Equivalent, the present invention is not limited herein.

Step 1034: Calculate the difference between the ith second feature map corresponding to the third image and the ith second feature map corresponding to the second image, and obtain m fourth significant feature maps, where i is less than or equal to m, The k third salient feature maps and the m fourth salient feature maps form the second difference feature map.

The number of second feature maps extracted from the second image and the third image is the same, and the value of i varies from 1 to m. The second feature map corresponding to the third image and the second feature corresponding to the second image are calculated layer by layer. The difference value between the maps makes the features in the overlapping part of the third image and the second image more obvious, and the feature map obtained after the subtraction is called the fourth salient feature map.

Exemplarily, the value of m is 2, i is set to 1 first, and the difference between the first second feature map corresponding to the third image and the first second feature map corresponding to the second image is calculated to obtain the first Fourth salient feature map; i is set to 2, calculate the difference between the second second feature map corresponding to the third image and the second second feature map corresponding to the second image, and obtain the second fourth salient feature map ; so a total of 2 fourth salient feature maps are obtained.

In step 103, the k third salient feature maps extracted and calculated by the MobileNetV2 network and the m fourth salient feature maps extracted and calculated by the mbv2_ca network constitute the second difference feature map. The second image and the third image are made of the same There are two images captured by the second camera and the third camera with vertical visual feature differences in the vertical direction. Therefore, the second difference feature map generated by the processing and calculation of the second image and the third image will have a different feature map for objects with height. There are obvious differences in the vertical direction.

It should be noted that the MobileNetV2 network and the mbv2_ca network used in the above embodiments are exemplary descriptions of the embodiments of the present invention. Feature extraction is not limited in the present invention.

Step 104 , fuse the first difference feature map and the second difference feature map to obtain a fusion feature map.

The first difference feature map represents features in the horizontal direction, and the second difference feature map represents features in the vertical direction.

The fusion feature map generated by fusing the first difference feature map and the second difference feature map can fully reflect the salient features of obstacles, so that the features of obstacles of various shapes, heights, and widths can be displayed. , to improve the detection accuracy and prevent obstacles that are too narrow or too short to be detected.

In some embodiments of the present invention, step 104 includes:

Step 1041: Calculate the sum of the ith first salient feature map in the first difference feature map and the ith third salient feature map in the second difference feature map, where i is less than or equal to k, and k first salient feature maps are obtained. Fusion feature maps.

The first difference feature map includes k first salient feature maps, and the second difference feature map includes k third salient feature maps. They respectively represent the features in the horizontal and vertical directions extracted by the MobileNetV2 network. i varies from 1 to k, and the sum of the first difference feature map and the second difference feature map in the i-th layer is calculated layer by layer, so that the features in the horizontal and vertical directions can constitute the salient features of the complete obstacle.

Exemplarily, k is 3, i is first set to 1, and the sum of the first first salient feature map and the first third salient feature map is calculated to obtain the first first fusion feature map; i becomes 2 , Calculate the sum of the second first salient feature map and the second third salient feature map, and get the second first fusion feature map; i becomes 3, calculate the third first salient feature map and the third The sum of the third salient feature maps, and the third first fusion feature map is obtained. After the above calculation, a total of k first fusion feature maps are obtained. The first fusion feature map is to extract features in the horizontal and vertical directions through the MobileNetV2 network. The complete salient features of obstacles obtained by post-combination.

It should be noted that the value of k in the above embodiments is an exemplary description of the embodiments of the present invention. In other embodiments of the present invention, k may also be other values, which are not limited herein.

Step 1042: Calculate the sum of the i-th second salient feature map in the first difference feature map and the i-th fourth salient feature map in the second difference feature map, where i is less than or equal to m, and m second salient feature maps are obtained. Fusion feature maps.

The first difference feature map includes m second salient feature maps, and the second difference feature map includes m fourth salient feature maps. They represent the features in the horizontal and vertical directions extracted by the mbv2_ca network calculation, respectively. i varies from 1 to m, and the sum of the first difference feature map and the second difference feature map in the i-th layer is calculated layer by layer, so that the features in the horizontal and vertical directions can constitute the salient features of the complete obstacle.

Exemplarily, m is 2, i is first set to 1, and the sum of the first second salient feature map and the first fourth salient feature map is calculated to obtain the first second fusion feature map; i becomes 2 , calculate the sum of the second second salient feature map and the second fourth salient feature map, and obtain the second second fusion feature map; after the above calculation, a total of m second fusion feature maps are obtained, and the second fusion feature The figure is the complete salient features of obstacles obtained by combining the features extracted in the horizontal and vertical directions through the mbv2_ca network.

It should be noted that the value of m in the above embodiments is an exemplary description of the embodiments of the present invention, and in other embodiments of the present invention, m may also be other values, which are not limited in the present invention.

Step 1043: Calculate the product of the first fused feature map and the second fused feature map and the corresponding hyperparameters to obtain (k+m) fused hyperparameter feature maps.

In order to more clearly express the characteristics of obstacles, (k+m) hyperparameters are preset, which are used to multiply with k first fusion feature maps and m second fusion feature maps respectively to highlight or weaken some features . The k first fusion feature maps obtained in step 1041 are multiplied with the corresponding k hyperparameters to obtain k fusion hyperparameter feature maps; the m second fusion feature maps obtained in step 1042 and the corresponding m hyperparameters Multiply to get m fused hyperparameter feature maps. Therefore, a total of (k+m) fused hyperparameter feature maps are obtained.

Exemplarily, three first fusion feature maps are obtained in step 1041, and two second fusion feature maps are obtained in step 1042. The 3 first fusion feature maps are multiplied with 3 hyperparameters respectively to obtain 3 fusion hyperparameter feature maps; the 2 second fusion feature maps are multiplied with 2 hyperparameters respectively to obtain 2 fusion hyperparameter feature maps; A total of 5 fused hyperparameter feature maps are obtained.

Step 1044: Calculate the sum of (k+m) fused hyperparameter feature maps to obtain a fused feature map.

Calculate the sum of (k+m) fused hyperparameter feature maps, that is, add up the feature maps of all obstacles adjusted by hyperparameters to obtain a better expressive fused feature map. The fusion feature map can reflect the features of obstacles more accurately.

Step 105: Determine the area where the obstacle is located based on the fusion feature map.

Figure 1C shows the fusion feature map of the conical roadblock using the three-camera-based low and small obstacle detection method. As shown in Figure 1C, the fusion feature map obtained after calculation processing has obvious and continuous grayscales The area with different values, the area is the area where the obstacle is located.

It should be noted that in the above embodiment, the obvious and continuous areas with different gray values are used as the area where the obstacle is located, which is an exemplary illustration of the embodiment of the present invention. In other embodiments of the present invention, other The method determines the region where the obstacle in the fusion feature map is located, which is not limited in the present invention.

In some embodiments of the present invention, step 105 includes:

Step 1051: Compare the grayscale value of each pixel in the fusion feature map with a preset grayscale threshold.

The preset grayscale threshold is used to determine whether there is an area with obstacles.

The gray value of each pixel in the fusion feature map is compared with the preset gray threshold, and the area with obstacles is detected according to the preset gray threshold.

Step 1053: Take the area formed by the pixels whose gray value is greater than or equal to the gray threshold value as the area where the obstacle is located.

When there is an area formed by several pixels whose gray value is greater than or equal to the gray threshold in the fusion feature map, the area is regarded as the area where the obstacle is located.

Exemplarily, the preset grayscale threshold is 200. When the grayscale values of several pixels in the fusion feature map are all greater than or equal to 200, the area formed by these pixels is regarded as the area where the obstacle is located, That is, the obstacle has been detected.

It should be noted that the preset grayscale threshold 200 in the above-mentioned embodiment is an exemplary illustration of the embodiment of the present invention. In other embodiments of the present invention, the grayscale threshold value may also be based on the situation and accuracy of application in an actual scene. The actual numerical value is determined according to the requirements, and the present invention is not limited here.

It should be noted that, for the sake of simple description, the method embodiments are expressed as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the described action sequence, because According to embodiments of the present invention, certain steps may be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

In this embodiment, a first image, a second image, and a third image that are synchronously collected by the first camera, the second camera, and the third camera are acquired; the difference between the first image and the second image is calculated to obtain a first difference feature map Calculate the difference between the third image and the second image to obtain a second difference feature map; fuse the first difference feature map and the second difference feature map to obtain a fusion feature map; determine the area where the obstacle is located based on the fusion feature map. Through the combination of three cameras and two networks, the generated difference features in two different directions are fused to form a complete obstacle difference feature, which can effectively identify low and small obstacles on indoor and outdoor roads and reduce the number of obstacles on the road. The influence of interference factors such as marking lines and shadows on the recognition accuracy can be complementary to general visual perception and laser perception, and enhance the ability to perceive obstacles in the process of automatic driving.

Embodiment 2

2 is a structural block diagram of a low-profile obstacle detection device provided in Embodiment 2 of the present invention, which may specifically include the following modules:

An image acquisition module 201, configured to acquire a first image, a second image and a third image synchronously collected by the first camera, the second camera and the third camera;

a first difference feature calculation module 202, configured to calculate the difference between the first image and the second image to obtain a first difference feature map;

The second difference feature calculation module 203 is configured to calculate the difference between the third image and the second image to obtain a second difference feature map;

Difference fusion module 204, configured to fuse the first difference feature map and the second difference feature map to obtain a fusion feature map;

The post-processing module 205 is configured to determine the area where the obstacle is located based on the fusion feature map.

In some embodiments of the present invention, the image acquisition module 201 includes:

The first homography matrix generation submodule is used to calculate the first homography matrix according to four pairs of key marker point pairs in the first image and the second image, wherein each pair of the key marker point pairs is is a pair of pixels representing the same feature in the first image and the second image;

A first image alignment module, configured to multiply the first homography matrix by a matrix composed of pixel values of the first image to align the first image and the second image.

In some embodiments of the present invention, the image acquisition module 201 includes:

A second homography matrix generating submodule, configured to calculate a second homography matrix according to four pairs of key marker points in the second image and the third image, wherein each pair of the key marker points is are pixel pairs representing the same feature in the second image and the third image;

A third image alignment module, configured to multiply the second homography matrix by a matrix composed of pixel values of the third image to align the third image and the second image.

In some embodiments of the present invention, the first difference feature calculation module 202 includes:

The first feature map generation sub-module is used to input the first image and the second image into the MobileNetV2 network respectively for processing, and take the feature maps output by the first k convolution blocks in the MobileNetV2 network to obtain the k first feature maps corresponding to the first image and k first feature maps corresponding to the second image;

The first salient feature map generation sub-module is used to calculate the difference between the ith first feature map corresponding to the first image and the ith first feature map corresponding to the second image, and obtain k th first feature maps. a salient feature map, where i is less than or equal to k;

The second feature map generation sub-module is used to input the first image and the second image into the mbv2_ca network for processing, and take the feature maps output by the first m convolution blocks in the mbv2_ca network to obtain the m second feature maps corresponding to the first image and m second feature maps corresponding to the second image;

The second salient feature map generation sub-module is used to calculate the difference between the ith second feature map corresponding to the first image and the ith second feature map corresponding to the second image, and obtain m second feature maps. A salient feature map, wherein i is less than or equal to m, and the k first salient feature maps and the m second salient feature maps constitute a first difference feature map.

In some embodiments of the present invention, the second difference feature calculation module 203 includes:

The first feature map generation sub-module is used to input the third image into the MobileNetV2 network for processing, take the feature maps output by the first k convolution blocks in the MobileNetV2 network, and obtain k corresponding to the third image. the first feature map;

The third salient feature map generation sub-module is used to calculate the difference between the ith first feature map corresponding to the third image and the ith first feature map corresponding to the second image, respectively, to obtain k th first feature maps. Three salient feature maps, where i is less than or equal to k;

The second feature map generation sub-module is used to input the third image into the mbv2_ca network for processing, take the feature maps output by the first m convolution blocks in the mbv2_ca network, and obtain m corresponding to the third image. the second feature map;

The fourth salient feature map generation sub-module is used to calculate the difference between the ith second feature map corresponding to the third image and the ith second feature map corresponding to the second image, to obtain m fourth A salient feature map, wherein i is less than or equal to m, and k third salient feature maps and m fourth salient feature maps constitute the second difference feature map.

In some embodiments of the present invention, the difference fusion module 204 includes:

The first fusion feature map generation sub-module is used to calculate the i-th first salient feature map in the first difference feature map and the i-th third salient feature map in the second difference feature map. and value, where i is less than or equal to k, and k first fusion feature maps are obtained;

The second fusion feature map generation sub-module is used to calculate the i-th second salient feature map in the first difference feature map and the i-th fourth salient feature map in the second difference feature map. and value, where i is less than or equal to m, to obtain m second fusion feature maps;

The fusion hyperparameter feature map generation submodule is used to calculate the product of the first fusion feature map and the second fusion feature map and the corresponding hyperparameters to obtain (k+m) fusion hyperparameter feature maps;

The fusion feature map generation sub-module is used to calculate the sum of the (k+m) fusion hyperparameter feature maps to obtain a fusion feature map.

In some embodiments of the present invention, the post-processing module 205 includes:

a grayscale comparison submodule, configured to compare the grayscale value of each pixel in the fusion feature map with a preset grayscale threshold;

The obstacle determination sub-module is used for taking the area formed by the pixels whose gray value is greater than or equal to the gray threshold value as the area where the obstacle is located.

The three-camera-based low-profile obstacle detection device provided by the embodiment of the present invention can execute the three-camera-based low-profile obstacle detection method provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method. .

Embodiment 3

FIG. 3 is a schematic structural diagram of a low-profile obstacle detection device according to Embodiment 3 of the present invention. Figure 3 shows a block diagram of an exemplary low profile obstacle detection device 12 suitable for use in implementing embodiments of the present invention. The low and small obstacle detection device 12 shown in FIG. 3 is only an example, and should not impose any limitation on the function and scope of use of the embodiment of the present invention.

As shown in FIG. 3 , the low-profile obstacle detection device 12 is embodied in the form of a general-purpose computing device. Components of low profile obstacle detection device 12 may include, but are not limited to, one or more processors or processing units 16 , system memory 28 , and a bus 18 connecting various system components including system memory 28 and processing unit 16 .

Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect ( PCI) bus.

Low profile obstacle detection device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the low-profile obstacle detection device 12, including volatile and non-volatile media, removable and non-removable media.

System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and/or cache memory 32 . The low profile obstacle detection device 12 may further include other removable/non-removable, volatile/non-volatile computer system storage media. For example only, storage system 34 may be used to read and write to non-removable, non-volatile magnetic media (not shown in FIG. 3, commonly referred to as a "hard disk drive"). Although not shown in Figure 3, a disk drive may be provided for reading and writing to removable non-volatile magnetic disks (eg "floppy disks"), as well as removable non-volatile optical disks (eg CD-ROM, DVD-ROM) or other optical media) to read and write optical drives. In these cases, each drive may be connected to bus 18 through one or more data media interfaces. Memory 28 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of various embodiments of the present invention.

A program/utility 40 having a set (at least one) of program modules 42, which may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data , each or some combination of these examples may include an implementation of a network environment. Program modules 42 generally perform the functions and/or methods of the described embodiments of the present invention.

The low-profile obstacle detection device 12 may also communicate with one or more external devices 14 (eg, keyboards, pointing devices, displays 24, etc.) 12, and/or with any device (eg, network card, modem, etc.) that enables the low-profile obstacle detection device 12 to communicate with one or more other computing devices. Such communication may take place through input/output (I/O) interface 22 . Also, the low-profile obstacle detection device 12 may communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and/or a public network such as the Internet) through a network adapter 20 . As shown, the network adapter 20 communicates with other modules of the low profile obstacle detection device 12 via the bus 18 . It should be understood that, although not shown, other hardware and/or software modules may be used in conjunction with the low-profile obstacle detection device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, for example, to implement the low and small obstacle detection method provided by the embodiments of the present invention.

Embodiment 4

4 is a schematic structural diagram of a robot according to Embodiment 4 of the present invention, which may specifically include the following equipment:

a first camera 401, used for collecting a first image;

a second camera 402, configured to collect a second image;

a third camera 403, configured to collect a third image;

Low and small obstacle detection device 12 .

In order for those skilled in the art to better understand the embodiments of the present application, in this specification, an example of the structure of a robot is described.

Exemplarily, the first camera 401, the second camera 402, and the third camera 403 are all connected to the low and low obstacle detection device 12, and the collected first, second, and third images are transmitted to the low and low obstacle detection device 12. The small obstacle device 12 is processed. 3 and 4 , the first camera 401 , the second camera 402 , and the third camera 403 are equivalent to the external device 14 in FIG. 3 , and are connected to the low-profile obstacle device 12 through the I/O interface 22 .

Specifically, the robot may be a robot with different purposes and different forms, such as a delivery robot, a cleaning robot, a wheeled robot, or a mobile robot. This embodiment is only an example and not a limitation.

The robot provided by the embodiment of the present invention can execute the processing method of the robot provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

Embodiment 5

Embodiment 5 of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, each process of the foregoing three-camera-based low-profile obstacle detection method is implemented , and can achieve the same technical effect, in order to avoid repetition, it is not repeated here.

Wherein, the computer-readable storage medium may include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. More specific examples (a non-exhaustive list) of computer readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), Erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

Note that the above are only preferred embodiments of the present invention and applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present invention. The scope is determined by the scope of the appended claims.

Claims (11)

1. A short and small obstacle detection method based on three cameras is characterized by comprising the following steps:

acquiring a first image, a second image and a third image which are synchronously acquired by a first camera, a second camera and a third camera;

calculating the difference between the first image and the second image to obtain a first difference characteristic diagram;

calculating the difference between the third image and the second image to obtain a second difference characteristic diagram;

fusing the first difference feature map and the second difference feature map to obtain a fused feature map;

and determining the area where the obstacle is located based on the fused feature map.

2. The method of claim 1, wherein the obtaining the first image, the second image, and the third image synchronously captured by the first camera, the second camera, and the third camera comprises:

calculating a first homography matrix according to four pairs of key mark point pairs in the first image and the second image, wherein each pair of key mark point pairs is a pixel point pair which represents the same characteristic in the first image and the second image;

multiplying the first homography matrix with a matrix consisting of pixel values of the first image to align the first image and the second image.

3. The method of claim 1, wherein the obtaining the first image, the second image, and the third image synchronously captured by the first camera, the second camera, and the third camera comprises:

calculating a second homography matrix according to four pairs of key mark point pairs in the second image and the third image, wherein each pair of key mark point pairs is a pixel point pair which represents the same characteristic in the second image and the third image;

multiplying the second homography matrix with a matrix consisting of pixel values of the third image to align the third image and the second image.

4. The method of claim 1, wherein calculating the difference between the first image and the second image to obtain a first difference feature map comprises:

inputting the first image and the second image into a MobileNet V2 network for processing, and taking feature maps output by the first k convolution blocks in the MobileNet V2 network to obtain k first feature maps corresponding to the first image and k first feature maps corresponding to the second image;

respectively calculating the difference value of the ith first feature map corresponding to the first image and the ith first feature map corresponding to the second image to obtain k first significant feature maps, wherein i is less than or equal to k;

inputting the first image and the second image into an mbv2_ ca network respectively for processing, and taking feature maps output by the first m convolution blocks in the mbv2_ ca network to obtain m second feature maps corresponding to the first image and m second feature maps corresponding to the second image;

and calculating the difference value between the ith second feature map corresponding to the first image and the ith second feature map corresponding to the second image to obtain m second significant feature maps, wherein i is less than or equal to m, and k first significant feature maps and m second significant feature maps form a first difference feature map.

5. The method of claim 4, wherein calculating the difference between the third image and the second image to obtain a second difference feature map comprises:

inputting the third image into a MobileNet V2 network for processing, and taking feature maps output by the first k convolution blocks in the MobileNet V2 network to obtain k first feature maps corresponding to the third image;

respectively calculating the difference value of the ith first feature map corresponding to the third image and the ith first feature map corresponding to the second image to obtain k third significant feature maps, wherein i is less than or equal to k;

inputting the third image into mbv2_ ca network for processing, and taking feature maps output by the first m convolution blocks in mbv2_ ca network to obtain m second feature maps corresponding to the third image;

and calculating the difference value between the ith second feature map corresponding to the third image and the ith second feature map corresponding to the second image to obtain m fourth significant feature maps, wherein i is less than or equal to m, and the k third significant feature maps and the m fourth significant feature maps form the second difference feature map.

6. The method according to claim 5, wherein the fusing the first difference feature map and the second difference feature map to obtain a fused feature map comprises:

calculating the sum of the ith first significant feature map in the first difference feature map and the ith third significant feature map in the second difference feature map, wherein i is less than or equal to k, and obtaining k first fusion feature maps;

calculating the sum of the ith second significant feature map in the first significant feature map and the ith fourth significant feature map in the second significant feature map, wherein i is less than or equal to m, and obtaining m second fusion feature maps;

calculating the product of the first fusion feature map and the second fusion feature map and the corresponding hyper-parameters to obtain (k + m) fusion hyper-parameter feature maps;

and (k + m) sums of the fusion hyper-parameter feature maps are calculated to obtain a fusion feature map.

7. The method of claim 1, wherein determining the region in which the obstacle is located based on the fused feature map comprises:

comparing the gray value of each pixel in the fusion characteristic graph with a preset gray threshold value;

and taking the area formed by the pixels with the gray value larger than or equal to the gray threshold value as the area where the obstacle is positioned.

8. A short small obstacle detection device, comprising:

the image acquisition module is used for acquiring a first image, a second image and a third image which are synchronously acquired by the first camera, the second camera and the third camera;

the first difference feature calculation module is used for calculating the difference between the first image and the second image to obtain a first difference feature map;

the second difference feature calculation module is used for calculating the difference between the third image and the second image to obtain a second difference feature map;

the difference fusion module is used for fusing the first difference feature map and the second difference feature map to obtain a fusion feature map;

and the post-processing module is used for determining the area where the obstacle is located based on the fusion feature map.

9. A short small obstacle detection device, comprising:

at least one processor; and at least one memory storing instructions executable by the at least one processor;

the instructions are executable by the at least one processor to cause the at least one processor to implement a three-camera based short and small obstacle detection method according to any one of claims 1-7.

10. A robot, characterized in that the robot comprises:

the first camera is used for acquiring a first image;

the second camera is used for acquiring a second image;

the third camera is used for acquiring a third image;

and a short and small obstacle detecting device according to claim 8 or 9.

11. A computer-readable storage medium, characterized in that a computer program is stored thereon, which, when being executed by a processor, implements the three-camera based short and small obstacle detection method according to any one of claims 1 to 7.

Images