CN114299131A

CN114299131A - Three-camera-based short and small obstacle detection method and device and terminal equipment

Info

Publication number: CN114299131A
Application number: CN202111622043.XA
Authority: CN
Inventors: 赖志林; 周东开
Original assignee: Guangzhou Saite Intelligent Technology Co Ltd
Current assignee: Guangzhou Saite Intelligent Technology Co Ltd
Priority date: 2021-12-28
Filing date: 2021-12-28
Publication date: 2022-04-08

Abstract

The embodiment of the invention provides a method, a device and terminal equipment for detecting short and small obstacles based on three cameras, wherein the method comprises the following steps: acquiring a first image, a second image and a third image which are synchronously acquired by a first camera, a second camera and a third camera; calculating the difference between the first image and the second image to obtain a first difference characteristic diagram; calculating the difference between the third image and the second image to obtain a second difference characteristic diagram; fusing the first difference feature map and the second difference feature map to obtain a fused feature map; and determining the area where the obstacle is located based on the fused feature map. The generated difference characteristics in two different directions are fused to form a complete obstacle difference characteristic, so that short and short obstacles are effectively identified, the influence of interference factors such as marked lines and shadows on the identification accuracy is reduced, and the sensing capability of the obstacles is enhanced.

Description

Three-camera-based short and small obstacle detection method and device and terminal equipment

Technical Field

The embodiment of the invention relates to the technical field of automatic driving visual perception, in particular to a three-camera-based short and small obstacle detection method, a three-camera-based short and small obstacle detection device and terminal equipment.

Background

As the vehicle based on the assistant driving and the automatic driving is applied to various scenes, the demand for the detection of the obstacle is increasing. The obstacle is accurately detected, so that the safety of the vehicle in driving can be improved, and the normal running of the vehicle is ensured.

At present, in the automatic driving perception technology, there are two main methods for identifying obstacles, namely visual perception identification and laser perception identification. The visual perception includes monocular vision combined with laser radar, TOF (Time of flight), and binocular vision obstacle avoidance. The obstacles identified by the monocular vision combined with the laser radar and the TOF mainly comprise limited categories such as people and vehicles, and the identification capability of the obstacles outside the limited categories or the obstacles with smaller ground is poor; although the binocular vision is not limited to the categories, the binocular vision is very sensitive to the requirements of illumination, and the recognition effect is poor due to the fact that the binocular vision is slightly bright or dark. As for the laser sensing identification, although the identification of the obstacle is not limited to the obstacle category, the laser point cloud is sparse, and the effect is not good when detecting a short obstacle.

Disclosure of Invention

The embodiment of the invention provides a method and a device for detecting short and small obstacles based on three cameras, a robot and a storage medium, and aims to solve the problem of improving the detection accuracy of the short and small obstacles.

In a first aspect, an embodiment of the present invention provides a method for detecting short and small obstacles based on three cameras, including:

acquiring a first image, a second image and a third image which are synchronously acquired by a first camera, a second camera and a third camera;

calculating the difference between the first image and the second image to obtain a first difference characteristic diagram;

calculating the difference between the third image and the second image to obtain a second difference characteristic diagram;

fusing the first difference feature map and the second difference feature map to obtain a fused feature map;

and determining the area where the obstacle is located based on the fused feature map.

In a second aspect, an embodiment of the present invention further provides a short and small obstacle detection device, including:

the image acquisition module is used for acquiring a first image, a second image and a third image which are synchronously acquired by the first camera, the second camera and the third camera;

the first difference feature calculation module is used for calculating the difference between the first image and the second image to obtain a first difference feature map;

the second difference feature calculation module is used for calculating the difference between the third image and the second image to obtain a second difference feature map;

the difference fusion module is used for fusing the first difference feature map and the second difference feature map to obtain a fusion feature map;

and the post-processing module is used for determining the area where the obstacle is located based on the fusion feature map.

In a third aspect, an embodiment of the present invention further provides a short and small obstacle detection device, including:

at least one processor; and at least one memory storing instructions executable by the at least one processor;

the instructions are executable by the at least one processor to cause the at least one processor to implement the three-camera based short and small obstacle detection method according to the first aspect.

In a fourth aspect, an embodiment of the present invention further provides a robot, including:

the first camera is used for acquiring a first image;

the second camera is used for acquiring a second image;

the third camera is used for acquiring a third image;

and a short and small obstacle detecting device according to the second or third aspect.

In a fifth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the three-camera-based short and small obstacle detection method according to the first aspect.

In this embodiment, a first image, a second image and a third image synchronously acquired by a first camera, a second camera and a third camera are obtained; calculating the difference between the first image and the second image to obtain a first difference characteristic diagram; calculating the difference between the third image and the second image to obtain a second difference characteristic diagram; fusing the first difference characteristic diagram and the second difference characteristic diagram to obtain a fused characteristic diagram; and determining the area where the obstacle is located based on the fused feature map. Through the combination of the three cameras and the two networks, the generated difference characteristics in two different directions are fused to form a complete obstacle difference characteristic, so that short and small obstacles on indoor and outdoor pavements can be effectively identified, the influence of interference factors such as marked lines and shadows on roads on the identification accuracy is reduced, the obstacle difference characteristic can be complemented with general visual perception and laser perception, and the perception capability of obstacles in the automatic driving process is enhanced.

Drawings

Fig. 1A is a flowchart of a three-camera-based short and small obstacle detection method according to an embodiment of the present invention;

fig. 1B is a flowchart illustrating exemplary data processing of a three-camera-based short and small obstacle detection method according to an embodiment of the present invention;

fig. 1C is a fused feature diagram obtained using a three-camera based short and short obstacle detection method for a conical barricade;

fig. 2 is a schematic structural diagram of a short and small obstacle detection device according to a second embodiment of the present invention;

fig. 3 is a schematic structural diagram of a short and small obstacle detection device according to a third embodiment of the present invention;

fig. 4 is a schematic structural diagram of a robot according to a fourth embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.

Example one

Fig. 1A is a flowchart of a three-camera-based short and small obstacle detection method according to an embodiment of the present invention, and fig. 1B is a flowchart of an example of data processing of the three-camera-based short and small obstacle detection method according to an embodiment of the present invention, where this embodiment is applicable to a case of improving detection accuracy of short and small obstacles, the method may be implemented by a three-camera-based short and small obstacle detection apparatus, and the three-camera-based short and small obstacle detection apparatus may be implemented by software and/or hardware, and may be configured in a computer device and a robot, for example, an intelligent robot, a server, a personal computer, and the like, and specifically includes the following steps:

step 101, acquiring a first image, a second image and a third image which are synchronously acquired by a first camera, a second camera and a third camera.

Illustratively, the first camera, the second camera and the third camera are all arranged on the same plane and are all used for acquiring a space scene in front of the current plane. The first camera and the second camera are aligned at a preset distance in the same horizontal direction at intervals, the second camera and the third camera are aligned at a preset distance in the same vertical direction at intervals, and the third camera is inclined downwards by a preset angle so as to obtain an image of a ground part of a larger part. And setting three buffer queues, wherein the images shot and collected by the first camera, the second camera and the third camera are respectively input into each buffer queue, and the images shot at the same time are respectively selected from the three buffer queues to be used as a first image, a second image and a third image.

Illustratively, the first camera and the second camera are symmetrically arranged at an interval of 30cm in the same horizontal direction, and the angle of the first camera and the angle of the second camera are also horizontal to the ground. The second camera and the third camera are arranged at an interval of 40cm in the same vertical direction, and the third camera is arranged above the second camera and is inclined downwards by 20 degrees. First camera, second camera, third camera all last to shoot the scene in camera the place ahead to carry out the collection of image. The first camera, the second camera and the third camera transmit acquired data into different queues of the three buffer queues respectively, images at the same time are obtained from the three buffer queues and are used as a first image, a second image and a third image respectively, and if three images which are completely matched in time do not exist, three images with time difference not exceeding a preset interval threshold value, for example 10ms are selected and used as the first image, the second image and the third image respectively. The resolution ratios of the images acquired by the three cameras are 1028 × 720, and the frame rate is 30 frames/second.

It should be noted that, in the above embodiments, the first camera and the second camera are in the same horizontal direction, and the second camera and the third camera are in the same vertical direction, which is an exemplary illustration of the embodiments of the present invention, in other embodiments of the present invention, the first camera, the second camera, and the third camera may also have other different position setting methods, for example, the first camera and the second camera are in the same horizontal direction, and the third camera is located right above a midpoint of the first camera and the second camera, and the present invention is not limited herein.

In some embodiments of the present invention, step 101 comprises:

step 1011, calculating a first homography matrix according to the four pairs of key mark points in the first image and the second image.

The homography transformation is used for describing the position mapping relation of an object between two images, a homography matrix H between the two images is calculated through coordinates of four pairs of key mark points selected by the two images, then a projective transformation function is called, one image is transformed into data of the other image, and a transformation matrix used for the projective transformation is called as a homography matrix. The homography matrix can be represented by a 3 x 3 nonsingular matrix H as follows:

the homography matrix H is a homogeneous matrix with 8 unknowns, normalizing the last element to 1.

Four pairs of points of the same feature are selected in the first image and the second image as key mark point pairs, and each pair of key mark point pairs is a pixel point pair representing the same feature in the first image and the second image.

Obtaining a first homography matrix H for aligning the first image with the second image by using the coordinate relation of four pairs of key mark points in the first image and the second image according to the following formula₁The value of (c).

In the following embodiments of the present invention, before calculating the first homography matrix, the first image and the second image may be respectively corrected by using the internal reference matrices of the first camera and the second camera, so as to display the image and provide standard position information for navigation. After the first image and the second image are corrected by using the internal reference matrix, the corrected first image and the corrected second image are used for marking key marking point pairs so as to calculate the first homography matrix.

It should be noted that, in the above embodiment, the process of obtaining the first homography matrix through four pairs of key mark points is an exemplary description of the embodiment of the present invention, and in other embodiments of the present invention, there may be other processes of obtaining the first homography matrix through four pairs of key mark points, which is not limited herein.

Step 1012 multiplies the first homography matrix with a matrix comprised of pixel values of the first image to align the first image with the second image.

For points in the first image plane, we use w-1 to normalizePoint values are integrated and a coordinate matrix is formed by two coordinates x and y of image coordinates and the coordinate matrix [ x y 1 ]]^-1Using the coordinate matrix and the first homography matrix H₁Coordinate matrix [ x ' y ' w ']^-1I.e. the first image and the second image are aligned by the following formula.

In some embodiments of the present invention, step 101 comprises:

and 1013, calculating a first homography matrix according to the four pairs of key mark points in the second image and the third image.

Four pairs of points of the same feature are selected in the second image and the third image as key mark point pairs, and each pair of key mark point pairs is a pixel point pair representing the same feature in the second image and the third image.

Obtaining a second homography matrix H for aligning the third image with the second image by using the coordinate relation of the four pairs of key mark points in the second image and the third image according to the following formula₂The value of (c).

In a next embodiment of the present invention, before calculating the second homography matrix, the second image and the third image may be respectively corrected by using the internal reference matrices of the second camera and the third camera, so as to display the image and provide standard position information for navigation. After the second image and the third image are corrected by using the internal reference matrix, the corrected second image and the corrected third image are used for marking key marking point pairs so as to calculate a second homography matrix.

It should be noted that, the process of obtaining the second homography matrix through four pairs of key mark points in the foregoing embodiment is an exemplary description of the embodiment of the present invention, and in other embodiments of the present invention, there may be other processes of obtaining the second homography matrix through four pairs of key mark points, and the present invention is not limited herein.

Step 1014 multiplies the second homography matrix with a matrix comprised of pixel values of the third image to align the third image with the second image.

For points in the second image plane, the point values are normalized using w ═ 1, and a coordinate matrix [ x y 1 ] is formed with the two coordinates x, y of the image coordinates]^-1Using the coordinate matrix and a second homography matrix H₂Coordinate matrix [ x ' y ' w ']^-1I.e. the second image and the third image are aligned by the following formula.

And 102, calculating the difference between the first image and the second image to obtain a first difference characteristic diagram.

The first image and the second image comprise a common part, the first image and the second image are horizontally aligned, and the first difference feature map is generated by calculating the difference between the first image and the second image, so that the features in the horizontal direction in the front area of the current camera can be obtained. For objects with a width, there will be more pronounced features.

It should be noted that, the horizontal direction in the above embodiments is an exemplary description of the embodiments of the present invention, and in other embodiments of the present invention, according to the setting conditions of the first camera and the second camera, the difference between the first image and the second image may also be calculated to obtain features in other directions, and the present invention is not limited herein.

In some embodiments of the present invention, step 102 comprises:

and step 1021, respectively inputting the first image and the second image into a MobileNet V2 network for processing, and taking feature maps output by the first k convolution blocks in the MobileNet V2 network to obtain k first feature maps corresponding to the first image and k first feature maps corresponding to the second image.

The MobileNetV2 network is a network that can significantly reduce model parameters and computational effort while maintaining similar accuracy, now extending the low-dimensional compressed representation of the input to the higher dimension, using lightweight deep convolution for filtering; the features are then projected back into the low-dimensional compressed representation using a linear bottle neck (linear bottle). The deep Convolution in the MobileNetV2 network is 3 × 3 depth Separable Convolution (Depthwise Separable Convolution). The MobileNetV2 network includes a plurality of sequentially connected volume blocks, with the output of a previous volume block being the input of a subsequent volume block.

After the first image is adjusted to a preset size and input into a MobileNet V2 network, taking feature maps output by the first k convolution blocks in the MobileNet V2 network as k first feature maps corresponding to the first image, wherein the first feature maps are feature maps containing image features output by the convolution blocks of the MobileNet V2 network.

And after the second image is adjusted to a preset size and is input into a MobileNet V2 network, taking feature maps output by the first k convolution blocks in the MobileNet V2 network as k first feature maps corresponding to the second image. The number of the first feature maps extracted from the first image and the second image must be the same.

Illustratively, k is 3, after the first image and the second image are adjusted to 512 × 256, the first image and the second image are respectively input into a MobileNetV2 network, and feature maps output by the first 3 convolution blocks are respectively extracted, so as to obtain 3 first feature maps corresponding to the first image and 3 first feature maps corresponding to the second image. They include features of the first image and the second image, respectively.

It should be noted that the value of k in the foregoing embodiments is an exemplary description of the embodiments of the present invention, and in other embodiments of the present invention, k may also be another value, and the present invention is not limited herein.

And step 1022, calculating differences between the ith first feature map corresponding to the first image and the ith first feature map corresponding to the second image respectively to obtain k first significant feature maps, wherein i is less than or equal to k.

The number of the first feature maps extracted from the first image and the second image is the same, the value of i is changed from 1 to k, the difference value of the first feature map corresponding to the first image and the first feature map corresponding to the second image is calculated layer by layer, so that the feature in the overlapped part of the first image and the second image is more obvious, and the feature map obtained after subtraction is called a first significant feature map.

Exemplarily, the value of k is 3, i is set to 1, and the difference between the 1 st first feature map corresponding to the first image and the 1 st first feature map corresponding to the second image is calculated to obtain the 1 st first salient feature map; setting i as 2, and calculating the difference value between the 2 nd first feature map corresponding to the first image and the 2 nd first feature map corresponding to the second image to obtain a 2 nd first significant feature map; and i is set to be 3, calculating the difference value between the 3 rd first feature map corresponding to the first image and the 3 rd first feature map corresponding to the second image to obtain the 3 rd first significant feature map, so that k first significant feature maps are obtained in total.

And step 1023, respectively inputting the first image and the second image into an mbv2_ ca network for processing, and taking feature maps output by the first m convolution blocks in the mbv2_ ca network to obtain m second feature maps corresponding to the first image and m second feature maps corresponding to the second image.

mbv2_ ca network is a network of MobileNet V2 with an attention mechanism attention block added. The attention mechanism can be considered as a resource allocation mechanism, for the original evenly allocated resources, the important units are divided into more points and less points according to the importance degree of the attention object, the unimportant or bad units are divided into less points, in the structural design of the deep neural network, the resources to be allocated by attention are basically weights, and the convolution feature expression capability is effectively improved by adding an attention block. mbv2_ ca network includes a plurality of sequentially connected volume blocks, the output of a previous volume block being the input of a subsequent volume block.

After the first image is adjusted to a preset size and input into an mbv2_ ca network, feature maps output by the first m convolution blocks in the mbv2_ ca network are taken as m second feature maps corresponding to the first image, and the second feature maps are feature maps containing image features output by the convolution blocks in the mbv2_ ca network.

After the second image is adjusted to a preset size and input into an mbv2_ ca network, feature maps output by the first m convolution blocks in the mbv2_ ca network are taken as m second feature maps corresponding to the second image. The number of the second feature maps extracted from the first image and the second image must be the same.

The first image and the second image are subjected to feature extraction by using an mbv2_ ca network to obtain m second feature maps, and the m second feature maps can be combined with k first feature maps obtained by using the MobileNetV2 network feature extraction in the step 1021 to form (k + m) features, so that more different features in the first image and the second image can be conveniently searched.

Illustratively, m is 2, after the first image and the second image are adjusted to 512 × 256, the first image and the second image are respectively input into mbv2_ ca networks, and feature maps output by the first 2 convolution blocks are respectively extracted, so as to obtain 2 second feature maps corresponding to the first image and 2 second feature maps corresponding to the second image. They include features of the first image and the second image, respectively.

It should be noted that the value of m in the foregoing embodiments is an exemplary description of the embodiments of the present invention, and in other embodiments of the present invention, m may also be another value, and the values of m and k may be equal or unequal, which is not limited herein.

Step 1024, calculating a difference value between the ith second feature map corresponding to the first image and the ith second feature map corresponding to the second image to obtain m second significant feature maps, wherein i is less than or equal to m, and the k first significant feature maps and the m second significant feature maps form a first difference feature map.

The number of the second feature maps extracted from the first image and the second image is the same, the value of i is changed from 1 to m, the difference value of the second feature map corresponding to the first image and the second feature map corresponding to the second image is calculated layer by layer, so that the feature in the overlapped part of the first image and the second image is more obvious, and the feature map obtained after subtraction is called a second significant feature map.

Exemplarily, the value of m is 2, i is set to 1, and the difference between the 1 st second feature map corresponding to the first image and the 1 st second feature map corresponding to the second image is calculated to obtain the 1 st second significant feature map; setting i as 2, and calculating the difference value between the 2 nd second feature map corresponding to the first image and the 2 nd second feature map corresponding to the second image to obtain a 2 nd second significant feature map; thus, a total of 2 second salient feature maps are obtained.

The first difference feature map is composed of k first significant feature maps extracted and calculated by a MobileNet V2 network and m second significant feature maps extracted and calculated by a mbv2_ ca network, and the first image and the second image are two images shot by a first camera and a second camera which have horizontal visual difference in the same horizontal direction, so that the first difference feature map generated by processing and calculating the first image and the second image has obvious difference features in the horizontal direction for a wide object.

The shallow layer of the network extracts primary features of the image, such as texture, color, angular points and the like, and the deep layer of the network extracts semantic features which cannot be identified by naked eyes. In the embodiment, primary features similar to textures, colors and the like in each image can be extracted by extracting the feature map of the front k layer of the MobileNetV2 network and the feature map of the front m layer of the mbv2_ ca network, and the primary features are sufficiently used for detecting obstacles, so that the efficiency in the process of detecting short and short obstacles is improved.

And the primary features of the image are extracted by adopting a MobileNet V2 network and a mbv2_ ca network, and the primary features extracted by the two networks are combined to improve the detection accuracy.

It should be noted that the MobileNetV2 network and the mbv2_ ca network used in the above embodiments are exemplary illustrations of the embodiments of the present invention, and in some embodiments of the present invention, other networks may also be used to perform feature extraction on the first image and the second image, and the present invention is not limited herein.

And 103, calculating the difference between the third image and the second image to obtain a second difference characteristic diagram.

The second image and the third image comprise a common part, the second image and the third image are vertically aligned, a second difference feature map is generated by calculating the difference between the second image and the third image, and features in the vertical direction in the area in front of the current camera can be obtained. For objects with height, there will be more obvious features.

In some embodiments of the invention, step 103 comprises:

and step 1031, inputting the third image into a MobileNet V2 network for processing, and taking feature maps output by the first k convolution blocks in the MobileNet V2 network to obtain k first feature maps corresponding to the third image.

And after the third image is adjusted to a preset size and is input into a MobileNet V2 network, taking feature maps output by the first k convolution blocks in the MobileNet V2 network as k first feature maps corresponding to the third image.

Illustratively, k is 3, after the third image is adjusted to 512 × 256, the third image is input into a MobileNetV2 network, and feature maps output by the first 3 convolution blocks are extracted, so as to obtain 3 first feature maps corresponding to the third image. They include a third image.

And step 1032, calculating the difference value between the ith first feature map corresponding to the third image and the ith first feature map corresponding to the second image respectively to obtain k third significant feature maps, wherein i is less than or equal to k.

The number of the first feature maps extracted from the second image and the third image is the same, and the first feature maps are k, and the value of i is changed from 1 to k, and the difference value of the first feature map corresponding to the third image and the first feature map corresponding to the second image is calculated layer by layer, so that the feature of the overlapped part of the third image and the second image is more obviously represented.

Exemplarily, the value of k is 3, i is set to 1, and the difference between the 1 st first feature map corresponding to the third image and the 1 st first feature map corresponding to the second image is calculated to obtain the 1 st third significant feature map; setting i as 2, and calculating the difference value between the 2 nd first feature map corresponding to the third image and the 2 nd first feature map corresponding to the second image to obtain a 2 nd third significant feature map; and i is set to be 3, calculating the difference value between the 3 rd first feature map corresponding to the third image and the 3 rd first feature map corresponding to the second image to obtain a 3 rd third significant feature map, and therefore k third significant feature maps are obtained in total.

And 1033, inputting the third image into mbv2_ ca network for processing, and taking feature maps output by the first m convolution blocks in mbv2_ ca network to obtain m second feature maps corresponding to the third image.

And after the third image is adjusted to a preset size and input into an mbv2_ ca network, taking feature maps output by the first m convolution blocks in the mbv2_ ca network as m second feature maps corresponding to the third image.

Illustratively, m is 2, after the third image is adjusted to 512 × 256, the third image is input into mbv2_ ca network, and feature maps output by the first 2 convolution blocks are extracted, so as to obtain 2 second feature maps corresponding to the third image. They include features of the first image and the second image, respectively.

Step 1034, calculating a difference value between the ith second feature map corresponding to the third image and the ith second feature map corresponding to the second image to obtain m fourth significant feature maps, wherein i is less than or equal to m, and the k third significant feature maps and the m fourth significant feature maps form a second difference feature map.

The number of the second feature maps extracted from the second image and the third image is the same, the value of i is changed from 1 to m, the difference value between the second feature map corresponding to the third image and the second feature map corresponding to the second image is calculated layer by layer, so that the feature in the overlapped part of the third image and the second image is more obvious, and the feature map obtained after subtraction is called a fourth significant feature map.

Exemplarily, the value of m is 2, i is set to 1, and the difference between the 1 st second feature map corresponding to the third image and the 1 st second feature map corresponding to the second image is calculated to obtain the 1 st fourth significant feature map; setting i as 2, and calculating the difference value between the 2 nd second feature map corresponding to the third image and the 2 nd second feature map corresponding to the second image to obtain a 2 nd fourth significant feature map; thus, a total of 2 fourth saliency maps are obtained.

In step 103, k third salient feature maps extracted and calculated by using the MobileNetV2 network and m fourth salient feature maps extracted and calculated by using the mbv2_ ca network constitute a second difference feature map, and the second image and the third image are two images obtained by shooting a second camera and a third camera which have feature differences in vertical vision in the same vertical direction, so that the second difference feature map generated by processing and calculating the second image and the third image has obvious difference features in the vertical direction for a high-height object.

And step 104, fusing the first difference characteristic diagram and the second difference characteristic diagram to obtain a fused characteristic diagram.

The first difference feature map represents features in the horizontal direction, and the second difference feature map represents features in the vertical direction.

The fused feature map generated by fusing the first difference feature map and the second difference feature map can completely represent the remarkable features of the obstacles, so that the features of the obstacles with various shapes, heights and widths can be represented, the detection accuracy is improved, and the obstacles are prevented from being detected due to too narrow obstacles or too short obstacles.

In some embodiments of the present invention, step 104 comprises:

step 1041, calculating a sum of the ith first significant feature map in the first difference feature map and the ith third significant feature map in the second difference feature map, wherein i is less than or equal to k, and obtaining k first fusion feature maps.

The first difference feature map comprises k first salient feature maps, and the second difference feature map comprises k third salient feature maps. They represent the features extracted by the MobileNetV2 network calculation in the horizontal and vertical directions, respectively. And i is changed from 1 to k, and the sum value of the first difference characteristic diagram and the second difference characteristic diagram in the ith layer is calculated layer by layer, so that the characteristics in the horizontal direction and the vertical direction can form the remarkable characteristics of the complete obstacle.

Exemplarily, k is 3, i is set to 1, and the sum of the 1 st first significant feature map and the 1 st third significant feature map is calculated to obtain the 1 st first fused feature map; changing the i into 2, and calculating the sum of the 2 nd first significant feature map and the 2 nd third significant feature map to obtain a 2 nd first fusion feature map; and i is changed into 3, calculating the sum of the 3 rd first significant feature map and the 3 rd third significant feature map to obtain a 3 rd first fusion feature map, and obtaining k first fusion feature maps through the calculation, wherein the first fusion feature map is the complete significant feature of the obstacle obtained by combining the features extracted in the horizontal and vertical directions through a MobileNet V2 network.

Step 1042, calculating a sum of the ith second significant feature map in the first significant feature map and the ith fourth significant feature map in the second significant feature map, wherein i is less than or equal to m, and obtaining m second fusion feature maps.

The first difference feature map comprises m second salient feature maps, and the second difference feature map comprises m fourth salient feature maps. They represent features extracted by the mbv2_ ca network computation in the horizontal and vertical directions, respectively. And i is changed from 1 to m, and the sum value of the first difference characteristic diagram and the second difference characteristic diagram in the ith layer is calculated layer by layer, so that the characteristics in the horizontal direction and the vertical direction can form the remarkable characteristics of the complete obstacle.

Exemplarily, m is 2, i is set to 1, and the sum of the 1 st second significant feature map and the 1 st fourth significant feature map is calculated to obtain the 1 st second fused feature map; changing the i into 2, and calculating the sum of the 2 nd second significant feature map and the 2 nd fourth significant feature map to obtain a 2 nd second fusion feature map; and obtaining m second fusion feature maps through the calculation, wherein the second fusion feature maps are the complete salient features of the obstacle obtained by combining the features extracted in the horizontal and vertical directions through an mbv2_ ca network.

It should be noted that the value of m in the foregoing embodiments is an exemplary description of the embodiments of the present invention, and m may also be another value in other embodiments of the present invention, and the present invention is not limited herein.

Step 1043, calculating the product of the first fused feature map and the second fused feature map and the corresponding hyper-parameters, and obtaining (k + m) fused hyper-parameter feature maps.

In order to more clearly represent the characteristics of the obstacle, (k + m) hyper-parameters are preset and are respectively multiplied by the k first fused characteristic maps and the m second fused characteristic maps to highlight or weaken partial characteristics. Multiplying the k first fusion feature maps obtained in the step 1041 by the corresponding k hyper-parameters to obtain k fusion hyper-parameter feature maps; and multiplying the m second fusion feature maps obtained in the step 1042 by the corresponding m hyper-parameters to obtain m fusion hyper-parameter feature maps. Therefore, (k + m) fusion hyper-parameter feature maps are obtained in total.

Illustratively, 3 first fused feature maps are obtained in step 1041, and 2 second fused feature maps are obtained in step 1042. Multiplying the 3 first fusion feature maps by the 3 hyper-parameters respectively to obtain 3 fusion hyper-parameter feature maps; multiplying the 2 second fusion feature maps by the 2 hyper-parameters respectively to obtain 2 fusion hyper-parameter feature maps; a total of 5 fused hyper-parameter feature maps are obtained.

And step 1044, calculating the sum of the (k + m) fused hyper-parameter feature maps to obtain a fused feature map.

And (k + m) fused hyperparameter characteristic diagrams are summed, namely all the characteristic diagrams of the obstacle after being subjected to hyperparameter adjustment are added to obtain a fused characteristic diagram with better expressive force. The fused characteristic diagram can more accurately represent the characteristics of the obstacles.

And 105, determining the area where the obstacle is located based on the fused feature map.

Fig. 1C is a fused feature map obtained by using a three-camera-based short and small obstacle detection method, and as shown in fig. 1C, the fused feature map obtained through calculation processing has a more obvious region with different continuous gray values, and the region is a region where an obstacle is located.

It should be noted that, in the foregoing embodiment, taking a relatively obvious and continuous region with different gray values as a region where an obstacle is located is an exemplary description of an embodiment of the present invention, in other embodiments of the present invention, a region where an obstacle in a fusion feature map is located may also be determined by other methods, which is not limited herein.

In some embodiments of the invention, step 105 comprises:

step 1051, comparing the gray value of each pixel in the fusion characteristic graph with a preset gray threshold value.

The preset threshold value is used for determining whether there is an area where an obstacle exists.

And comparing the gray value of each pixel in the fusion characteristic image with a preset gray threshold value, and detecting the region with the obstacle according to the preset gray threshold value.

And step 1053, taking the area formed by the pixels with the gray value larger than or equal to the gray threshold value as the area where the obstacle is located.

And when a plurality of pixel points with gray values larger than or equal to the gray threshold value form an area in the fusion characteristic graph, the area is regarded as the area where the barrier is located.

Illustratively, the preset grayscale threshold is 200, and when the grayscale values of a plurality of pixel points in the fusion feature map are all greater than or equal to 200, the region formed by the pixel points is regarded as the region where the obstacle is located, that is, the obstacle has been detected.

It should be noted that the preset grayscale threshold 200 in the above embodiment is an exemplary illustration of the embodiment of the present invention, and in other embodiments of the present invention, the grayscale threshold may also be determined according to the situation and accuracy requirement applied in the actual scene, and the present invention is not limited herein.

It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the illustrated order of acts, as some steps may occur in other orders or concurrently in accordance with the embodiments of the present invention. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that no particular act is required to implement the invention.

Example two

Fig. 2 is a block diagram of a structure of a short and small obstacle detection device according to a second embodiment of the present invention, which may specifically include the following modules:

the image acquisition module 201 is configured to acquire a first image, a second image and a third image which are acquired by a first camera, a second camera and a third camera synchronously;

a first difference feature calculating module 202, configured to calculate a difference between the first image and the second image to obtain a first difference feature map;

a second difference feature calculating module 203, configured to calculate a difference between the third image and the second image to obtain a second difference feature map;

a difference fusion module 204, configured to fuse the first difference feature map and the second difference feature map to obtain a fusion feature map;

and the post-processing module 205 is configured to determine an area where the obstacle is located based on the fused feature map.

In some embodiments of the present invention, the image obtaining module 201 includes:

a first homography matrix generation submodule, configured to calculate a first homography matrix according to four pairs of key mark point pairs in the first image and the second image, where each pair of key mark point pairs is a pixel point pair representing the same feature in the first image and the second image;

a first image alignment module for multiplying the first homography matrix with a matrix consisting of pixel values of the first image to align the first image and the second image.

a second homography matrix generation submodule, configured to calculate a second homography matrix according to four pairs of key mark point pairs in the second image and the third image, where each pair of key mark point pairs is a pixel point pair representing the same feature in the second image and the third image;

a third image alignment module for multiplying the second homography matrix with a matrix consisting of pixel values of the third image to align the third image with the second image.

In some embodiments of the present invention, the first difference feature calculating module 202 includes:

a first feature map generation sub-module, configured to input the first image and the second image into a MobileNetV2 network respectively for processing, and obtain k first feature maps corresponding to the first image and k first feature maps corresponding to the second image by using feature maps output by the first k convolution blocks in the MobileNetV2 network;

a first salient feature map generation submodule, configured to calculate a difference between an ith first feature map corresponding to the first image and an ith first feature map corresponding to the second image, respectively, to obtain k first salient feature maps, where i is less than or equal to k;

a second feature map generation submodule, configured to input the first image and the second image into mbv2_ ca network respectively for processing, and take feature maps output by previous m convolution blocks in the mbv2_ ca network to obtain m second feature maps corresponding to the first image and m second feature maps corresponding to the second image;

and the second significant feature map generation submodule is used for calculating a difference value between an ith second feature map corresponding to the first image and an ith second feature map corresponding to the second image to obtain m second significant feature maps, wherein i is less than or equal to m, and k first significant feature maps and m second significant feature maps form a first difference feature map.

In some embodiments of the present invention, the second difference feature calculating module 203 includes:

a first feature map generation submodule, configured to input the third image into a MobileNetV2 network for processing, and obtain k first feature maps corresponding to the third image by taking feature maps output by the first k convolution blocks in the MobileNetV2 network;

a third significant feature map generation submodule, configured to calculate a difference between an ith first feature map corresponding to the third image and an ith first feature map corresponding to the second image, respectively, to obtain k third significant feature maps, where i is less than or equal to k;

a second feature map generation submodule, configured to input the third image into mbv2_ ca network for processing, and take feature maps output by previous m convolution blocks in mbv2_ ca network to obtain m second feature maps corresponding to the third image;

and the fourth significant feature map generation submodule is used for calculating a difference value between an ith second feature map corresponding to the third image and an ith second feature map corresponding to the second image to obtain m fourth significant feature maps, wherein i is less than or equal to m, and the k third significant feature maps and the m fourth significant feature maps form the second difference feature map.

In some embodiments of the present invention, the difference fusion module 204 includes:

a first fused feature map generation submodule, configured to calculate a sum of an ith first significant feature map in the first difference feature map and an ith third significant feature map in the second difference feature map, where i is less than or equal to k, so as to obtain k first fused feature maps;

a second fused feature map generation submodule, configured to calculate a sum of an ith second significant feature map in the first significant feature map and an ith fourth significant feature map in the second significant feature map, where i is less than or equal to m, to obtain m second fused feature maps;

the fusion hyper-parameter feature map generation sub-module is used for calculating the product of the first fusion feature map and the second fusion feature map and corresponding hyper-parameters to obtain (k + m) fusion hyper-parameter feature maps;

and the fusion feature map generation submodule is used for calculating the sum of (k + m) fusion hyper-parameter feature maps to obtain a fusion feature map.

In some embodiments of the present invention, the post-processing module 205 comprises:

the gray level comparison submodule is used for comparing the gray level value of each pixel in the fusion characteristic diagram with a preset gray level threshold value;

and the obstacle judgment submodule is used for taking an area formed by the pixels with the gray values larger than or equal to the gray threshold as an area where the obstacle is positioned.

The three-camera-based short and small obstacle detection device provided by the embodiment of the invention can execute the three-camera-based short and small obstacle detection method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.

EXAMPLE III

Fig. 3 is a schematic structural diagram of a short and small obstacle detection device according to a third embodiment of the present invention. Fig. 3 shows a block diagram of an exemplary short obstacle detection device 12 suitable for implementing an embodiment of the present invention. The short and small obstacle detecting device 12 shown in fig. 3 is only an example, and should not bring any limitation to the function and the range of use of the embodiment of the present invention.

As shown in fig. 3, the short and small obstacle detecting apparatus 12 is in the form of a general-purpose computing device. The components of the low and small obstacle detection device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.

Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

Low and small obstacle detection device 12 typically includes a variety of computer system readable media. These media may be any available media that can be accessed by low short obstacle detection device 12, including volatile and non-volatile media, removable and non-removable media.

The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. The low short obstacle detection device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 3, and commonly referred to as a "hard drive"). Although not shown in FIG. 3, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.

A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.

Short small obstacle detection apparatus 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), with one or more devices that enable a user to interact with short small obstacle detection apparatus 12, and/or with any device (e.g., network card, modem, etc.) that enables short small obstacle detection apparatus 12 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 22. Also, low and small obstacle detection device 12 may also communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via network adapter 20. As shown, the network adapter 20 communicates with the other modules of the low short obstacle detection device 12 via the bus 18. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the short obstacle detection device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

The processing unit 16 executes various functional applications and data processing by executing programs stored in the system memory 28, for example, implementing a low and small obstacle detection method provided by an embodiment of the present invention.

Example four

Fig. 4 is a schematic structural diagram of a robot according to a fourth embodiment of the present invention, which may specifically include the following devices:

a first camera 401, configured to acquire a first image;

a second camera 402 for acquiring a second image;

a third camera 403, configured to acquire a third image;

a short and small obstacle detecting device 12.

In order to make the embodiments of the present application better understood by those skilled in the art, an example of the structure of a robot is described in the present specification.

Illustratively, the first camera 401, the second camera 402 and the third camera 403 are all connected with the short and small obstacle detection device 12, and the acquired first image, second image and third image are transmitted to the short and small obstacle detection device 12 for processing. Referring to fig. 3 and 4, the first camera 401, the second camera 402, and the third camera 403 correspond to the external device 14 in fig. 3, and are connected to the short and small obstacle device 12 through the I/O interface 22.

Specifically, the robot may be a robot with different applications and different forms, such as a delivery robot, a cleaning robot, a wheel robot, or a mobile robot, which is only an example and not a limitation.

The robot provided by the embodiment of the invention can execute the processing method of the robot provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.

EXAMPLE five

The fifth embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the processes of the three-camera-based short and small obstacle detection method are implemented, and the same technical effects can be achieved, and are not described herein again to avoid repetition.

A computer readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A short and small obstacle detection method based on three cameras is characterized by comprising the following steps:

2. The method of claim 1, wherein the obtaining the first image, the second image, and the third image synchronously captured by the first camera, the second camera, and the third camera comprises:

calculating a first homography matrix according to four pairs of key mark point pairs in the first image and the second image, wherein each pair of key mark point pairs is a pixel point pair which represents the same characteristic in the first image and the second image;

multiplying the first homography matrix with a matrix consisting of pixel values of the first image to align the first image and the second image.

3. The method of claim 1, wherein the obtaining the first image, the second image, and the third image synchronously captured by the first camera, the second camera, and the third camera comprises:

calculating a second homography matrix according to four pairs of key mark point pairs in the second image and the third image, wherein each pair of key mark point pairs is a pixel point pair which represents the same characteristic in the second image and the third image;

multiplying the second homography matrix with a matrix consisting of pixel values of the third image to align the third image and the second image.

4. The method of claim 1, wherein calculating the difference between the first image and the second image to obtain a first difference feature map comprises:

inputting the first image and the second image into a MobileNet V2 network for processing, and taking feature maps output by the first k convolution blocks in the MobileNet V2 network to obtain k first feature maps corresponding to the first image and k first feature maps corresponding to the second image;

respectively calculating the difference value of the ith first feature map corresponding to the first image and the ith first feature map corresponding to the second image to obtain k first significant feature maps, wherein i is less than or equal to k;

inputting the first image and the second image into an mbv2_ ca network respectively for processing, and taking feature maps output by the first m convolution blocks in the mbv2_ ca network to obtain m second feature maps corresponding to the first image and m second feature maps corresponding to the second image;

and calculating the difference value between the ith second feature map corresponding to the first image and the ith second feature map corresponding to the second image to obtain m second significant feature maps, wherein i is less than or equal to m, and k first significant feature maps and m second significant feature maps form a first difference feature map.

5. The method of claim 4, wherein calculating the difference between the third image and the second image to obtain a second difference feature map comprises:

inputting the third image into a MobileNet V2 network for processing, and taking feature maps output by the first k convolution blocks in the MobileNet V2 network to obtain k first feature maps corresponding to the third image;

respectively calculating the difference value of the ith first feature map corresponding to the third image and the ith first feature map corresponding to the second image to obtain k third significant feature maps, wherein i is less than or equal to k;

inputting the third image into mbv2_ ca network for processing, and taking feature maps output by the first m convolution blocks in mbv2_ ca network to obtain m second feature maps corresponding to the third image;

and calculating the difference value between the ith second feature map corresponding to the third image and the ith second feature map corresponding to the second image to obtain m fourth significant feature maps, wherein i is less than or equal to m, and the k third significant feature maps and the m fourth significant feature maps form the second difference feature map.

6. The method according to claim 5, wherein the fusing the first difference feature map and the second difference feature map to obtain a fused feature map comprises:

calculating the sum of the ith first significant feature map in the first difference feature map and the ith third significant feature map in the second difference feature map, wherein i is less than or equal to k, and obtaining k first fusion feature maps;

calculating the sum of the ith second significant feature map in the first significant feature map and the ith fourth significant feature map in the second significant feature map, wherein i is less than or equal to m, and obtaining m second fusion feature maps;

calculating the product of the first fusion feature map and the second fusion feature map and the corresponding hyper-parameters to obtain (k + m) fusion hyper-parameter feature maps;

and (k + m) sums of the fusion hyper-parameter feature maps are calculated to obtain a fusion feature map.

7. The method of claim 1, wherein determining the region in which the obstacle is located based on the fused feature map comprises:

comparing the gray value of each pixel in the fusion characteristic graph with a preset gray threshold value;

and taking the area formed by the pixels with the gray value larger than or equal to the gray threshold value as the area where the obstacle is positioned.

8. A short small obstacle detection device, comprising:

9. A short small obstacle detection device, comprising:

the instructions are executable by the at least one processor to cause the at least one processor to implement a three-camera based short and small obstacle detection method according to any one of claims 1-7.

10. A robot, characterized in that the robot comprises:

the first camera is used for acquiring a first image;

the second camera is used for acquiring a second image;

the third camera is used for acquiring a third image;

and a short and small obstacle detecting device according to claim 8 or 9.

11. A computer-readable storage medium, characterized in that a computer program is stored thereon, which, when being executed by a processor, implements the three-camera based short and small obstacle detection method according to any one of claims 1 to 7.