TWI807904B - Method for training depth identification model, method for identifying depth of images and related devices - Google Patents

Abstract

The present application relates to an image analysis technology and provides a method for training a depth identification model, a method for identifying depth of images and related devices. This application includes: obtaining a plurality of static objects and a plurality of dynamic objects by performing an instance segmentation on a first image and a second image, and obtaining a dynamic position of each of the plurality of dynamic objects; selecting a plurality of target dynamic objects and a plurality of characteristic dynamic objects from the plurality of dynamic objects according to a number of pixels and a preset position; generating a target image and a target projection image according to a dynamic pose matrix of the plurality of target dynamic objects and characteristic dynamic objects corresponding to the plurality of target dynamic objects, a static pose matrix of the plurality of static objects, the dynamic position, a preset threshold matrix and a preset initial projection image; generating a depth identification model according to an initial depth image, the target image, the target projection image and a preset depth recognition network, and obtaining an identification result of an image to be identified by inputting the image to be identified into the depth identification model.

Description

Depth recognition model training method, image depth recognition method and related equipment

The invention relates to the field of image processing, in particular to a depth recognition model training method, an image depth recognition method and related equipment.

In the current scheme of deep recognition of vehicle images, the training images can be used to train the deep network. However, since the training images used include dynamic objects, the trained depth recognition model will not be able to accurately identify the depth information of the vehicle image, making it difficult to determine the real distance between the vehicle and various objects or obstacles in the surrounding environment, thereby affecting driving safety.

In view of the above, it is necessary to provide a depth recognition model training method, an image depth recognition method and related equipment, which solve the technical problem of inaccurate recognition of depth information of vehicle images.

The present application provides an image depth recognition method, the image depth recognition method comprising: acquiring a first image and a second image, performing instance segmentation on the first image based on an instance segmentation network to obtain a first static object corresponding to the first image, a plurality of first dynamic objects, and a first dynamic position of each first dynamic object, and performing instance segmentation on the second image based on the instance segmentation network to obtain a second static object and a plurality of second dynamic objects corresponding to the second image, selecting a plurality of target dynamic objects from the plurality of first dynamic objects based on the number of pixels and preset positions of each first dynamic object, and based on the position of each second dynamic object The number of pixels and the preset position from the plurality of second Select a plurality of characteristic dynamic objects among the dynamic objects, identify whether there is a corresponding characteristic dynamic object for each target dynamic object, and determine the corresponding target dynamic object and characteristic dynamic object as the recognition object, according to the dynamic pose matrix corresponding to the recognition object, the static pose matrix corresponding to the first static object and the second static object, and the preset threshold matrix, identify the object state of the target dynamic object in the recognition object, generate a target image according to the object state, the first dynamic position, and the first image, and generate an initial projection image corresponding to the object state, the first dynamic position, and the first image For the target projection image, adjust the obtained depth recognition network based on the gradient error between the initial depth image corresponding to the first image and the target image and the photometric error between the target projection image and the target image to obtain a depth recognition model.

According to an optional embodiment of the present application, the instance segmentation network includes a feature extraction layer, a classification layer, and a mapping layer. The instance-based segmentation network performs instance segmentation on the first image to obtain the first static object corresponding to the first image, a plurality of first dynamic objects, and the first dynamic position of each first dynamic object. Segmenting the standardized image to obtain a rectangular area corresponding to each pixel in the initial feature map; classifying the initial feature map based on the classification layer to obtain a predicted probability that each pixel in the initial feature map belongs to a first preset category; determining a pixel corresponding to a predicted probability greater than a preset threshold in the initial feature map as a target pixel, and determining a plurality of rectangular areas corresponding to multiple target pixels as a plurality of feature areas; Divide a plurality of said mapping areas to obtain a plurality of divided areas corresponding to each mapped area; determine a central pixel point in each divided area, and calculate a pixel value of said central pixel point; perform pooling processing on a plurality of pixel values corresponding to a plurality of said central pixel points, to obtain a mapping probability value corresponding to each mapped area; restore said plurality of mapped areas, and splice the restored multiple mapped areas to obtain a target feature map, The mapping probability value, the plurality of restored mapping areas and the second preset category generate the first static object corresponding to the first image, the plurality of first dynamic objects, and a first dynamic position of each first dynamic object.

According to an optional embodiment of the present application, the generating the first static object corresponding to the first image, the plurality of first dynamic objects, and the first dynamic position of each first dynamic object according to the target feature map, the mapping probability value, the restored plurality of mapping areas, and the second preset category includes: classifying each pixel of the target feature map according to the mapping probability value and the second preset category, obtaining the pixel category of each pixel in the restored mapping area, determining an area composed of a plurality of pixel points corresponding to the same pixel category in the restored mapping area as the first object, and obtaining the first object. The pixel coordinates of all pixel points in the object, and determining the pixel coordinates as the first position corresponding to the first object, dividing the plurality of the first objects into the plurality of first dynamic objects and the first static objects according to preset rules, and determining the first position corresponding to each first dynamic object as the first dynamic position.

According to an optional embodiment of the present application, the selecting a plurality of target dynamic objects from the plurality of first dynamic objects based on the number of pixels and preset positions of each first dynamic object includes: counting the number of pixels contained in each first dynamic object, sorting the plurality of first dynamic objects according to the number of pixels, and selecting the first dynamic objects whose sorted number of pixels are at the preset position as the plurality of target dynamic objects.

According to an optional embodiment of the present application, the identifying whether each target dynamic object has a corresponding feature dynamic object includes: obtaining multiple target element information of each target dynamic object, and obtaining feature element information corresponding to each target element information in the same category of feature dynamic objects, matching each target element information with the corresponding feature element information to obtain a matching value between the target dynamic object and the feature dynamic object of the same type, and if the matching value is within a preset range, it is determined that there is a corresponding feature dynamic object in the target dynamic object.

According to an optional embodiment of the present application, according to the dynamic pose matrix corresponding to the identified object, the static pose matrix corresponding to the first static object and the second static object, and a preset threshold Identifying the object state of the target dynamic object in the recognition object by matrix includes: subtracting each matrix element in the static pose matrix from the corresponding matrix element in the dynamic pose matrix corresponding to the recognition object to obtain a pose difference value, taking an absolute value of the pose difference value to obtain the pose absolute value in the static pose matrix, arranging the pose absolute values according to the element position of each pose absolute value in the static pose matrix to obtain a pose absolute value matrix, and calculating each pose absolute value in the pose absolute value matrix Compared with the corresponding pose threshold value in the preset threshold value matrix, if there is at least one pose absolute value greater than the corresponding pose threshold value in the pose absolute value matrix, it is determined that the object state of the target dynamic object in the recognition object is moving, or, if all the pose absolute values in the pose absolute value matrix are less than or equal to the corresponding threshold value, then it is determined that the object state of the target dynamic object in the recognition object is static.

According to an optional embodiment of the present application, generating the target image according to the object state, the first dynamic position, and the first image includes: if the object state of any target dynamic object in the identified objects is moving, then performing mask processing on the any target dynamic object in the first image based on the first dynamic position of the any target dynamic object to obtain the target image; or, if the object states of all target dynamic objects in the identified objects are stationary, then determining the first image as the target image.

According to an optional embodiment of the present application, the adjusting the depth recognition network based on the gradient error between the initial depth image corresponding to the first image and the target image and the photometric error between the projected target image and the target image to obtain the depth recognition model includes: calculating a depth loss value of the depth recognition network based on the gradient error and the photometric error, adjusting the depth recognition network based on the depth loss value until the depth loss value drops to a minimum, and obtaining the depth recognition model.

The present application provides an image depth recognition method. The image depth recognition method includes: acquiring an image to be recognized, inputting the image to be recognized into a depth recognition model, and obtaining a target depth image of the image to be recognized and depth information of the image to be recognized. The depth recognition model is obtained by executing the depth recognition model training method described above.

The present application provides a computer device, which includes: a memory storing at least one instruction; and a processor executing the at least one instruction to implement the depth recognition model training method or the image depth recognition method.

The present application provides a computer-readable storage medium, wherein at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is executed by a processor in a computer device to implement the depth recognition model training method or the image depth recognition method.

It can be seen from the above technical solution that the present application performs instance segmentation on the first image to obtain the first static object corresponding to the first image, multiple first dynamic objects and the first dynamic position of each first dynamic object, and select multiple target dynamic objects from the multiple first dynamic objects based on the number of pixels and preset positions of each first dynamic object, and can select multiple target dynamic objects from the multiple first dynamic objects. Since the number of the multiple first dynamic objects is reduced, it is possible to improve the training speed of the depth recognition network, identify whether each target dynamic object has a corresponding characteristic dynamic object, and select the second image and For each target dynamic object with the same feature dynamic object, by calculating the dynamic pose matrix of each target dynamic object and the same feature dynamic object, and comparing the dynamic pose matrix with the preset threshold value matrix, it can be determined whether the state of each target dynamic object in the first image is moving, and a target image is generated according to the state of the target dynamic object in the recognition object, the first dynamic position and the first image, and the target dynamic object that moves in the first image can be filtered based on the first dynamic position, and the target image is generated. As a result, the depth value of the pixel corresponding to the target dynamic object in the initial depth image changes, and by filtering out the target dynamic object whose state is moving in the target image, the depth value is not used for calculation when calculating the loss value, and the impact of the moving target dynamic object on the calculation loss value can be avoided. The target image retains the static target dynamic object, which can retain more image information of the first image. can improve the recognition of the deep recognition model Do not be accurate.

1: Computer equipment

2: Shooting equipment

12: Storage

13: Processor

101~107: Steps

108~109: Steps

FIG. 1 is a diagram of an application environment provided by an embodiment of the present application.

Fig. 2 is a flow chart of a method for training a depth recognition model provided by an embodiment of the present application.

FIG. 3 is a schematic diagram of a pixel coordinate system and a camera coordinate system provided by an embodiment of the present application.

Fig. 4 is a flowchart of an image depth recognition method provided by an embodiment of the present application.

FIG. 5 is a schematic structural diagram of a computer device provided by an embodiment of the present application.

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

As shown in FIG. 1 , it is an application environment diagram provided by the embodiment of the present application. The depth recognition model training method and the image depth recognition method can be applied to one or more computer equipment 1, and the computer equipment 1 communicates with the shooting device 2, and the shooting device 2 can be a monocular camera or other equipment for shooting.

The computer device 1 is a device that can automatically perform parameter value calculation and/or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to: microprocessor, Application Specific Integrated Circuit (ASIC), Programmable Gate Array (Field-Programmable Gate Array, FPGA), Digital Signal Processor (Digital Signal Processor, DSP), embedded devices, etc. The computer device 1 may be any electronic product capable of man-machine interaction with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (Personal Digital Assistant, PDA), a game console, an interactive network television (Internet Protocol Television, IPTV), a wearable smart device, and the like.

The computer equipment 1 may also include network equipment and/or user equipment. Wherein, the network device includes, but is not limited to, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on Cloud Computing.

The network where the computer device 1 is located includes, but is not limited to: Internet, wide area network, metropolitan area network, local area network, virtual private network (Virtual Private Network, VPN) and so on.

As shown in FIG. 2 , it is a flow chart of the depth recognition model training method provided by the embodiment of the present application. According to different requirements, the order of each step in the flowchart can be adjusted according to actual detection requirements, and some steps can be omitted. The execution body of the method is a computer device, such as the computer device 1 shown in FIG. 1 .

Step 101, acquiring a first image and a second image.

In at least one embodiment of the present application, the first image and the second image are three primary color (Red Green Blue, RGB) images of adjacent frames, and the generation time of the second image is longer than the generation time of the first image. The first image and the second image may include initial objects such as vehicles, ground, pedestrians, sky, trees, etc., and the first image and the second image include the same initial object.

In at least one embodiment of the present application, the obtaining the image to be recognized by the computer device includes: the computer device controls the shooting device to shoot the target scene to obtain the first image, and shoots the target scene again after a preset time interval to obtain the second image. Wherein, the photographing device may be a monocular camera, and the target scene may include target objects such as vehicles, ground, and pedestrians. It can be understood that the preset time is very small, for example, the preset time may be 10ms.

Step 102: Perform instance segmentation on the first image based on an instance segmentation network to obtain a first static object corresponding to the first image, a plurality of first dynamic objects, and a first dynamic position of each first dynamic object, and perform instance segmentation on the second image based on the instance segmentation network to obtain a second static object and a plurality of second dynamic objects corresponding to the second image.

In at least one embodiment of the present application, the first dynamic object and the second dynamic object refer to objects that can move. For example, the first dynamic object and the second dynamic object can be pedestrians and vehicles. The first static object and the second static object refer to non-movable objects. For example, the first static object and the second static object can be trees, the ground, etc.

In at least one embodiment of the present application, the instance segmentation network includes a feature extraction layer, a classification layer, and a mapping layer. The computer device performs instance segmentation on the first image based on the instance segmentation network to obtain the first static object corresponding to the first image, a plurality of first dynamic objects, and the first dynamic position of each first dynamic object. The multiple relationship between the dimensions of the standardized image and the convolution step in the feature extraction layer are used to segment the standardized image to obtain a rectangular area corresponding to each pixel in the initial feature map. Further, the computer device classifies the initial feature map based on the classification layer to obtain a predicted probability that each pixel in the initial feature map belongs to a first preset category. Further, the computer device determines a pixel corresponding to a predicted probability greater than a preset threshold in the initial feature map as a target pixel, and determines multiple rectangular areas corresponding to multiple target pixels as A plurality of feature regions, further, the computer device maps each feature region to the initial feature map based on the mapping layer, and obtains a mapping region corresponding to each feature region in the initial feature map, and further, the computer device divides a plurality of the map regions based on a preset number to obtain a plurality of divided regions corresponding to each mapped region, further, the computer device determines the central pixel in each divided region, and calculates the pixel value of the central pixel point, and further, the computer device performs pooling processing on a plurality of pixel values corresponding to the plurality of central pixel points to obtain each mapped region For the corresponding mapping probability value, further, the computer device restores the plurality of mapping areas, and stitches the restored multiple mapping areas to obtain a target feature map. Further, the computer device generates the first static object corresponding to the first image, the plurality of first dynamic objects, and the first dynamic position of each first dynamic object according to the target feature map, the mapping probability value, the restored multiple mapping areas, and the second preset category.

Wherein, the normalization process includes clipping, the shape of the normalized image is usually a square, and the feature extraction layer includes a convolution layer, a batch normalization layer, a pooling layer, and the like. Such as the special The extraction layer can be the VGG network after removing the fully connected layer. Wherein, the pixel value of the central pixel point is calculated through a bilinear interpolation method, and the bilinear interpolation method is a prior art, which will not be repeated in this application. The mapping layer may be an ROI Align layer. The first default category can be customized. For example, the first preset category may be foreground or background. The classification layer can be a fully connected layer and a softmax layer. The preset threshold can be set by itself, which is not limited in this application. The preset number can be set by itself, which is not limited in this application. The second preset category can be set according to the target objects appearing in the target scene, which is not limited in the present application. For example, the second preset category may include, but not limited to: cars, passenger cars, roads, pedestrians, street lamps, sky and buildings, and so on.

In this embodiment, the instance segmentation network further includes a fully convolutional neural network, and the plurality of mapping regions are restored based on the fully convolutional neural network.

Specifically, the computer device segmenting the standardized image based on the multiple relationship between the size of the initial feature map and the size of the normalized image and the convolution step in the feature extraction layer, and obtaining a rectangular area corresponding to each pixel in the initial feature map includes: the computer device segmenting the standardized image by using the product of the multiple relationship and the convolution step as width and height to obtain a rectangular area corresponding to each pixel in the initial feature map.

For example, the size of the standardized image is 800*800, the size of the initial feature map is 32*32, the convolution step is 4, the multiple relationship between the size 32*32 of the initial feature map and the size 800*800 of the standardized image is 25, the product of the multiple relationship and the convolution step is 100, and the computer device divides the standardized image into 8 rectangular areas, each of which has a size of 100*100.

Specifically, the preset number includes a first preset number and a second preset number, and the computer device divides the plurality of mapping areas based on the preset number to obtain a plurality of divided areas corresponding to each mapping area. Divide the area. Wherein, the first preset number and the second preset number can be set by yourself, which is not limited in this application. For example, the first preset number may be 7*7, and the second preset number may be 2*2. For example, when the size of the mapping area is 14*14, the mapping area is divided into 7*7 intermediate areas on average, and the size of each intermediate area is 2*2, and each intermediate area is divided into 2*2 divided areas on average, and the size of each divided area is about 0.5*0.5.

In this embodiment, the instance segmentation network also outputs the position of the first static object, the position of the second static object, the category of each target dynamic object, the category of the first static object, the category of the second static object, and the category of each feature dynamic object. Through the above embodiments, the first image and the second image are segmented based on the instance segmentation network, and each initial object in the first image and the second image can be distinguished according to a position, so that each initial object can be processed based on the position.

Specifically, the computer device generating the first static object corresponding to the first image, the plurality of first dynamic objects, and the first dynamic position of each first dynamic object according to the target feature map, the mapping probability value, the restored multiple mapping regions, and the second preset category includes: the computer device classifying each pixel of the target feature map according to the mapping probability value and the second preset category to obtain the pixel category of each pixel in the restored mapping region. For the first object, further, the computer device acquires pixel coordinates of all pixels in the first object, and determines the pixel coordinates as the first position corresponding to the first object, further, the computer device divides the plurality of first objects into the plurality of first dynamic objects and the first static object according to preset rules, and determines the first position corresponding to each first dynamic object as the first dynamic position.

Wherein, the preset rules determine the movable initial objects belonging to means of transportation, people or animals as the plurality of movable first dynamic objects, and determine the immovable initial objects belonging to plants and fixed objects as the first static objects. For example, pedestrians, kittens, puppies, bicycles and cars that can move are determined as the plurality of first dynamic objects, and will not be able to move. Initial objects such as trees, street lamps, and buildings are determined as the first static object.

In this embodiment, the division method of the plurality of second dynamic objects is basically the same as that of the plurality of first dynamic objects, and the division method of the second static objects is basically the same as that of the first static objects, so this application will not repeat them here.

Step 103, selecting a plurality of target dynamic objects from the plurality of first dynamic objects based on the number of pixels and the preset position of each first dynamic object, and selecting a plurality of characteristic dynamic objects from the plurality of second dynamic objects based on the number of pixels of each second dynamic object and the preset position.

In at least one embodiment of the present application, the computer device selecting a plurality of target dynamic objects from the plurality of first dynamic objects based on the number of pixels and preset positions of each first dynamic object includes: the computer device counts the number of pixels contained in each first dynamic object, and sorts the plurality of first dynamic objects according to the number of pixels. Wherein, the preset position can be set by itself. For example, the preset positions may be the first five.

In this embodiment, the selection method of the plurality of feature dynamic objects is basically the same as the selection method of the plurality of target dynamic objects, so the present application will not repeat them here. In at least one embodiment of the present application, the generation process of the second static image is basically the same as that of the first static image, and the generation process of the second dynamic image is basically the same as that of the first dynamic image, so this application will not repeat them here.

Through the above implementation, the plurality of target dynamic objects and the plurality of feature dynamic objects are selected based on the number of pixels and the preset positions, and the number of the plurality of first dynamic objects is reduced, so the training speed of the depth recognition network can be improved.

Step 104, identify whether each target dynamic object has a corresponding characteristic dynamic object, and determine the corresponding target dynamic object and characteristic dynamic object as the identification object.

In at least one embodiment of the present application, the computer device identifying whether there is a corresponding characteristic dynamic object for each target dynamic object includes: the computer device obtaining each target dynamic object multiple target element information, and obtain the feature element information corresponding to each target element information in the feature dynamic object of the same category, further, the computer device matches each target element information with the corresponding feature element information, and obtains a matching value between the target dynamic object and the feature dynamic object of the same category, and if the matching value is within a preset interval, the computer device determines that there is a corresponding feature dynamic object in the target dynamic object.

Wherein, the plurality of target element information and feature element information corresponding to each target element information may be acquired based on a target tracking algorithm. The target tracking algorithm is an existing technology, and the present application does not repeat it here. The preset interval can be set by itself, which is not limited in this application.

In this embodiment, the plurality of target element information may be parameters of features of the target dynamic object, and the plurality of feature element information may be parameters of features of the same type of feature dynamic objects. For example, when the target dynamic object is a car, the plurality of target element information may be the size of the car, the texture of the car, the position of the car, the outline of the car, and so on. Since the parameters of each target element information and corresponding feature element information are different, the matching processing methods are also different. The manner of the matching processing includes operations such as subtraction, addition, and weighting. For example, the target dynamic object in the first image and the characteristic dynamic object in the second image are both cars. The length of the car in the first image is 4.8 meters and the width is 1.65 meters. The length of the car in the second image is 4.7 meters and the width is 1.6 meters. The length of the car in the first image is 4.8 meters. When the first preset interval corresponding to the value is [0,0.12], and the second preset interval corresponding to the second matching value is [0,0.07], since the first matching value is in the first preset interval and the second matching value is in the second preset interval, the car in the second image is the same car as the car in the first image.

Through the above implementation method, multiple target element information of each target dynamic object and feature element information corresponding to each target element information in the same type of feature dynamic object are obtained, and the same type of feature dynamic object is selected, and the feature dynamic object and the target dynamic object can be identified more quickly. By selecting multiple target element information and combining each target element information with the corresponding feature element information Matching pixel information can more comprehensively extract the features of the target dynamic object and the feature dynamic objects of the same category, eliminate reasonable errors and improve matching accuracy.

Step 105 , according to the dynamic pose matrix corresponding to the recognized object, the static pose matrix corresponding to the first static object and the second static object, and a preset threshold matrix, identify the object state of the target dynamic object in the recognized object.

In at least one embodiment of the present application, the dynamic pose matrix refers to the transformation relationship from the camera coordinates of the pixel points corresponding to the recognition object to the world coordinates, the camera coordinates of the camera coordinates of the pixel points corresponding to the recognition object refer to the coordinates of each pixel point in the camera coordinate system, and the static pose matrix refers to the transformation relationship from the camera coordinates to the world coordinates corresponding to the first static object and the second static object.

As shown in FIG. 3 , it is a schematic diagram of the pixel coordinate system and the camera coordinate system provided by the embodiment of the present application. The computer device takes the pixel point O _uv of the first row and first column of the first image as the origin, takes the parallel line where the pixel point of the first row is located as the u axis, and takes the vertical line where the pixel point of the first column is located as the v axis to construct a pixel coordinate system. In addition, the computer device takes the light point O _XY of the monocular camera as the origin, the optical axis of the monocular camera as the Z axis, the line parallel to the u axis of the pixel coordinate system as the X axis, and the camera coordinate system constructed with the line parallel to the v axis of the pixel coordinate system as the Y axis.

In at least one embodiment of the present application, the computer device identifying the object state of the target dynamic object in the recognition object according to the dynamic pose matrix corresponding to the recognition object, the static pose matrix corresponding to the first static object and the second static object, and the preset threshold matrix includes: the computer device subtracts each matrix element in the static pose matrix from the corresponding matrix element in the dynamic pose matrix corresponding to the recognition object to obtain a pose difference. The absolute value of the pose. Further, the computer device arranges the absolute pose values according to the element positions of each absolute value of the pose in the static pose matrix to obtain an absolute pose value matrix. Further, the computer device compares each absolute value of the pose in the absolute value matrix of poses with the corresponding pose threshold in the preset threshold matrix. The computer device determines that the object state of the target dynamic object in the recognition object is moving, or if all the pose absolute values in the pose absolute value matrix are less than or equal to the corresponding threshold value, the computer device determines that the object state of the target dynamic object in the recognition object is stationary.

Specifically, the generation method of the dynamic pose matrix is as follows: the computer device determines the pixel point corresponding to the target dynamic object in the recognition object in the first image as the first pixel point, and determines the pixel point corresponding to the characteristic dynamic object in the recognition object in the second image as the second pixel point; further, the computer device obtains the first homogeneous coordinate matrix of the first pixel point, obtains the second homogeneous coordinate matrix of the second pixel point, obtains the inverse matrix of the internal reference matrix of the shooting device that captured the first image and the second image, and further, the computer The device calculates the first camera coordinate of the first pixel point according to the first homogeneous coordinate matrix and the inverse matrix of the internal reference matrix, and calculates the second camera coordinate of the second pixel point according to the second homogeneous coordinate matrix and the inverse matrix of the internal reference matrix. Further, the computer device calculates the first camera coordinate and the second camera coordinate based on a preset antipolar constraint relation to obtain a rotation matrix and a translation matrix. Further, the computer device splices the rotation matrix and the translation matrix to obtain the target pose matrix.

；則該像素點的齊次座標矩陣為

。將所述第一齊次座標矩陣及所述內參矩陣的逆矩陣進行相乘，得到所述第一像素點的第一相機座標，並將所述第二齊次座標矩陣及所述內參矩陣的逆矩陣進行相乘，得到所述第二像素點的第二相機座標。 Wherein, the first homogeneous coordinate matrix of the first pixel point refers to a matrix whose dimension is one-dimensional more than that of the pixel coordinate matrix, and the element value of one extra dimension is 1. The pixel coordinate matrix refers to a matrix generated according to the first pixel coordinate of the first pixel point, and the first pixel coordinate refers to the coordinate of the first pixel point in the pixel coordinate system. For example, the first pixel coordinate of the first pixel point in the pixel coordinate system is ( u, v ), and the pixel coordinate matrix of the first pixel point is

; then the homogeneous coordinate matrix of the pixel point is

. multiplying the first homogeneous coordinate matrix and the inverse matrix of the internal reference matrix to obtain the first camera coordinates of the first pixel, and multiplying the second homogeneous coordinate matrix and the inverse matrix of the internal reference matrix to obtain the second camera coordinates of the second pixel.

Wherein, the generation method of the second homogeneous coordinate matrix is basically the same as the generation method of the first homogeneous coordinate matrix, and the present application will not repeat them here.

；其中，pose為所述動態位姿矩陣，所述動態位姿矩陣為4x4的矩陣，R為所述旋轉矩陣，所述旋轉矩陣為3x3的矩陣，t為所述平移矩陣，所述平移矩陣為3x1的矩陣。 The rotation matrix can be expressed as:

; wherein, pose is the dynamic pose matrix, the dynamic pose matrix is a 4x4 matrix, R is the rotation matrix, the rotation matrix is a 3x3 matrix, t is the translation matrix, and the translation matrix is a 3x1 matrix.

Wherein, the calculation formulas of the translation matrix and the rotation matrix are: Wherein, K ^-1 p ₁ ( t x R )( K ^-1 p ₂ ) ^T =0; wherein, K ^-1 p ₁ is the coordinates of the first camera, K ^-1 p ₂ is the coordinates of the second camera, p ₁ is the first homogeneous coordinate matrix, p ₂ is the second homogeneous coordinate matrix, and K ^-1 is the inverse matrix of the internal reference matrix.

In this embodiment, the method of generating the static pose matrix is basically the same as the method of generating the dynamic pose matrix, so the present application will not repeat them here. Through the above-mentioned embodiment, when there are multiple identification objects, the number of the dynamic pose matrix is also multiple, since each dynamic pose matrix corresponds to each target dynamic object in the first image, therefore, the object state of the corresponding target dynamic object in the first image can be determined through each dynamic pose matrix, so that the object states of multiple target dynamic objects can be distinguished.

Step 106: Generate a target image according to the object state, the first dynamic position, and the first image, and generate a target projection image according to the object state, the first dynamic position, and an initial projection image corresponding to the first image.

In at least one embodiment of the present application, the target image refers to an image generated after processing the target dynamic object in the first image based on the first dynamic position and the state of the object. In at least one embodiment of the present application, the initial projected image represents an image of a transformation process, and the transformation process refers to a transformation process between pixel coordinates of pixels in the first image and corresponding pixel coordinates in the second image. In at least one embodiment of the present application, the computer device generating the initial projection image of the first image based on the first image, the initial depth image, and the target pose matrix includes: if the object state of any target dynamic object in the identified objects is moving, the computer device calculates the target dynamic object in the first image based on the first dynamic position of any target dynamic object Perform mask processing on any of the target dynamic objects to obtain the target image, or, if the object states of all the target dynamic objects in the identified objects are static, the computer device determines the first image as the target image.

Specifically, the method of generating the initial projected image includes: the computer device obtains an initial depth image of the first image, and obtains a target homogeneous coordinate matrix of each pixel in the first image, and obtains a depth value of each pixel in the first image from the initial depth image. Further, the computer device calculates the projected coordinates of each pixel in the first image based on the target pose matrix, the target homogeneous coordinate matrix of each pixel, and the depth value of each pixel. Points are arranged to obtain the initial projection image. Wherein, the computer device inputs the first image into the depth recognition network to obtain the initial depth image, and the depth value refers to the pixel value of each pixel in the initial depth image.

Specifically, the calculation formula of the projected coordinates of each pixel in the initial projected image is: P = K * pose * Z * K ^-1 * H ; wherein, P represents the projected coordinates of each pixel, K represents the internal reference matrix of the shooting device, pose represents the target pose matrix, K ^-1 represents the inverse matrix of K , H represents the target homogeneous coordinate matrix of each pixel in the first image, and Z represents the depth value of the corresponding pixel in the initial depth image.

In this embodiment, the target projection image includes a plurality of projection objects corresponding to the plurality of target dynamic objects, and the method of generating the target projection image according to the plurality of projection objects is basically the same as that of the target image, so this application will not repeat it here. Through the above-mentioned embodiment, when the object state of the target dynamic object in the identification object is moving, the target dynamic object can be accurately masked in the first image according to the first dynamic position of the target dynamic object, and the influence of the moving dynamic object on the calculation loss value can be avoided; when the object state of the target dynamic object in the identification object is stationary, the target dynamic object is retained in the first image, and more image information of the first image can be retained.

Step 107, based on the gradient error between the initial depth image and the target image and the photometric error between the target projection image and the target image, adjust the obtained depth recognition network to obtain a depth recognition model.

In at least one embodiment of the present application, the deep recognition model refers to a model generated after adjusting the deep recognition network. In at least one embodiment of the present application, the computer device adjusts the depth recognition network based on the gradient error between the initial depth image and the target image and the photometric error between the target projection image and the target image, and obtains the depth recognition model.

Wherein, the deep recognition network may be a deep neural network, and the deep recognition network may be obtained from a database on the Internet. Specifically, the calculation formula of the depth loss value is: Lc = Lt + Ls ; wherein, Lc represents the depth loss value, Lt represents the photometric error, and Ls represents the gradient error.

；其中，Lt表示所述光度誤差，α為預設的平衡參數，一般取值為0.85，SSIM(x,y)表示所述目標投影圖像與所述目標圖像之間的結構相似指數，∥x _i -y _i ∥表示所述目標投影圖像與所述目標圖像之間的灰度差值，x _i 表示所述目標投影圖像第i個像素點的像素值，y _i 表示所述目標圖像中與所述第i個像素點對應的像素點的像素值。所述結構相似指數的計算方式為現有技術，本申請在此不作贅述。 Wherein, the calculation formula of the photometric error is:

; wherein Lt represents the photometric error, α is a preset balance parameter, and generally takes a value of 0.85, SSIM ( x, y ) represents the structural similarity index between the target projection image and the target image, ∥ x _i − y _i ∥ represents the grayscale difference between the target projection image and the target image, xi represents the pixel value of the i-th pixel of the target projection image, _and y _i represents the pixel value of the pixel corresponding to the i -th pixel in the target image. The calculation method of the structural similarity index is the prior art, and the present application will not repeat it here.

；其中，Ls表示所述梯度誤差，x表示所述初始深度圖像，y表示所述目標圖像，D(u，v)表示所述初始深度圖像中第i個像素點的像素座標，I(u，v)表示所述目標圖像中第i個像素點的像素座標。 The calculation formula of the gradient error is:

; wherein, Ls represents the gradient error, x represents the initial depth image, y represents the target image, D ( u , v ) represents the pixel coordinates of the i-th pixel in the initial depth image, I ( u , v ) represents the pixel coordinates of the i-th pixel in the target image.

Through the above implementation manner, since the influence of the moving dynamic object on the calculation of the loss value of the depth recognition network is avoided, the accuracy of the depth recognition model can be improved.

As shown in FIG. 4 , it is a flow chart of the image depth recognition method provided by the embodiment of the present application.

Step 108, acquire the image to be recognized.

In at least one embodiment of the present application, the image to be recognized refers to an image for which depth information needs to be recognized. In at least one embodiment of the present application, the computer device obtains the image to be recognized from a preset database, and the preset database may be KITTI database, Cityscapes database, vKITTI database and so on.

Step 109, input the image to be recognized into the depth recognition model to obtain the target depth image of the image to be recognized and the depth information of the image to be recognized. The depth recognition model is obtained by executing the depth recognition model training method as described above.

In at least one embodiment of the present application, the target depth image refers to an image including depth information of each pixel in the image to be recognized, and the depth information of each pixel in the image to be recognized refers to the distance between the object to be recognized corresponding to each pixel in the image to be recognized and the shooting device that shoots the image to be recognized. In at least one embodiment of the present application, the method of generating the target depth image is basically the same as the method of generating the initial depth image, so the present application will not repeat them here. In at least one embodiment of the present application, the computer device acquires a pixel value of each pixel in the target depth image as depth information of a corresponding pixel in the image to be recognized.

Through the above implementation manner, since the accuracy of the depth recognition model is improved, the accuracy of the depth recognition of the image to be recognized can be improved.

It can be known from the above technical solution that the present application performs instance segmentation on the first image to obtain the first static object corresponding to the first image, multiple first dynamic objects, and the first dynamic position of each first dynamic object, and select multiple target dynamic objects from the multiple first dynamic objects based on the number of pixels and preset positions of each first dynamic object, and can select multiple target dynamic objects from the multiple first dynamic objects. Since the number of the multiple first dynamic objects is reduced, it is possible to improve the training speed of the depth recognition network and identify whether each target dynamic object has a corresponding characteristic dynamic object. A feature dynamic object that is the same as each target dynamic object in the second image can be selected, and by calculating a dynamic pose matrix of each target dynamic object and the same feature dynamic object, and comparing the dynamic pose matrix with the preset threshold matrix, it can be determined whether the state of each target dynamic object in the first image is moving, and a target image can be generated according to the state of the target dynamic object in the recognition object, the first dynamic position, and the first image, and the target dynamic object that moves in the first image can be filtered based on the first dynamic position to generate the target image. The position change of the moving target dynamic object will cause the depth value of the corresponding pixel of the target dynamic object to change in the initial depth image. By filtering out the moving target dynamic object in the target image, the depth value is not used for calculation when calculating the loss value, and the influence of the moving target dynamic object on the calculation loss value can be avoided. The target image retains the static target dynamic object, which can retain more image information of the first image. The impact of the training accuracy of the recognition model can further improve the recognition accuracy of the deep recognition model.

As shown in FIG. 5 , it is a schematic structural diagram of a computer device provided in an embodiment of the present application.

In one embodiment of the present application, the computer device 1 includes, but is not limited to, a storage 12, a processor 13, and a computer program stored in the storage 12 and operable on the processor 13, such as an image depth recognition program and a depth recognition model training program.

Those skilled in the art can understand that the schematic diagram is only an example of the computer device 1, and does not constitute a limitation to the computer device 1, and may include more or less components than those shown in the figure, or combine certain components, or different components, for example, the computer device 1 may also include input and output devices, network access devices, bus bars, etc. The processor 13 may be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field-programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete component gate circuits or transistor components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., and the processor 13 is the computing core and control center of the computer device 1, and uses various interfaces and lines to connect various parts of the entire computer device 1, and obtains the operating system of the computer device 1 and various application programs and program codes installed. For example, the processor 13 may acquire the first image captured by the photographing device 2 through an interface.

The processor 13 acquires the operating system of the computer device 1 and various installed applications. The processor 13 obtains the application program to implement the steps in the above embodiments of each depth recognition model training method and each image depth recognition method, such as the steps shown in FIG. 2 and FIG. 5 . Exemplarily, the computer program can be divided into one or more modules/units, and the one or more modules/units are stored in the storage 12 and retrieved by the processor 13 to complete the application. The one or more modules/units may be a series of computer program instruction segments capable of accomplishing specific functions, and the instruction segments are used to describe the acquisition process of the computer program in the computer device 1 .

The storage 12 can be used to store the computer programs and/or modules, and the processor 13 implements various functions of the computer device 1 by running or obtaining the computer programs and/or modules stored in the storage 12, and calling the data stored in the storage 12. The storage device 12 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playback function, an image playback function, etc.); In addition, the storage 12 may include a non-volatile storage, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a memory card (Flash Card), at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

The storage 12 may be an external storage and/or an internal storage of the computer device 1 . Further, the storage 12 may be a physical storage, such as a storage stick, a TF card (Trans-flash Card) and the like. If the integrated modules/units of the computer device 1 are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the methods of the above-mentioned embodiments, It can also be completed by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is acquired by a processor, it can realize the steps of the above-mentioned various method embodiments.

Wherein, the computer program includes computer program code, and the computer program code may be in the form of original code, object code, obtainable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a flash drive, a removable hard disk, a magnetic disk, an optical disk, a computer storage, and a read-only memory (ROM, Read-Only Memory).

In conjunction with FIG. 2 , the memory 12 in the computer device 1 stores a plurality of instructions to implement a depth recognition model training method, and the processor 13 can obtain the plurality of instructions to realize: obtaining a first image and a second image; performing instance segmentation on the first image based on an instance segmentation network to obtain a first static object corresponding to the first image, a plurality of first dynamic objects, and a first dynamic position of each first dynamic object, and performing instance segmentation on the second image based on the instance segmentation network to obtain a second static object and a plurality of second dynamic objects corresponding to the second image; based on the pixels of each first dynamic object selecting a plurality of target dynamic objects from the plurality of first dynamic objects based on the quantity and preset position, and selecting a plurality of characteristic dynamic objects from the plurality of second dynamic objects based on the number of pixels of each second dynamic object and the preset position; identifying whether each target dynamic object has a corresponding characteristic dynamic object, and determining the target dynamic object and the characteristic dynamic object with corresponding relationship as the recognition object; according to the dynamic pose matrix corresponding to the recognition object, the static pose matrix corresponding to the first static object and the second static object, and the preset threshold matrix, identify the object state of the target dynamic object in the recognition object; , the first dynamic position and the first image generate a target image, and generate a target projection image according to the object state, the first dynamic position, and the initial projection image corresponding to the first image; based on the gradient error between the initial depth image corresponding to the first image and the target image and the photometric error between the target projection image and the target image, adjust the acquired depth recognition network to obtain a depth recognition model.

4, the memory 12 in the computer device 1 stores a plurality of instructions to realize An image depth recognition method is presented. The processor 13 can obtain the multiple instructions to realize: obtain an image to be recognized, input the image to be recognized into a depth recognition model, and obtain a target depth image of the image to be recognized and depth information of the image to be recognized. Specifically, for the specific implementation method of the above instructions by the processor 13, reference may be made to the description of relevant steps in the embodiments corresponding to FIG. 2 and FIG. 4 , and details are not repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation. The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented not only in the form of hardware, but also in the form of hardware plus software function modules. Therefore, no matter from which point of view, all the embodiments should be regarded as exemplary and non-restrictive, and the scope of the application is defined by the appended claims rather than the above description, so all changes within the meaning and scope of the equivalent requirements of the claims are intended to be included in the application. Any attached reference mark in a claim shall not be deemed to limit the claim to which it relates.

In addition, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or devices stated in this application may also be realized by one unit or device through software or hardware. The terms first, second, etc. are used to denote names and do not imply any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application without limitation. Although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application.

101~107: Steps

Claims (10)

A depth recognition model training method applied to a computer device, the computer device communicating with a shooting device, wherein the depth recognition model training method includes: acquiring a first image and a second image, the first image and the second image are images of adjacent frames, and the second image and the first image have different generation times; performing instance segmentation on the first image based on an instance segmentation network to obtain a first static object corresponding to the first image, a plurality of first dynamic objects, and a first dynamic position of each first dynamic object, and performing instance segmentation on the second image based on the instance segmentation network to obtain the corresponding a second static object and a plurality of second dynamic objects; selecting a plurality of target dynamic objects from the plurality of first dynamic objects based on the number of pixels and preset positions of each first dynamic object, and selecting a plurality of characteristic dynamic objects from the plurality of second dynamic objects based on the number of pixels of each second dynamic object and the preset position; identifying whether there is a corresponding characteristic dynamic object for each target dynamic object, and determining the corresponding target dynamic object and the characteristic dynamic object as the recognition object; according to the dynamic pose matrix corresponding to the recognition object, the static pose matrix and the preset threshold matrix corresponding to the first static object and the second static object, Identifying the object state of the target dynamic object in the recognition object; generating a target image according to the object state, the first dynamic position, and the first image, and generating a target projection image according to the object state, the first dynamic position, and an initial projection image corresponding to the first image; based on the gradient error between the initial depth image corresponding to the first image and the target image and the photometric error between the target projection image and the target image, adjusting the acquired depth recognition network to obtain a depth recognition model, including: based on the gradient error and the photometric error, calculating the depth loss value of the depth recognition network, based on The depth loss value adjusts the depth recognition network until the depth loss value drops to a minimum to obtain the depth recognition model. The depth recognition model training method as described in claim 1, wherein, the example The segmentation network includes a feature extraction layer, a classification layer, and a mapping layer. The instance-based segmentation network performs instance segmentation on the first image to obtain the first static object corresponding to the first image, a plurality of first dynamic objects, and the first dynamic position of each first dynamic object. A rectangular area corresponding to each pixel in the initial feature map; classifying the initial feature map based on the classification layer to obtain a predicted probability that each pixel in the initial feature map belongs to a first preset category; determining a pixel corresponding to a predicted probability greater than a preset threshold in the initial feature map as a target pixel, and determining multiple rectangular areas corresponding to multiple target pixels as a plurality of feature areas; mapping each feature area to the initial feature map based on the mapping layer to obtain a mapping area corresponding to each feature area in the initial feature map; obtaining a plurality of divided areas corresponding to each mapped area; determining a central pixel in each divided area, and calculating a pixel value of the central pixel; performing pooling processing on a plurality of pixel values corresponding to a plurality of central pixels to obtain a mapping probability value corresponding to each mapped area; restoring the plurality of mapped areas, and splicing the restored multiple mapped areas to obtain a target feature map; generating a first static object corresponding to the first image, the plurality of first dynamic objects, and each first dynamic object according to the target feature map, the mapped probability value, the restored multiple mapped areas, and the second preset category The first dynamic position of the object. The depth recognition model training method as described in claim 2, wherein, according to Generating the first static object corresponding to the first image, the plurality of first dynamic objects, and the first dynamic position of each first dynamic object corresponding to the target feature map, the mapping probability value, the plurality of restored mapping areas, and the second preset category includes: classifying each pixel of the target feature map according to the mapping probability value and the second preset category, and obtaining the pixel category of each pixel in the restored mapping area; determining an area composed of a plurality of pixel points corresponding to the same pixel category in the restored mapping area as the first object; obtaining pixel locations of all pixels in the first object and determining the pixel coordinates as the first position corresponding to the first object; dividing the plurality of first objects into the plurality of first dynamic objects and the first static object according to preset rules, and determining the first position corresponding to each first dynamic object as the first dynamic position. The method for training a depth recognition model according to claim 1, wherein the selecting a plurality of target dynamic objects from the plurality of first dynamic objects based on the number of pixels and preset positions of each first dynamic object includes: counting the number of pixels contained in each first dynamic object; sorting the plurality of first dynamic objects according to the number of pixels; selecting a first dynamic object whose sorted number of pixels is at the preset position as the plurality of target dynamic objects. The method for training a deep recognition model as described in claim 1, wherein the identifying whether each target dynamic object has a corresponding feature dynamic object includes: obtaining multiple target element information of each target dynamic object, and obtaining feature element information corresponding to each target element information in the same category of feature dynamic objects; matching each target element information with the corresponding feature element information to obtain a matching value between the target dynamic object and the same type of feature dynamic object; The depth recognition model training method as described in claim 1, wherein the identifying the object state of the target dynamic object in the recognition object according to the dynamic pose matrix corresponding to the recognition object, the static pose matrix corresponding to the first static object and the second static object, and the preset threshold value matrix includes: subtracting each matrix element in the static pose matrix from the corresponding matrix element in the dynamic pose matrix corresponding to the recognition object to obtain a pose difference; taking the absolute value of the pose difference to obtain the pose absolute value in the static pose matrix; The element position of each pose absolute value in the static pose matrix, arrange the pose absolute values to obtain a pose absolute value matrix; compare each pose absolute value in the pose absolute value matrix with the corresponding pose threshold in the preset threshold value matrix, if there is at least one pose absolute value greater than the corresponding pose threshold in the pose absolute value matrix, then determine that the object state of the target dynamic object in the recognition object is moving; or if all the pose absolute values in the pose absolute value matrix are less than or equal to the corresponding threshold , then it is determined that the object state of the target dynamic object in the identified object is static. The method for training a depth recognition model according to claim 1, wherein said generating a target image according to said object state, said first dynamic position and said first image comprises: if the object state of any target dynamic object in said recognition objects is moving, then performing mask processing on said any target dynamic object in said first image based on the first dynamic position of said any target dynamic object to obtain said target image; or if the object states of all target dynamic objects in said recognition objects are all static, then determining said first image as said target image. The depth recognition model training method according to claim 1, wherein, adjusting the depth recognition network based on the gradient error between the initial depth image corresponding to the first image and the target image and the photometric error between the target projection image and the target image, and obtaining the depth recognition model includes: Calculating a depth loss value of the depth recognition network based on the gradient error and the photometric error includes: determining the sum of the gradient error and the photometric error as a depth loss value; adjusting the depth recognition network based on the depth loss value until the depth loss value drops to a minimum, and obtaining the depth recognition model. An image depth recognition method applied to computer equipment, wherein the image depth recognition method includes: acquiring an image to be recognized; inputting the image to be recognized into a depth recognition model to obtain a target depth image of the image to be recognized and depth information of the image to be recognized, and the depth recognition model is obtained by executing the depth recognition model training method described in any one of claims 1 to 8. A computer device, wherein the computer device includes: a memory storing at least one instruction; and a processor executing the at least one instruction to implement the depth recognition model training method as described in any one of claims 1 to 8, or the image depth recognition method as described in claim 9.