TWI777538B - Image processing method, electronic device and computer-readable storage media - Google Patents

Abstract

The present disclosure provides an image processing method, electronic device, and computer-readable storage media. The method includes: identifying a target area where a target object in a first image is located; based on the target region corresponding to the target object, determining the first & two-dimensional position information of multiple key points representing the pose of the target object in the first image, the relative depth of each key point to the reference node of the target object, and the absolute depth of the reference node of the target object in the camera coordinate system h; based on the first & two-dimensional position information, relative depth, and absolute depth of the target object, determining the three-dimensional position information of multiple key points of the target object in the camera coordinate system.

Description

Image processing method, electronic device and computer-readable storage medium

The present invention relates to the technical field of image processing, and in particular, to an image processing method, an electronic device, and a computer-readable storage medium.

3D human posture detectors are widely used in security, games, entertainment and other fields. The current 3D human posture detection method usually recognizes the first 2D position information of the human body key points in the image, and then converts the first 2D position information into a 3D position according to the predetermined positional relationship between the human body key points. News.

The human pose obtained by the current 3D human pose detection method has a large error.

Embodiments of the present invention provide at least an image processing method, an electronic device, and a computer-readable storage medium.

In a first aspect, an embodiment of the present invention provides an image processing method, including: identifying a target area where a target object is located in a first image; and determining a plurality of target objects based on the target area where the target object is located The first two-dimensional position information of the key points in the first image, the relative depth of each key point relative to the reference node of the target object, and the reference node of the target object in the camera coordinate system The absolute depth of the target object; based on the first two-dimensional position information and the relative depth corresponding to the multiple key points of the target object, and the absolute depth corresponding to the reference node, determine the depth of the target object. The three-dimensional position information of each key point in the camera coordinate system.

In this way, the embodiment of the present invention can more accurately obtain the three-dimensional position information of the multiple key points of the target object in the camera coordinate system, respectively, and the three-dimensional position information of the multiple key points of the target object in the camera coordinate system can represent the target object. The higher the accuracy of the 3D position information, the higher the accuracy of the obtained 3D pose of the target object.

In a possible implementation manner, the method further includes: obtaining the pose of the target object based on the three-dimensional position information of the multiple key points of the target object in the camera coordinate system respectively.

In this way, based on the three-dimensional position information of the multiple key points of the target object in the camera coordinate system obtained in the embodiment of the present invention, since the three-dimensional position information has higher precision, the posture of the target object determined based on the three-dimensional position information is also more precise.

In a possible implementation manner, the identifying the target area where the target object in the first image is located includes: performing feature extraction on the first image to obtain a feature map of the first image; In the feature map, a plurality of target bounding boxes are determined from a plurality of candidate bounding boxes generated in advance; based on the plurality of the target bounding boxes, a target area where the target object is located is determined.

In this way, it is divided into two steps to determine the target area where the target object is located, and the position of each target object in the first image can be accurately detected from the first image, so as to improve the human body in the subsequent key point detection process. Information integrity, and detection accuracy.

In a possible implementation manner, the determining the target area where the target object is located based on the plurality of target bounding boxes includes: determining each target based on the plurality of target bounding boxes and the feature map. Feature sub-maps of the bounding box; performing bounding box regression processing on the feature sub-maps corresponding to the plurality of target bounding boxes to obtain the target area where the target object is located.

In this way, the bounding box regression processing is performed on the feature submaps corresponding to the multiple target bounding boxes, so that the positions of the respective target objects in the first image can be accurately detected from the first image.

In a possible implementation manner, determining the absolute depth of the reference node of the target object in the camera coordinate system based on the target area where the target object is located includes: based on the target area where the target object is located and the first image, determine the target feature map of the target object; perform depth recognition processing on the target feature map corresponding to the target object to obtain the normalized absolute depth of the reference node of the target object; based on the normalized absolute depth The depth and the parameter matrix of the camera are used to obtain the absolute depth of the reference node of the target object in the camera coordinate system.

In this way, it can be avoided as much as possible that the absolute depth of the reference node is directly predicted based on the target feature map due to different internal parameters of the cameras, resulting in different absolute depths obtained for different first images obtained by different cameras at the same viewing angle and at the same position. .

In a possible implementation manner, the performing depth recognition processing on the target feature map corresponding to the target object to obtain the normalized absolute depth of the reference node of the target object includes: determining based on the first image. An initial depth image; wherein the primitive value of any first primitive point in the initial depth image is a second primitive point corresponding to the position of the first primitive point in the first image the initial depth value in the camera coordinate system; based on the target feature map corresponding to the target object, determine the second two-dimensional position information of the reference node corresponding to the target object in the first image; based on The second two-dimensional position information and the initial depth image determine the initial depth value of the reference node corresponding to the target object; based on the initial depth value of the reference node and the target feature map, determine the The normalized absolute depth of the reference node of the target object.

In this way, the normalized absolute depth of the reference node obtained through this process can be made more accurate.

In a possible implementation manner, determining the normalized absolute depth of the reference node of the target object based on the initial depth value of the reference node and the target feature map includes: The feature map is subjected to at least one level of first convolution processing to obtain the feature vector of the target object; the feature vector and the initial depth value are spliced to obtain a spliced vector, and at least one level of the second convolution is performed on the spliced vector. product processing to obtain the modified value of the initial depth value; based on the modified value of the initial depth value and the initial depth value, the normalized absolute depth is obtained.

In a possible implementation manner, the parameter matrix includes: the focal length of the camera; Obtaining the absolute depth of the reference node of the target object in the camera coordinate system based on the normalized absolute depth and the parameter matrix of the camera, including: Based on the normalized absolute depth, the focal length, the area of the target region, and the area of the target bounding box, the absolute depth of the reference node of the target object in the camera coordinate system is obtained.

In a possible implementation, the image processing method is applied to a pre-trained neural network, and the neural network includes three branch networks: a target detection network, a key point detection network, and a depth prediction network. ; the target detection network is used to obtain the target area where the target object is located; the key point detection network is used to obtain the first number of key points of the target object in the first image respectively. Two-dimensional position information, and the relative depth of each key point relative to the reference node of the target object; the depth prediction network is used to obtain the absolute depth of the reference node in the camera coordinate system.

In this way, through the target detection network, the key point detection network and the depth prediction network three branch networks, an end-to-end target object pose detection framework is formed, and the first image is processed based on the framework to obtain the first image. The three-dimensional position information of multiple key points of each target object in the camera coordinate system, the processing speed is faster and the recognition accuracy is higher.

In a second aspect, an embodiment of the present invention further provides an image processing device, including: a recognition module for recognizing a target area where a target object in a first image is located; a first detection module for based on the target the target area where the object is located, and determine the first two-dimensional position information of multiple key points of the target object in the first image, and the relative depth of each key point relative to the reference node of the target object , and the absolute depth of the reference node of the target object in the camera coordinate system; the second detection module is used for the first two-dimensional position information and the described first two-dimensional position information respectively corresponding to a plurality of key points of the target object based on The relative depth and the absolute depth corresponding to the reference node determine the three-dimensional position information of the multiple key points of the target object respectively in the camera coordinate system.

In a possible implementation manner, the second detection module is further configured to obtain the posture of the target object based on the three-dimensional position information of a plurality of key points of the target object in the camera coordinate system respectively.

In a possible implementation manner, the recognition module, when recognizing the target area where the target object in the first image is located, is used to: perform feature extraction on the first image to obtain the first image. A feature map of the image; based on the feature map, determine multiple target bounding boxes from multiple candidate bounding boxes generated in advance; based on the multiple target bounding boxes, determine the target area where the target object is located.

In a possible implementation manner, the identification module, when determining the target area where the target object is located based on a plurality of the target bounding boxes, is used for: based on the plurality of the target bounding boxes and the feature map. , determine the feature sub-map of each target bounding box; perform bounding box regression processing on the feature sub-maps corresponding to the plurality of target bounding boxes respectively to obtain the target area where the target object is located.

In a possible implementation manner, wherein the first detection module, when determining the absolute depth of the reference node of the target object in the camera coordinate system based on the target area where the target object is located, is used for: based on the The target area where the target object is located and the first image, determine the target feature map of the target object; perform depth recognition processing on the target feature map corresponding to the target object, and obtain the normalization of the reference nodes of the target object The normalized absolute depth; based on the normalized absolute depth and the parameter matrix of the camera, the absolute depth of the reference node of the target object in the camera coordinate system is obtained.

In a possible implementation manner, the first detection module, when performing depth recognition processing on the target feature map corresponding to the target object to obtain the normalized absolute depth of the reference node of the target object, is used for: Based on the first image, an initial depth image is determined; wherein, the primitive value of any first primitive point in the initial depth image is the same as the first primitive point in the first image The initial depth value of the second primitive point corresponding to the position of the target object in the camera coordinate system; based on the target feature map corresponding to the target object, it is determined that the reference node corresponding to the target object is in the first image. based on the second two-dimensional position information and the initial depth image, determine the initial depth value of the reference node corresponding to the target object; based on the initial depth value of the reference node and For the target feature map, the normalized absolute depth of the reference node of the target object is determined.

In a possible implementation manner, the first detection module is used for determining the normalized absolute depth of the reference node of the target object based on the initial depth value of the reference node and the target feature map. : perform at least one-level first convolution processing on the target feature map corresponding to the target object to obtain a feature vector of the target object; splicing the feature vector and the initial depth value to obtain a splicing vector, The splicing vector is subjected to at least one stage of second convolution processing to obtain a modified value of the initial depth value; based on the modified value of the initial depth value and the initial depth value, the normalized absolute depth is obtained.

In a possible implementation manner, the parameter matrix includes: the focal length of the camera; the first detection module obtains the reference node of the target object based on the normalized absolute depth and the parameter matrix of the camera At absolute depth in the camera coordinate system, used to: Based on the normalized absolute depth, the focal length, the area of the target region, and the area of the target bounding box, the absolute depth of the reference node of the target object in the camera coordinate system is obtained.

In a possible implementation, the image processing device uses a pre-trained neural network to realize image processing, and the neural network includes three branches: a target detection network, a key point detection network, and a depth prediction network. network; the target detection network is used to obtain the target area where the target object is located; the key point detection network is used to obtain the respective key points of the target object in the first image. The first two-dimensional position information, and the relative depth of each key point relative to the reference node of the target object; the depth prediction network is used to obtain the absolute depth of the reference node in the camera coordinate system.

In a third aspect, an embodiment of the present invention further provides a computer device, including: a processor and a memory that are connected to each other, the memory stores machine-readable instructions executable by the processor, and when the computer device runs, the The machine-readable instructions are executed by the processor to implement the first aspect or the steps of the image processing method in any possible implementation manner of the first aspect.

In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and the computer program executes the first aspect or any one of the first aspects when the computer program is run by the processor. Steps of an image processing method in a possible implementation.

In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below, and are described in detail as follows in conjunction with the accompanying drawings.

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only These are some embodiments of the present invention, but not all embodiments. The elements of the embodiments of the invention generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Thus, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present invention.

The three-dimensional human pose detection method is usually to identify the first two-dimensional position information of human key points in the image to be recognized through a neural network, and then based on the mutual positional relationship between human key points (such as the connection relationship between different key points) , distance range between adjacent key points, etc.) to convert the first two-dimensional position information of each human body key point into three-dimensional position information; The positional relationships are also different, resulting in large errors in the three-dimensional human pose obtained by this method.

In addition, the accuracy of the current 3D human pose detection method is based on the accurate estimation of the key points of the human body. However, due to the occlusion of clothing and limbs, in many cases, the key points of the human body cannot be accurately identified from the image. As a result, the three-dimensional human body posture error obtained by the above method will be further enlarged.

The defects existing in the above solutions are all the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions to the above problems proposed by the present invention hereinafter should be the inventors. Contributions made to the invention in the course of the invention.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

Based on the above research, the present invention provides an image processing method and device. By identifying the target area where the target object is located in the first image, and based on the target area, multiple key points representing the posture of the target object are determined in the first image respectively. The first two-dimensional position information in the image, the relative depth of each key point relative to the reference node of the target object, and the absolute depth of the reference node of the target object in the camera coordinate system, so as to be based on the first two-dimensional position of the target object information, relative depth, and absolute depth, to more accurately obtain the three-dimensional position information of multiple key points of the target object in the camera coordinate system.

In order to facilitate the understanding of this embodiment, an image processing method disclosed in the embodiment of the present invention is first introduced in detail. The execution subject of the image processing method provided by the embodiment of the present invention is generally a computer device with Computer equipment includes, for example: terminal equipment or server or other processing equipment, and the terminal equipment can be user equipment (UE), mobile equipment, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (Personal Digital Assistant) Assistant, PDA), handheld devices, computing devices, in-vehicle devices, wearable devices, etc. In some possible implementations, the image processing method may be implemented by the processor calling computer-readable instructions stored in the memory.

The following describes the image processing method provided by the embodiment of the present invention.

Referring to FIG. 1, which is a flowchart of an image processing method provided by an embodiment of the present invention, the method includes steps S101-S103, wherein: S101: Identify the target area where the target object in the first image is located; S102: Based on the target area where the target object is located, determine first two-dimensional position information of multiple key points of the target object in the first image, and each of the key points is relative to the target object. The relative depth of the reference node, and the absolute depth of the reference node of the target object in the camera coordinate system; S103: Determine a plurality of key points of the target object based on the first two-dimensional position information and the relative depth corresponding to the plurality of key points of the target object respectively, and the absolute depth corresponding to the reference node The three-dimensional position information of the points in the camera coordinate system respectively.

The above S101 to S103 are respectively described in detail below.

I: In the above S101, the first image includes at least one target object. The target object is, for example, a person, an animal, a robot, or a vehicle whose posture is to be determined.

In a possible implementation, when more than one target object is included in the first image, the categories of different target objects may be the same or different; for example, multiple target objects are all people; or multiple target objects All are vehicles. For another example, the target objects in the first image include: people and animals; or the target objects in the first image include people and vehicles, and the target object category is specifically determined according to actual application scenario requirements.

The target area where the target object is located refers to an area in the first image that includes the target object.

Exemplarily, as shown in FIG. 2 , an embodiment of the present invention provides a specific method for identifying a target area where a target object is located in a first image, including: S201: Perform feature extraction on the first image to obtain a feature map of the first image. Here, for example, a neural network can be used to perform feature extraction on the first image to obtain a feature map of the first image. S202: Based on the feature map, determine multiple target bounding boxes from multiple pre-generated candidate bounding boxes, and determine a target area corresponding to the target object based on the target bounding boxes.

In a specific implementation, for example, a bounding box prediction algorithm can be used to obtain multiple target bounding boxes. For example, bounding box prediction algorithms include RoIAlign, ROI-Pooling, etc. Taking RoIAlign as an example, RoIAlign can traverse multiple pre-generated candidate bounding boxes to determine that the sub-images corresponding to each candidate bounding box belong to any of the first image. The region of interest (ROI) value of a target object, the higher the ROI value, the greater the probability that the sub-image corresponding to the corresponding candidate bounding box belongs to a certain target object; After the ROI values corresponding to the candidate bounding boxes, according to the descending order of the ROI values corresponding to the candidate bounding boxes, a plurality of target bounding boxes are determined from the candidate bounding boxes.

The target bounding box is, for example, a rectangle; the information of the target bounding box includes, for example, the coordinates of any vertex in the target bounding box in the first image, and the height and width values of the target bounding box. Or, the information of the target bounding box includes, for example, the coordinates of any vertex in the target bounding box in the feature map of the first image, and the height and width values of the target bounding box.

After the multiple target bounding boxes are obtained, the respective target regions corresponding to all the target objects in the first image are determined based on the multiple target bounding boxes.

Referring to FIG. 3 , an embodiment of the present invention provides a specific example of determining a target area corresponding to a target object based on a target bounding box, including the following steps.

S301: Based on the plurality of target bounding boxes and the feature maps, determine a feature submap of each of the target bounding boxes.

In a specific implementation, when the information of the target bounding box includes the coordinates of any vertex on the target bounding box in the first image, and the height and width values of the target bounding box, the feature points in the feature map and The primitive points in the first image have a certain positional mapping relationship; according to the relevant information of the target bounding box and the mapping relationship between the feature map and the first image, each feature map is determined from the feature map of the first image. The feature submaps corresponding to the target bounding boxes respectively.

When the information of the target bounding box includes the coordinates of any vertex in the target bounding box in the feature map of the first image, as well as the height and width values of the target bounding box, it can be directly based on the target bounding box, from the first A feature submap corresponding to each target bounding box is determined in the feature map of an image.

S302: Perform bounding box regression processing on the feature submaps corresponding to the plurality of target bounding boxes to obtain a target area where the target object is located.

Here, for example, a bounding box regression (Bounding-Box Regression) algorithm may be used to perform bounding box regression processing on the feature submaps corresponding to each target bounding box, so as to obtain multiple bounding boxes including the complete target object.

Using the bounding box regression algorithm, the target object can be accurately determined from the corresponding target area, so as to distinguish the target object from the image background, thereby reducing the influence of the image background on the subsequent image processing process.

Each bounding box in the plurality of bounding boxes corresponds to a target object, and the area determined based on the bounding box corresponding to the target object is the target area where the corresponding target object is located.

At this time, the obtained number of target areas is consistent with the number of target objects in the first image, and each target object corresponds to a target area; The target regions corresponding to the target objects in the occlusion relationship have a certain degree of overlap.

In another embodiment of the present invention, other target detection algorithms may also be used to identify the target area where the target object in the first image is located. For example, a semantic segmentation algorithm is used to determine the semantic segmentation result of each primitive point in the first image, and then according to the semantic segmentation result, the positions of the primitive points belonging to different target objects in the first image are determined; The minimum bounding box is obtained from the primitive points belonging to the same target object, and the area corresponding to the minimum bounding box is determined as the target area where the target object is located.

II: In the above S102, the image coordinate system refers to a two-dimensional coordinate system established by the length and width of the first image; the camera coordinate system refers to the direction of the optical axis of the camera and the parallel A three-dimensional coordinate system established in two directions on the optical axis and in the plane where the optical center of the camera is located.

The key points of the target object, for example, are located on the target object, have a mutual relationship, and can represent the pose of the target object after being connected according to the relationship; for example, when the target object is a human body, the key points include, for example The key points of each joint of the human body. In the image coordinate system, the key point is expressed as a two-dimensional coordinate value; in the camera coordinate system, it is expressed as a three-dimensional coordinate value.

In a specific implementation, for example, a key point detection network can be used to perform key point detection processing based on the target feature map of the target object to obtain two-dimensional position information of multiple key points of the target object in the first image, and each The relative depth of a keypoint relative to the reference node of the target object. Here, for the acquisition method of the target feature map, reference may be made to the following description of S401, which will not be repeated here.

The reference node is, for example, any primitive point on a predetermined part of the target object. Exemplarily, the reference node can be pre-determined according to actual needs; for example, when the target object is a human body, a primitive point on the human pelvis can be determined as a reference node, or any primitive point on the human body can be determined as a reference node. , or determine the primitive point on the center of the chest and abdomen of the human body as the reference node; the specific can be set as required.

The absolute depth of each key point relative to the reference node of the target object, for example, the difference between the coordinate value of the key point in the depth direction of the camera coordinate system and the coordinate value of the reference node in the depth direction of the camera coordinate system. The absolute depth of the key point, for example, the coordinate value of the key point in the depth direction of the camera coordinate system.

Referring to FIG. 4 , an embodiment of the present invention provides a specific method for determining the absolute depth of a reference node of a target object in a camera coordinate system based on a target area corresponding to a target object, including the following steps.

S401: Determine a target feature map of the target object based on the target area where the target object is located and the first image.

Here, for example, based on a feature map of the first image obtained by performing feature extraction on the first image, and the target area, a target feature map of the target object may be determined from the feature map.

Here, the feature points in the feature map extracted for the first image and the primitive points in the first image have a certain position mapping relationship; after obtaining the target area where each target object is located, according to the position mapping relationship, The position of each target object in the feature map of the first image is determined, and then the target feature map related to each target object is cut out from the feature map of the first image.

S402: Perform depth recognition processing on the target feature map corresponding to the target object, to obtain the normalized absolute depth of the reference node of the target object.

Here, because the internal parameters of different cameras are different, the target object will be different in the imaging of different cameras; if the absolute depth of the reference node of the target object is to be directly determined, there will be errors caused by the camera internal parameters, so the embodiment of the present invention will In order to reduce the influence of image differences caused by different camera internal parameters on the absolute depth, we can first obtain the normalized absolute depth of the reference node of the target object based on the target feature map, and then use the normalized absolute depth and the camera to obtain the normalized absolute depth. Internal parameter to get the absolute depth of the reference node. The normalized absolute depth is the absolute depth obtained by normalizing the reference node by using the parameter matrix of the camera. After obtaining the normalized absolute depth, the parameter matrix of the camera can be used to restore the absolute depth of the reference node.

In a possible implementation, for example, a pre-trained depth prediction network may be used to perform depth detection processing on the target feature map to obtain the normalized absolute depth of the reference node of the target object.

In another embodiment of the present invention, as shown in FIG. 5 , another specific method for obtaining the normalized absolute depth of a reference node is provided, including the following steps.

S501: Determine an initial depth image based on the first image; wherein, the primitive value of any first primitive point in the initial depth image is the same as the first image in the first image The position of the primitive point corresponds to the initial depth value of the second primitive point in the camera coordinate system.

In a specific implementation, the first primitive point in the initial depth image has a one-to-one correspondence with the second primitive point in the first image, that is, the coordinates of the first primitive point in the initial depth image The value is the same as the coordinate value of the second primitive point corresponding to the position in the first image.

Exemplarily, a depth prediction network can be used to determine the initial depth value of each primitive point (second primitive point) in the first image; the initial depth value of each first primitive point constitutes the first The initial depth image of the image; the primitive value of any primitive point (the first primitive point) in the initial depth image is the primitive point at the corresponding position in the first image (the second image element point) initial depth value.

S502: Based on the target feature map corresponding to the target object, determine the second two-dimensional position information of the reference node corresponding to the target object in the first image, and based on the second two-dimensional position information, and the initial depth image, to determine the initial depth value of the reference node corresponding to the target object.

Here, the target feature map corresponding to the target object may be, for example, a target feature map determined for each target object from the feature map of the first image based on the target region corresponding to each target object.

After obtaining the target feature maps corresponding to each target object, for example, a pre-trained reference node detection network can be used to determine the second two-dimensional position information of the reference nodes of the target object in the first image based on the target feature map. Then, using the second two-dimensional position information, the primitive point corresponding to the reference node is determined from the initial depth image, and the primitive value of the primitive point determined from the initial depth image is determined as the initial value of the reference node. depth value.

S503: Determine the normalized absolute depth of the reference node of the target object based on the initial depth value of the reference node and the target feature map.

Exemplarily, for example, at least one level of first convolution processing may be performed on the target feature map corresponding to the target object to obtain a feature vector of the target object; the feature vector and the initial depth value are spliced to obtain splicing. vector, and perform at least one level of second convolution processing on the splicing vector to obtain the modified value of the initial depth value; based on the modified value of the initial depth value and the initial depth value, obtain the normalized value absolute depth.

Here, for example, a neural network for adjusting the initial depth value can be used, and the neural network includes a plurality of convolutional layers; wherein, some of the convolutional layers in the plurality of convolutional layers are used to perform at least one-level processing on the target feature map. The first convolution processing; other convolution layers are used to perform at least one second convolution processing on the splicing vector, and then obtain the correction value; then adjust the initial depth value according to the correction value to obtain the normalization of the reference node of the target object depth.

Following the above S402, the specific method for determining the absolute depth of the reference node of the target object in the camera coordinate system provided by the embodiment of the present invention further includes: S403: Based on the normalized absolute depth and the parameter matrix of the camera, obtain the absolute depth of the reference node of the target object in the camera coordinate system.

In a specific implementation, in the process of performing image processing on different first images, different first images may be captured by different cameras; and for different cameras, the corresponding camera internal parameters may be different; Here, the camera internal parameters include, for example, the focal length of the camera on the x-axis, the focal length of the camera on the y-axis, and the coordinates of the x-axis and y-axis of the optical center of the camera in the camera coordinate system.

If the camera's internal parameters are different, even the first image obtained from the same perspective and the same position will be different; if the absolute depth of the reference node is directly predicted based on the target feature map, it will cause different images obtained by different cameras from the same perspective and the same position. The absolute depths obtained for the first images are different.

In order to avoid the above situation, the embodiment of the present invention directly predicts the normalized depth of the reference node, and the normalized absolute depth is obtained without considering the camera's internal parameters; and then according to the camera's internal parameters and the normalized absolute depth , which restores the absolute depth of the reference node.

When restoring the absolute depth of the reference node based on the normalized absolute depth, for example, the target may be obtained based on the normalized absolute depth, the focal length, the area of the target region, and the area of the target bounding box. The absolute depth of the object's reference node in the camera coordinate system.

（1） 其中，

表示參考節點的歸一化絕對深度；

表示參考節點絕對深度；

表示目標區域的面積；

表示目標邊界框的面積。

表示相機焦距。示例性的，相機座標系為三維座標系；包括x、y和z三個座標軸；相機座標系的原點為相機的光心；相機的光軸為相機座標系的z軸；光心所在的、且垂直於z軸的平面為x軸和y軸所在的平面；

為相機在x軸上的焦距；

為相機在y軸上的焦距。Exemplarily, the normalized absolute depth and absolute depth of the reference node of any target object satisfy the following formula (1):

(1) Among them,

Indicates the normalized absolute depth of the reference node;

Indicates the absolute depth of the reference node;

represents the area of the target area;

Represents the area of the target bounding box.

Indicates the camera focal length. Exemplarily, the camera coordinate system is a three-dimensional coordinate system; it includes three coordinate axes, x, y, and z; the origin of the camera coordinate system is the optical center of the camera; the optical axis of the camera is the z-axis of the camera coordinate system; , and the plane perpendicular to the z-axis is the plane where the x-axis and the y-axis are located;

is the focal length of the camera on the x-axis;

is the focal length of the camera on the y-axis.

It should be noted here that, in the above S202, it can be known that there are multiple target bounding boxes determined by RoIAlign; and the areas of the multiple target bounding boxes are all equal.

Since the camera focal length has been determined when the camera acquires the first image, and the target area and the target bounding box are also determined when the target area is determined, after obtaining the normalized absolute depth of the reference node, according to the above formula (1 ) to get the absolute depth of the reference node of the target object.

。III: In the above S103, it is assumed that each target object includes J key points, and there are N target objects in the first image; wherein, the three-dimensional poses of the N target objects are expressed as:

.

可以表示為：

。 其中，

表示第m個目標對象的第j個關鍵點在相機座標系中x軸方向的座標值；

表示第m個目標對象的第j個關鍵點在相機座標系中y軸方向的座標值；

表示第m個目標對象的第j個關鍵點在相機座標系中z軸方向的座標值。Among them, the three-dimensional pose of the m-th target object

It can be expressed as:

. in,

Indicates the coordinate value of the j-th key point of the m-th target object in the x-axis direction of the camera coordinate system;

Represents the coordinate value of the j-th key point of the m-th target object in the y-axis direction of the camera coordinate system;

Indicates the coordinate value of the j-th key point of the m-th target object in the z-axis direction of the camera coordinate system.

。其中，第m個目標對象所在的目標區域

表示為：

；此處，

和

表示目標區域的左上角所在的頂點的座標值；

和

分別表示目標區域的寬度值和高度值。The target area where N target objects are located is expressed as:

. Among them, the target area where the mth target object is located

Expressed as:

; here,

and

Indicates the coordinate value of the vertex where the upper left corner of the target area is located;

and

Represents the width and height of the target area, respectively.

；其中，第m個目標對象相對於參考節點的三維姿勢

表示為：

；其中，

表示第m個目標對象的第j個關鍵點在圖像座標系中x軸的座標值；

表示第m個目標對象的第j個關鍵點在圖像座標系中y軸的座標值；也即，

表示第m個目標對象的第j個關鍵點在圖像座標系中的二維座標值。

表示第m個目標對象的第j個節點相對於第m個目標對象的參考節點的相對深度。The 3D poses of N target objects relative to the reference node are expressed as:

; where, the 3D pose of the mth target object relative to the reference node

Expressed as:

;in,

Indicates the coordinate value of the jth key point of the mth target object in the x-axis of the image coordinate system;

Represents the coordinate value of the j-th key point of the m-th target object in the y-axis of the image coordinate system; that is,

Indicates the two-dimensional coordinate value of the jth key point of the mth target object in the image coordinate system.

Represents the relative depth of the jth node of the mth target object relative to the reference node of the mth target object.

（2） 其中，

表示第m個目標對象的參考節點在相機座標系中的絕對深度值。此處，需要注意的是，該

基於上述公式（1）對應的實施例獲得。Using the camera's internal parameter matrix K, the three-dimensional pose of the m-th target object is obtained by back-projection, where the three-dimensional coordinate information of the j-th node of the m-th target object satisfies the following formula (2)

(2) where,

Indicates the absolute depth value of the reference node of the mth target object in the camera coordinate system. Here, it should be noted that the

Obtained based on the embodiment corresponding to the above formula (1).

； 其中：

為相機在相機座標系中x軸上的焦距；

為相機在相機座標系中在y軸上的焦距；

為相機的光心在相機座標系中在x軸上的座標值；

表示相機的光心在相機座標系中在y軸上的座標值。The internal parameter matrix K is for example:

; in:

is the focal length of the camera on the x-axis in the camera coordinate system;

is the focal length of the camera on the y-axis in the camera coordinate system;

is the coordinate value of the optical center of the camera on the x-axis in the camera coordinate system;

Indicates the coordinate value of the camera's optical center on the y-axis in the camera coordinate system.

Through the above process, the three-dimensional position information of multiple key points of the target object in the camera coordinate system can be obtained respectively; for the m-th target object, the three-dimensional position information corresponding to the J key points of the target object respectively represents the m-th target object. The 3D pose of the target object.

The embodiment of the present invention identifies the target area where the target object is located in the first image, and based on the target area, determines the first two-dimensional position information, each The relative depth of the key point relative to the reference node of the target object, and the absolute depth of the reference node of the target object in the camera coordinate system, so that based on the first two-dimensional position information, relative depth, and absolute depth of the target object, more accurate The three-dimensional position information of the multiple key points of the target object in the camera coordinate system is obtained.

In another embodiment of the present invention, another image processing method is also provided, wherein the image processing method is applied to a pre-trained neural network.

Wherein, the neural network includes three branch networks: a target detection network, a key point detection network and a depth prediction network; the target detection network is used to obtain the target area where the target object is located; the key The point detection network is used to obtain first two-dimensional position information of a plurality of key points of the target object in the first image respectively, and the relative relationship of each of the key points to the reference node of the target object. Depth; the depth prediction network is used to obtain the absolute depth of the reference node in the camera coordinate system.

The specific working process of the above-mentioned three branch networks can be referred to as shown in the above-mentioned embodiment, and details are not repeated here.

In the embodiment of the present invention, an end-to-end target object gesture detection framework is formed through three branch networks, a target detection network, a key point detection network, and a depth prediction network. Based on the framework, the first image is processed to obtain the first The three-dimensional position information of multiple key points of each target object in the image in the camera coordinate system, the processing speed is faster and the recognition accuracy is higher.

Referring to FIG. 6 , an embodiment of the present invention also provides a specific example of a target object gesture detection framework, including: There are three network branches: target detection network, key point detection network, and depth prediction network; The target detection network performs feature extraction on the first image to obtain a feature map of the first image; then, according to the first feature map, RoIAlign is used to determine multiple target boundaries from multiple pre-generated candidate bounding boxes box; perform bounding box regression processing on multiple target bounding boxes to obtain target regions corresponding to each target object. The target feature map corresponding to the target area is transmitted to the key point detection network and the depth prediction network.

The key point detection network, based on the target feature map, determines the first two-dimensional position information of a plurality of key points representing the posture of the target object in the first image, and the reference node of each key point relative to the target object. relative depth. The first two-dimensional position information and relative depth of each key point in each target feature map constitute the three-dimensional pose of the target object in the target feature map. The three-dimensional posture at this time is the three-dimensional posture with the self as the reference.

The depth prediction network, based on the target feature map, determines the absolute depth of the reference node of the target object in the camera coordinate system.

Finally, according to the first two-dimensional position information of the target object, the relative depth, and the absolute depth of the reference node, the three-dimensional position information of the multiple key points of the target object in the camera coordinate system is determined respectively. For each target object, the three-dimensional position information of a plurality of key points on the target object in the camera coordinate system respectively constitutes the three-dimensional pose of the target object in the camera coordinate system. The 3D pose at this time is the 3D pose with the camera as a reference.

Referring to FIG. 7 , an embodiment of the present invention further provides another specific example of a target object gesture detection framework, including: Target detection network, key point detection network, and depth prediction network; The target detection network performs feature extraction on the first image to obtain a feature map of the first image; then, according to the first feature map, RoIAlign is used to determine multiple target boundaries from multiple pre-generated candidate bounding boxes box; perform bounding box regression processing on multiple target bounding boxes to obtain target regions corresponding to each target object. The target feature map corresponding to the target area is transmitted to the key point detection network and the depth prediction network.

The key point detection network, based on the target feature map, determines the first two-dimensional position information of a plurality of key points representing the posture of the target object in the first image, and the reference node of each key point relative to the target object. relative depth. The first two-dimensional position information and relative depth of each key point in each target feature map constitute the three-dimensional pose of the target object in the target feature map. The three-dimensional posture at this time is the three-dimensional posture with the self as the reference.

The depth prediction network, based on the first image, obtains an initial depth image; and based on the target feature map corresponding to the target object, determines the second two-dimensional position information of the reference node corresponding to the target object in the first image , and based on the second two-dimensional position information and the initial depth image, determine the initial depth value of the reference node corresponding to the target object; and perform at least one level of first convolution on the target feature map corresponding to the target object processing to obtain the feature vector of the target object; splicing the eigenvector and the initial depth value of the reference node to form a splicing vector, and performing at least one level of second convolution processing on the splicing vector to obtain the initial depth The correction value of the value; the correction value is added to the initial depth value of the reference node to obtain the normalized absolute depth value of the reference node.

Then, through the above formula (1), the absolute depth value of the reference node is restored, and then according to the first two-dimensional position information of the target object, the relative depth, and the absolute depth of the reference node, multiple key points of the target object are determined The three-dimensional position information of the points in the camera coordinate system respectively. For each target object, the three-dimensional position information of a plurality of key points on the target object in the camera coordinate system respectively constitutes the three-dimensional pose of the target object in the camera coordinate system. The 3D pose at this time is the 3D pose with the camera as a reference.

Through either of the above two target object gesture detection frameworks, the three-dimensional position information of multiple key points of each target object in the first image in the camera coordinate system can be obtained, with faster processing speed and higher recognition accuracy. .

Those skilled in the art can understand that in the above method of the specific implementation, the writing order of each step does not mean a strict execution order but constitutes any limitation on the implementation process, and the specific execution order of each step should be based on its function and possible Internal logic is determined.

Based on the same inventive concept, the embodiment of the present invention also provides an image processing apparatus corresponding to the image processing method. For the implementation of the apparatus, reference may be made to the implementation of the method, and the repetition will not be repeated.

8 , which is a schematic diagram of an image processing apparatus provided by an embodiment of the present invention, the apparatus includes: an identification module 81 , a first detection module 82 , and a second detection module 83 ; wherein, The identification module 81 is used to identify the target area where the target object in the first image is located; The first detection module 82 is used to determine, based on the target area corresponding to the target object, the first two-dimensional position information, each the relative depth of the key point relative to the reference node of the target object, and the absolute depth of the reference node of the target object in the camera coordinate system; The second detection module 83 is configured to determine, based on the first two-dimensional position information, the relative depth, and the absolute depth of the target object, that a plurality of key points of the target object are located in the camera respectively. 3D position information in a coordinate system.

In a possible implementation manner, the recognition module 81, when recognizing the target area where the target object in the first image is located, is used to: performing feature extraction on the first image to obtain a feature map of the first image; Based on the feature map, a plurality of target bounding boxes are determined from a plurality of pre-generated candidate bounding boxes, and based on the target bounding boxes, a target area corresponding to the target object is determined.

In a possible implementation manner, the identification module 81, when determining the target area corresponding to the target object based on the target bounding box, is used to: Based on a plurality of the target bounding boxes and the feature maps, determine a feature submap corresponding to each of the target bounding boxes; A bounding box regression process is performed based on the feature sub-maps corresponding to the plurality of target bounding boxes, to obtain a target area corresponding to the target object.

In a possible implementation manner, the first detection module 82, when determining the absolute depth of the reference node of the target object in the camera coordinate system based on the target area corresponding to the target object, is used to: determining a target feature map corresponding to the bullseye image based on the target area corresponding to the target object and the first image; Performing depth recognition processing based on the target feature map corresponding to the target object to obtain the normalized absolute depth of the reference node of the target object; Based on the normalized absolute depth and the parameter matrix of the camera, the absolute depth of the reference node of the target object in the camera coordinate system is obtained.

In a possible implementation, the first detection module 82 is used to perform depth recognition processing based on the target feature map corresponding to the target object to obtain the normalized absolute depth of the reference node of the target object. : Based on the first image, an initial depth image is obtained; wherein, the primitive value of any first primitive point in the initial depth image represents the relationship between the first primitive in the first image and the first primitive the initial depth value of the second primitive point corresponding to the point position in the camera coordinate system; Based on the target feature map corresponding to the target object, the second two-dimensional position information of the reference node corresponding to the target object in the first image is determined, and based on the second two-dimensional position information and the the initial depth image, and determine the initial depth value of the reference node corresponding to the target object; Based on the initial depth value of the reference node corresponding to the target object and the target feature map corresponding to the target object, the normalized absolute depth of the reference node of the target object is determined.

In a possible implementation, the first detection module 82 determines the depth of the target object based on the initial depth value of the reference node corresponding to the target object and the target feature map corresponding to the target object. When referencing the normalized absolute depth of a node, it is used to: Perform at least one-level first convolution processing on the target feature map corresponding to the target object to obtain a feature vector of the target object; Splicing the feature vector and the initial depth value to form a splicing vector, and performing at least a second convolution process on the splicing vector to obtain a correction value of the initial depth value; The normalized absolute depth is obtained based on the corrected value of the initial depth value and the initial depth value.

In a possible implementation manner, a pre-trained neural network is deployed in the image processing device, and the neural network includes three branch networks: a target detection network, a key point detection network, and a depth prediction network. ; the target detection network is used to obtain the target area where the target object is located; the key point detection network is used to obtain the first number of key points of the target object in the first image respectively. Two-dimensional position information, and the relative depth of each key point relative to the reference node of the target object; the depth prediction network is used to obtain the absolute depth of the reference node in the camera coordinate system.

The embodiment of the present invention identifies the target area where the target object is located in the first image, and based on the target area, determines the first two-dimensional position information, each The relative depth of the key point relative to the reference node of the target object, and the absolute depth of the reference node of the target object in the camera coordinate system, so that based on the first two-dimensional position information, relative depth, and absolute depth of the target object, more accurate The three-dimensional position information of the multiple key points of the target object in the camera coordinate system is obtained.

In addition, in the embodiment of the present invention, an end-to-end target object pose detection framework is formed through three branch networks, a target detection network, a key point detection network, and a depth prediction network. Based on the framework, the first image is processed to obtain The three-dimensional position information of the multiple key points of each target object in the first image respectively in the camera coordinate system, the processing speed is faster and the recognition accuracy is higher.

For the description of the processing flow of each module in the device and the interaction flow between the modules, reference may be made to the relevant descriptions in the above method embodiments, which will not be described in detail here.

An embodiment of the present invention also provides a computer device 10, as shown in FIG. 9, which is a schematic structural diagram of the computer device 10 provided by the embodiment of the present invention, including: A processor 11 and a memory 12; the memory 12 stores machine-readable instructions executable by the processor 11, and when the computer device runs, the machine-readable instructions are executed by the processor to achieve the following step: Identifying the target area where the target object in the first image is located; Based on the target area corresponding to the target object, first two-dimensional position information of a plurality of key points representing the posture of the target object in the first image is determined, and each of the key points is relative to the target object. The relative depth of the reference node, and the absolute depth of the reference node of the target object in the camera coordinate system; Based on the first two-dimensional position information, the relative depth, and the absolute depth of the target object, three-dimensional position information of a plurality of key points of the target object in the camera coordinate system is determined.

For the specific execution process of the above instruction, reference may be made to the steps of the image processing method described in the embodiment of the present invention, which will not be repeated here.

Embodiments of the present invention further provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the steps of the image processing method described in the above method embodiments are executed. Wherein, the storage medium may be a volatile or non-volatile computer-readable storage medium.

An embodiment of the present invention further provides a computer program product, the computer program product carries a program code, and the instructions included in the program code can be used to execute the steps of the image processing method described in the above method embodiments. For details, please refer to the above method The embodiments are not repeated here.

Wherein, the above-mentioned computer program product can be specifically realized by means of hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the system and device described above, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here. In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods may be implemented in other manners. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or elements may be combined or may be Integration into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or may be distributed to multiple network units . Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a processor-executable non-volatile computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several The instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, removable hard disk, Read-Only Memory (ROM), Random Access Memory (RAM), disk or CD, etc. that can store program codes medium.

Finally, it should be noted that the above-mentioned embodiments are only specific implementations of the present invention, and are used to illustrate the technical solutions of the present invention, but not to limit them. The protection scope of the present invention is not limited thereto, although referring to the foregoing The embodiment has been described in detail the present invention, those of ordinary skill in the art should understand: any person skilled in the art who is familiar with the technical field within the technical scope disclosed by the present invention can still modify the technical solutions described in the foregoing embodiments. Or can easily think of changes, or equivalently replace some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be covered in the present invention. within the scope of protection. Therefore, the protection scope of the present invention should be based on the protection scope of the patent application.

10: Computer equipment 11: Processor 12: Memory 81: Identify the module 82: The first detection module 83: The second detection module S101~S103, S201~S202, S301~S302, S401~S403, S501~S503: Steps

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings that are used in the embodiments, which are incorporated into the specification and constitute a part of the specification. The drawings illustrate embodiments consistent with the present invention, and together with the description, are used to illustrate the technical solutions of the present invention. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as a limitation of the scope. Other related figures are obtained from these figures. 1 shows a flowchart of an image processing method provided by an embodiment of the present invention; 2 shows a flowchart of a specific method for identifying a target area where a target object is located in a first image provided by an embodiment of the present invention; 3 shows a specific example of determining a target area corresponding to a target object based on a target bounding box provided by an embodiment of the present invention; 4 shows a flowchart of a specific method for determining the absolute depth of a reference node of a target object in a camera coordinate system provided by an embodiment of the present invention; 5 shows a flowchart of another specific method for obtaining the normalized absolute depth of a reference node provided by an embodiment of the present invention; 6 shows a specific example of a target object gesture detection framework provided by an embodiment of the present invention; FIG. 7 shows a specific example of another target object gesture detection framework provided by an embodiment of the present invention; FIG. 8 shows a schematic diagram of an image processing apparatus provided by an embodiment of the present invention; FIG. 9 shows a schematic diagram of a computer device provided by an embodiment of the present invention.

S101~S103: Steps

Claims (10)

An image processing method, applied to computer equipment, the method comprising: identifying a target area where a target object is located in a first image; determining the target area based on the target area where the target object is located and the first image The target feature map of the target object; the depth recognition process is performed on the target feature map corresponding to the target object to obtain the normalized absolute depth of the reference node of the target object; based on the normalized absolute depth and the parameter matrix of the camera , obtain the absolute depth of the reference node of the target object in the camera coordinate system; based on the target area where the target object is located, determine the position of the key points of the target object in the first image respectively first two-dimensional position information, and the relative depth of each key point relative to the reference node of the target object; the first two-dimensional position information and the The relative depth and the absolute depth corresponding to the reference node determine the three-dimensional position information of the multiple key points of the target object respectively in the camera coordinate system. The image processing method according to claim 1, further comprising: obtaining the pose of the target object based on the three-dimensional position information of the multiple key points of the target object in the camera coordinate system respectively. The image processing method according to claim 1 or 2, wherein the identifying the target area where the target object in the first image is located includes: Perform feature extraction on the first image to obtain a feature map of the first image; based on the feature map, determine a plurality of target bounding boxes from a plurality of pre-generated candidate bounding boxes; The target bounding box determines the target area where the target object is located. The image processing method according to claim 3, wherein the determining the target area where the target object is located based on the plurality of target bounding boxes includes: based on the plurality of target bounding boxes and the feature map , determine the feature submaps of each of the target bounding boxes; perform bounding box regression processing on the feature submaps corresponding to the multiple target bounding boxes respectively, to obtain the target area where the target object is located; based on the normalization The absolute depth and the parameter matrix of the camera are used to obtain the absolute depth of the reference node of the target object in the camera coordinate system. The image processing method according to claim 1, wherein the performing depth recognition processing on the target feature map corresponding to the target object to obtain the normalized absolute depth of the reference node of the target object includes: The first image is determined, and the initial depth image is determined; wherein, the primitive value of any first primitive point in the initial depth image is the position of the first primitive point in the first image The initial depth value of the corresponding second primitive point in the camera coordinate system; based on the target feature map corresponding to the target object, determine the first image of the reference node corresponding to the target object. two-dimensional location information information; based on the second two-dimensional position information and the initial depth image, determine the initial depth value of the reference node corresponding to the target object; based on the initial depth value of the reference node and the target feature map, The normalized absolute depth of the reference node of the target object is determined. The image processing method according to claim 5, wherein the determining the normalized absolute depth of the reference node of the target object based on the initial depth value of the reference node and the target feature map includes: The target feature map corresponding to the target object is subjected to at least one level of first convolution processing to obtain a feature vector of the target object; the feature vector and the initial depth value are spliced to obtain a splicing vector, and the splicing The vector is subjected to at least one second convolution process to obtain a modified value of the initial depth value; the normalized absolute depth is obtained based on the modified value of the initial depth value and the initial depth value. The image processing method according to claim 1, wherein the parameter matrix includes: the focal length of the camera; the reference node of the target object is obtained based on the normalized absolute depth and the parameter matrix of the camera The absolute depth in the camera coordinate system includes: obtaining the reference node of the target object based on the normalized absolute depth, the focal length, the area of the target area, and the area of the target bounding box Absolute depth in the camera coordinate system. The image processing method according to claim 1 or 2, wherein , the image processing method is applied to a pre-trained neural network, and the neural network includes three branch networks: a target detection network, a key point detection network, and a depth prediction network; the target detection network The network is used to obtain the target area where the target object is located; the key point detection network is used to obtain the first two-dimensional position information, and the relative depth of each key point relative to the reference node of the target object; the depth prediction network is used to obtain the absolute depth of the reference node in the camera coordinate system. A computer device, comprising: a processor and a memory connected to each other, the memory stores machine-readable instructions executable by the processor, and when the computer device runs, the machine-readable instructions are executed by the processor The steps are executed to realize the image processing method according to any one of claim items 1 to 8. A computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, executes the steps of the image processing method according to any one of claims 1 to 8.

Images