JP2022541709A - Attitude detection and video processing method, device, electronic device and storage medium - Google Patents

Abstract

The present invention relates to attitude detection and video processing methods and devices, electronic equipment and storage media. The pose detection method comprises obtaining a target image, obtaining continuous depth information and position information of a target object in the target image according to the target image, and according to the continuous depth information and the position information, the determining the pose of the target subject. Through the above process, the pose of the target object can be detected more accurately, and the accuracy and effect of pose detection can be improved. [Selection drawing] Fig. 1

Description

This application is based on and claims priority from a Chinese patent application with application number 202010566388.7 filed with the Chinese Patent Office on June 19, 2020. , the entire content of the Chinese patent application is incorporated herein by reference.

The present invention relates to the field of image processing technology, and more particularly to pose detection and video processing methods, devices, electronic devices and storage media.

3D human body pose estimation refers to estimating the 3D position of a human body in an image or a video. This task is one active research topic in the field of computer vision and is a key step in many application programs (eg motion identification, human-computer interaction and autonomous driving). Achieving highly accurate 3D position information prediction via input images is one of the current pressing problems.

The present invention proposes a technical solution for pose detection.

According to one aspect of the invention, there is provided a pose detection method, the method comprising:
obtaining a target image; obtaining continuous depth information and position information of a target object within the target image according to the target image; and determining a pose of the target object according to the continuous depth information and the position information. including doing and

In one possible implementation, obtaining continuous depth information and position information of a target object in the target image according to the target image includes passing the target image through a first neural network model to: obtaining continuous depth and location information of a target subject; and wherein said first neural network model is obtained by training via first training data and second training data, wherein: The first training data are training images including a training object, and the second training data include continuous depth information of the training object and position information of the training object.

In one possible implementation, determining the pose of the target object according to the continuous depth information and the position information includes: applying the continuous depth information and the position information to the target object through a second neural network model; and training the second neural network model via second training data and third training data, wherein the second training data is a training object continuous The third training data includes depth information and position information of the training object, and the third training data includes the pose of the training object.

In one possible implementation, the second training data is generated according to third training data, the third training data includes the posture of the training object, and the second training data is generated according to the third training data. generating comprises obtaining discrete depth information of the training object and position information of the training object according to the pose of the training object in the third training data; and processing at least a portion of the discrete depth information. and obtaining continuous depth information of the training object; and generating the second training data according to the continuous depth information of the training object and the position information of the training object.

In one possible implementation, processing the at least a portion of the discrete depth information to obtain continuous depth information for the training target comprises establishing at least one connection corresponding to at least a portion of the discrete depth information. obtaining; determining continuous depth sub-information of at least one connection according to discrete depth information corresponding to the connection; obtaining information.

In one possible implementation, determining continuous depth sub-information of at least one of said connections according to discrete depth information corresponding to said connections comprises: via linear interpolation according to discrete depth information corresponding to said connections; obtaining first continuous depth sub-information of at least one point on a connection; determining a connection range corresponding to at least one said connection; and corresponding to said connection according to said first continuous depth sub-information. Determining second continuous depth sub-information for at least one point within a connection range, and obtaining continuous depth sub-information corresponding to said connection according to said first continuous depth sub-information and/or second continuous depth sub-information. to obtain continuous depth sub-information for the at least one connection.

In one possible implementation, determining second continuous depth sub-information of at least one point within a connection range corresponding to said connection according to said first continuous depth sub-information is performed if said connection range corresponds to said connection. using the discrete depth information corresponding to the connection as second continuous depth sub-information for at least one point within the connection range, if within a preset range of the corresponding discrete depth information; and , the first point of at least one point within said connection range according to the first continuous depth sub-information closest to the point within said connection range within said connection, if it is outside the preset range of discrete depth information corresponding to said connection; and obtaining two consecutive depth sub-information.

According to one aspect of the invention, there is provided a video processing method, the method comprising:
performing image acquisition for a current scenario to obtain an acquired video; selecting from the acquired video a target image comprising at least two frames of the target object; performing pose detection on the target object in the target images of at least two frames via the described pose detection method to determine at least two poses of the target object in the acquired video; including.

In one possible implementation, obtaining successive poses of the target object according to at least two poses of the target object and time of frames in the acquired video; and tracking.

According to one aspect of the present invention, there is provided an attitude detection device, the device comprising:
a target image acquisition unit configured to acquire a target image; an information acquisition unit configured to acquire continuous depth information and position information of a target object in the target image according to the target image; a pose determiner configured to determine a pose of the target object according to the continuous depth information and the position information.

In one possible implementation, the information acquisition unit passes the target image through a first neural network model to acquire continuous depth information and position information of a target object in the target image, and is configured to be obtained by training via first training data and second training data, wherein said first training data is a training image containing a training target, and said second training data is , continuous depth information of the training object, and location information of the training object.

In one possible implementation, the pose determiner acquires the pose of the target object from the continuous depth information and the position information through a second neural network model, the second neural network model configured to train via two training data and third training data, wherein said second training data includes continuous depth information of a training object and position information of said training object; and said third training data contains the posture of the training target.

In one possible implementation, the second training data is generated according to third training data, the third training data includes the posture of the training object, and the second training data is generated according to the third training data. generating comprises obtaining discrete depth information of the training object and position information of the training object according to the pose of the training object in the third training data; and processing at least a portion of the discrete depth information. and obtaining continuous depth information of the training object; and generating the second training data according to the continuous depth information of the training object and the position information of the training object.

In one possible implementation, processing the at least a portion of the discrete depth information to obtain continuous depth information for the training target comprises establishing at least one connection corresponding to at least a portion of the discrete depth information. obtaining; determining continuous depth sub-information of at least one connection according to discrete depth information corresponding to the connection; obtaining information.

In one possible implementation, determining continuous depth sub-information of at least one of said connections according to discrete depth information corresponding to said connections comprises: via linear interpolation according to discrete depth information corresponding to said connections; obtaining first continuous depth sub-information of at least one point on a connection; determining a connection range corresponding to at least one said connection; and corresponding to said connection according to said first continuous depth sub-information. Determining second continuous depth sub-information for at least one point within a connection range, and obtaining continuous depth sub-information corresponding to said connection according to said first continuous depth sub-information and/or second continuous depth sub-information. including doing and

In one possible implementation, determining second continuous depth sub-information of at least one point within a connection range corresponding to said connection according to said first continuous depth sub-information is performed if said connection range corresponds to said connection. using the discrete depth information corresponding to the connection as second continuous depth sub-information for at least one point within the connection range, if within a preset range of the corresponding discrete depth information; and , the first point of at least one point within said connection range according to the first continuous depth sub-information closest to the point within said connection range within said connection, if it is outside the preset range of discrete depth information corresponding to said connection; and obtaining two consecutive depth sub-information.

According to one aspect of the invention, there is provided a video processing device, the device comprising:
an image acquisition unit configured to perform image acquisition for a current scenario to obtain an acquired video; and to select from the acquired video at least two frames of a target image comprising a target object. and performing pose detection on the target object in at least two frames of the target image to obtain a a pose obtaining unit configured to determine at least two poses of the target object of .

In one possible implementation, the video processing unit further obtains successive poses of the target object according to at least two poses of the target object and time of frames in the acquired video; It is configured to track the target object according to its pose.

According to one aspect of the present invention, an electronic device is provided, the electronic device comprising:
a processor and a memory configured to store processor-executable instructions, wherein the processor is configured to invoke instructions stored by the memory to perform the pose detection method described above. be done.

According to one aspect of the present invention, there is provided a computer readable storage medium having computer program instructions stored thereon, said computer program instructions, when executed by a processor, implementing the pose detection method described above.

According to one aspect of the invention, there is provided a computer program product comprising computer readable code, said computer readable code being executed in an electronic device and, when executed by a processor within said electronic device, the attitude detection method described above. Realize

In an implementation of the present invention, through obtaining the target image and the continuous depth information and position information of the target object within the target image, the pose of the target object is determined according to the continuous depth information and the position information. Through the above process, the continuous depth information of the target object can be used to predict the pose of the target object. It can accurately detect and improve the accuracy and effectiveness of attitude detection.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention.

Other features and aspects of the present invention will become apparent from the following detailed description of illustrative embodiments with reference to the drawings.

The drawings herein are incorporated in and constitute a part of this specification, and these drawings illustrate embodiments consistent with the present invention, and together with the description, the technical solution of the embodiments of the present invention. used to describe policies.
Fig. 3 shows a flow chart of a posture detection method according to an embodiment of the present invention; FIG. 4 shows a schematic diagram of acquiring target object continuous depth information according to one embodiment of the present invention; Fig. 3 shows a schematic diagram of determining a connection range according to an embodiment of the present invention; 1 shows a flow chart of a video processing method according to an embodiment of the present invention; 1 shows a schematic diagram according to one application of the invention; FIG. 1 shows a block diagram of an attitude detection device according to an embodiment of the present invention; FIG. 1 shows a block diagram of a video processing device according to one embodiment of the present invention; FIG. 1 shows a block diagram of an electronic device according to an embodiment of the invention; FIG. 1 shows a block diagram of an electronic device according to an embodiment of the invention; FIG.

Various exemplary embodiments, features, and aspects of the invention are described in detail below with reference to the drawings. The same reference numbers in the drawings indicate elements of the same or similar function. Although various aspects of the illustrative embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specified otherwise.

As used herein, the proprietary term "exemplary" means "used as an example, example, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior or preferred over other embodiments.

The term "and/or" herein is a mere association describing the subject being associated and indicates that three types of relationships may exist, e.g. , A and B exist at the same time, and B exists independently. Further, the term "at least one" herein refers to any combination of one of the plurality or at least two of the plurality, including, for example, at least one of A, B, C indicates that it includes any one or more elements selected from the set consisting of A, B and C.

Moreover, numerous specific details are given in the following specific embodiments in order to better explain the embodiments of the present invention. It should be understood by those skilled in the art that the present invention may be similarly practiced without some of the specific details. In some instances, methods, means, elements and circuits that are well known to those skilled in the art have not been described in detail in order to emphasize the subject matter of this disclosure.

FIG. 1 shows a flow chart of an attitude detection method according to an embodiment of the present invention, which can be applied to an attitude detection device, such as a terminal device, a server or other processing equipment. could be. Here, the terminal equipment includes user equipment (UE), mobile equipment, user terminal, terminal, mobile phone, cordless phone, personal digital assistant (PDA), handheld equipment, computing equipment, in-vehicle equipment, It can be a wearable device or the like.
In some possible implementations, the pose detection method can be implemented by a processor via calling computer readable instructions stored in memory.

As shown in FIG. 1, the pose detection method may include the following steps.

In step S11, a target image is acquired.

In step S12, obtain continuous depth information and position information of the target object in the target image according to the target image.

In step S13, the pose of the target object is determined according to the continuous depth information and the position information.

Here, the target image can be any image for posture detection, and its implementation mode is not limited in the embodiments of the present invention and can be flexibly determined according to actual cases. In one possible implementation, the target image may be a human body image for human body pose detection, and in one possible implementation, the target image may be a machine image for machine pose detection (such as robot pose detection). It may be an image or the like. The target object can be an object for performing pose detection in the target image, and its implementation mode can be flexibly determined according to the actual case of the target image and pose detection. , if the target image is a human body image for human body pose detection, the target object may be the complete human body included in the target image, or the arm, thigh, torso, head, etc. included in the target image. In one possible implementation, if the target image is a machine image for machine pose detection, the target object is the position of the machine object contained in the target image. It may be the entirety or some part of the machine contained in the target image, such as the arm of the machine or a walking mechanism that supports the machine to move. The embodiments disclosed below all take the target object to be the human body, and the pose detection is human three-dimensional pose detection. It can be flexibly expanded with reference to the embodiments and will not be described in detail.

The number of target images is not limited in the embodiment of the present invention, and may be one or more. The pose detection can be performed for multiple target images at the same time, and can be flexibly determined according to the actual needs of detection. The number of target objects included in the target image is not limited in the embodiments of the present invention. In such an implementation method, one target image may include a plurality of target objects at the same time, and may be flexibly determined according to actual cases.

In step S11, the method of acquiring the target image is also not limited to the embodiments of the present invention, and in one possible implementation method, the posture detection device actively acquires images (such as taking pictures or videos) of the target object. In one possible implementation, the pose detection device may passively receive the target image, how to implement It may be determined flexibly according to the actual situation of the attitude detection device, and the embodiments of the present invention are not limited.

The continuous depth information of the target object can be the depth information of a plurality of continuous sampling points of the target object, and in the process of detecting the target object, the target object's head, neck, shoulders, elbows, hands, waist, knees and feet. can detect several keypoints of the target object, such as , are discrete from each other, e.g., the distance of the arm between the target subject's elbow and wrist, so it can be seen that the depth information of the keypoints is typically discrete depth information. Continuous depth information is continuous information obtained by performing depth estimation for all points between adjacent discrete keypoints based on the rigid structure of the human torso. can be represented as a continuous depth feature map.

Therefore, obtaining continuous depth information of the target object in the target image in step S12 may be obtaining depth information of a plurality of consecutive sampling points contained within the coverage of the target object in the target image. , the method of acquiring the continuous depth information can be flexibly selected according to actual cases.

In the process of performing pose detection on a target object, what we finally need to obtain may be the 3D coordinates of the target object, and the depth information is one dimension of the 3D coordinates. So, in one possible implementation, step S12 may require two-dimensional position information of the target object to help determine the pose of the target object. The implementation form of the location information is not limited, and can be flexibly selected according to actual cases. In one possible implementation, the location information of the target object can include a two-dimensional heat map of the target object, wherein: The number and types of 2D heatmaps included can be flexibly determined according to the actual case. It may include a 2D heatmap of the torso, and so on.

In some embodiments of the present invention, the manner of obtaining the location information of the target object can also be flexibly determined according to actual cases, please refer to each disclosed embodiment for details.

In one possible implementation, step S12 may include passing the target image through a first neural network model to acquire continuous depth and position information of the target object within the target image.

The first neural network model can be a neural network model with the function of extracting continuous depth information and position information of the target object, and its implementation is not limited in the embodiments of the present invention, and can be flexibly set according to actual cases. can do. Since its input data is the target image, and its output data is continuous depth information of the target object within the target image, in one possible implementation, the first neural network model uses the first training data and the second training data It can be a neural network model obtained by training a first initial neural network through data. Here, the first training data may be training images including the training target, and the second training data may include continuous depth information of the training target and location information of the training target. where the training object included in the training images may be the same as the target object, or may be a related object that belongs to the same type as the target object but is not the same object as the target object, etc.; Here, the implementation can refer to the implementation of the target object and will not be repeated here. The implementation of the training image can refer to the implementation of the target image and will not be repeated here. How to obtain the second training data and how to obtain the continuous depth information and position information of the target object in the second training data can be referred to each subsequent disclosed embodiment, which will be described in detail here. do not do. The number of first training data and second training data is not limited in the embodiments of the present invention and can be flexibly determined according to actual cases.

The first initial neural network may be a general neural network such as VGG, ResNet or GoogleNet, or a lightweight neural network such as MobileNet V2 or ShuffleNet V2, where any neural network The selection as the first initial neural network can be flexibly selected according to actual needs.

Obtaining a first neural network model with continuous depth information extraction function through training by inputting the first training data and the second training data into the first initial neural network, and using the first neural network model to obtain the continuous depth information and position information of the target object in the target image; Information can be acquired, and the accuracy of the acquired continuous depth information is high, which reduces the difficulty of attitude detection and improves the accuracy of attitude detection. At the same time, training over a second training data containing continuous depth information to obtain a first neural network model to facilitate the first neural network model to learn more structured depth information. , thereby giving the first neural network model a more robust generalization capability and facilitating it to be suitable for more application scenarios.

In one possible implementation, step S12 includes:
passing the target image through a fourth neural network model to obtain continuous depth information of the target object in the target image;
passing the target image through a third neural network model to obtain position information of the target object within the target image.

According to the above disclosed embodiments, in one possible implementation, continuous depth information and position information of the target object can also be obtained respectively through inputting the target image into two different neural networks respectively. . In this case, the fourth neural network model may be a neural network model that only has a target object continuous depth information extraction function, and the third neural network model is a target object position information extraction function only. obtain. So, in one possible implementation, the first neural network model can also be a neural network model obtained by training a fourth initial neural network via the first training data and the fourth training data, 1 The implementation of training data can refer to the above disclosed embodiments and will not be repeated here. In this case, the implementation of the fourth training data can be flexibly changed, for example the fourth training data may contain only continuous depth information of the training object. The location information of the target object can be used as training data for a third neural network model jointly with the first training data to train a third initial neural network training, thereby obtaining a third neural network model. can. The implementation of the third initial neural network can be flexibly determined, and here can refer to the first initial neural network, which will not be repeated here.

By inputting the target image into the fourth neural network model and the third neural network model respectively, the continuous depth information and the position information of the target object are obtained respectively, and through the above process, the fourth neural network model and the third neural network model. improving the accuracy of the fourth neural network model and the third neural network model by simplifying the function of the neural network model, thereby effectively improving the depth of continuous depth information and location information obtained; It is possible to improve the accuracy of orientation detection.

The implementation method of determining the pose of the target object according to the continuous depth information in step S13 is also not limited in the embodiments of the present invention. In one possible implementation, step S13 may involve obtaining the pose of the target object through continuous depth information and position information through a second neural network model.

The second neural network model can be a neural network model with posture detection function, and its implementation is not limited by the embodiments of the present invention and can be flexibly set according to actual cases. Since its input data is continuous depth and position information, and its output data is the pose of the target subject, in one possible implementation, the second neural network model uses the second training data and the third training data. may be a neural network model obtained by training a second initial neural network via Here, the second training data may include the continuous depth information of the training object and the position information of the training object, and the third training data may include the posture of the training object. Here, the implementation of the second training data is the same as the second training data mentioned in the above disclosed embodiments, and will not be repeated here. The third training data can be the pose prediction result of the training target, and its implementation is not limited. It can be used as a target posture prediction result.

The implementation of the second initial neural network can refer to the implementation of the first initial neural network described above, that is, it can be configured by a general neural network, and in one possible implementation, the second initial The neural network can be a neural network composed of convolution layers and pooling layers connected in sequence, where the number of convolution layers and pooling layers, the connection order, etc. are not limited in the embodiments of the present invention. .

By inputting the second training data and the third training data into the second initial neural network and training it, obtaining a second neural network model having a posture detection function, and using the second neural network model, continuous A pose of the target object is obtained according to the depth information and the position information. Through the above process, the second training data containing continuous depth information can be used to train to obtain a second neural network model, providing more useful information to the second neural network model to provide more A correct prediction result can be obtained to achieve more accurate pose prediction, and multiple target images or multiple target objects can be processed simultaneously to facilitate obtaining multiple pose detection results. Improve detection accuracy and convenience.

In an implementation of the present invention, through obtaining the target image and the continuous depth information and position information of the target object within the target image, the pose of the target object is determined according to the continuous depth information and the position information. Through the above process, continuous depth information can be used as intermediate supervisory information in the target object pose prediction process to realize pose detection, thereby more accurately detecting the pose of the target object and improving the accuracy of pose detection. It can improve accuracy and effectiveness.

As can be seen from each of the above disclosed embodiments, in one possible implementation, via steps S11 to 13, the key to realizing pose detection is via training data containing continuous depth information of the target object. , training a first neural network model and a second neural network model. For the image used as one training data, the depth information of key points can be collected by wearing a specific sensor device on a real person, and the acquisition of depth information of each continuous sampling point is realized. When labeling by manual labeling method, a huge workload occurs, consuming time and labor.

Thus, in one possible implementation, the second training data can be generated according to third training data, the third training data including the pose of the training target.

Here, generating the second training data according to the third training data may include the following steps.

At S21, the discrete depth information of the training object in the third training data and the position information of the training object are obtained according to the posture of the training object in the third training data.

At S22, at least a portion of the discrete depth information is processed to obtain continuous depth information for the training target.

At S23, the second training data is generated according to the continuous depth information of the training object and the position information of the training object.

Here, the implementation manner of the third training data is the same as the above disclosed embodiments, and will not be repeated here. According to the above-disclosed embodiment, in one possible implementation, the 3D position information of the target's keypoints can be used as the target's pose prediction result, so correspondingly, the training target's The three-dimensional position information of the key points can be the pose prediction result of the training object, that is, the third training data can directly include the discrete depth information and position information of the training object, that is, step Via S21, the discrete depth information and location information of the training object can be directly obtained according to the third training data.

As described in the disclosed embodiments above, the discrete depth information can be the depth information of certain keypoints or sampling points, so in step S21 the number of discrete depth information acquired is the number of the third training It can be determined according to the number of key points in the data or the number of sampling points with different depth values, and is not limited in the embodiments of the present invention.

In the process of acquiring the continuous depth information of the training target through step S22, it may be obtained by processing according to each piece of discrete depth information acquired, or the acquired discrete depth information may be processed according to the information It may be obtained by selectively processing a part, where whether to use all of the acquired discrete depth information or to use a part of the discrete depth information, and which discrete depth information How to choose to use the information can be flexibly chosen according to actual cases, and is not limited in the embodiments of the present invention.

After obtaining the continuous depth information of the training object, the second training data can be generated according to the continuous depth information and the location information of the training object via step S23. The implementation method of step S23 is not limited in the embodiments of the present invention. In one possible implementation method, the continuous depth information can be embodied in the form of a continuous depth feature map, and the position information can be a two-dimensional heat map. so that the continuous depth information and the location information can be directly jointly used as the second training data.

significantly reducing the difficulty and workload of obtaining the second training data by generating the second training data including continuous depth information and position information of the training target according to the third training data including the training target posture; The convenience of the pose detection implementation process can be improved.

In some embodiments of the present invention, how to process at least a portion of the discrete depth information in step S22 to obtain the continuous depth information of the training target is flexibly determined according to actual cases. can do. In one possible implementation, step S22 may include the following steps.

At step S221, at least one connection corresponding to at least a portion of the discrete depth information is obtained.

In step S222, determine continuous depth sub-information of at least one connection according to the discrete depth information corresponding to the connection.

At step S223, the at least one continuous depth sub-information is statistically obtained to obtain continuous depth information for training.

Here, the implementation form of connection corresponding to discrete depth information can be flexibly determined according to actual cases. In one possible implementation, the connection can be a virtual connection, for example connecting keypoints corresponding to discrete depth information to obtain a connecting line, thereby making the connecting line correspond to the discrete depth information. connection. In one possible implementation, the connections can be substantially unconnected correspondences, i.e. determining which discrete depth information corresponding keypoints can have a connection relationship, Do not connect the keypoints corresponding to these discrete depth information via connecting lines. Each subsequent disclosed embodiment will all be described as an example of using the obtained connection as a substantial connection line, and the method of obtaining a connection line by not substantially connecting is described in each subsequent It can be flexibly expanded with reference to the disclosed embodiments and will not be described in detail.

In the above disclosed embodiments, the process of acquiring continuous depth information of the target object may include processing and acquiring all of the acquired discrete depth information, or processing a portion of the discrete depth information. I proposed that it may be to process and acquire. Therefore, in the process of acquiring at least one connection corresponding to at least a portion of the discrete depth information in step S221, even if the connection to any two discrete depth information among all the acquired discrete depth information It may be a connection that selects a part of the discrete depth information, or it may be a selective connection according to the position of the discrete depth information in the training object, or it may be flexibly decided according to the actual needs. .

In one possible implementation, if the discrete depth information is the depth information of the keypoints of the training object to be acquired, it is possible to selectively connect the discrete depth information of the keypoints according to the position of the keypoints in the training object. For example, if the discrete depth information obtained includes wrists, elbows, shoulders and head, based on the torso structure of the human body, discrete depth information for wrists and elbows, discrete depth information for elbows and shoulders, , and the discrete depth information of the shoulder and the head can be connected respectively to obtain three connection lines, and other connection methods, such as connecting the wrist and the head, connecting the elbow and the head, etc., are similar to the human body There is no need to connect these discrete depth information as they are inconsistent with the body distribution scheme.

After connecting at least two pieces of discrete depth information to obtain at least one connection, continuous depth sub-information of at least one connection can be determined according to the discrete depth information corresponding to the connection via step S222. .

In one possible implementation, if the connection is a connection line, continuous depth sub-information corresponding to at least one connection line can be obtained according to the discrete depth information corresponding to the connection line end point. Here, the discrete depth information corresponding to the connecting line end point is the discrete depth information connected in the process of acquiring the connecting line in step S221. By way of example, in the above-disclosed embodiment, the discrete depth information of the wrist and elbow, the discrete depth information of the elbow and shoulder, and the discrete depth information of the shoulder and head can be connected to obtain three connecting lines. In one example, the continuous depth sub-information corresponding to the wrist-elbow connection line can be obtained through the wrist-discrete depth information and the elbow-discrete depth information, and the elbow-shoulder connection The continuous depth sub-information corresponding to the line can be obtained by the discrete depth information of the elbow and the discrete depth information of the shoulder. information and head discrete depth information. Here, the process of obtaining the continuous depth sub-information corresponding to the connecting line according to the discrete depth information of the connecting line end point can refer to the following disclosed embodiments, and will not be described in detail herein.

FIG. 2 shows a schematic diagram of obtaining training object continuous depth information according to one embodiment of the present invention, as shown in the figure, in one possible implementation scheme, the training object obtained from the third training data. The discrete depth information may include the training target wrist discrete depth information Pw and the training target hand elbow discrete depth information Pe, in which case the wrist discrete depth information Pw and Pe are connected to obtain the training target arm connection line. Then, continuous depth information corresponding to the arm connection line can be obtained according to the wrist discrete depth information Pw and the wrist discrete depth information Pe.

After obtaining the continuous depth sub-information corresponding to at least one connection, via step S223, these continuous depth sub-information can be statistically obtained to obtain the continuous depth information for the training object. The statistical method can be flexibly determined according to actual cases, and is not limited in the embodiments of the present invention. In one possible implementation, the continuous depth sub-information corresponding to each connection obtained can be combined and jointly used as the continuous depth information corresponding to the training object, and in one possible implementation, several It is also possible to remove the continuous depth sub-information that is obviously in error or that exceeds the coverage of the training target, and retain the remaining continuous depth sub-information to be jointly used as the continuous depth information of the training target, where: The screening method should be selected flexibly according to actual cases.

Obtaining continuous depth sub-information for at least one connection according to the discrete depth information corresponding to the connection through obtaining at least one connection corresponding to at least a portion of the given discrete depth information, thereby obtaining at least one The continuous depth information of the training target can be obtained by statistically analyzing the two continuous depth sub-information. Establishing a relationship with the continuous depth information to facilitate deriving the continuous value according to the discrete values, more conveniently determine the continuous depth information of the training object, and further improve the difficulty and task of obtaining the second training data. It can reduce the load and improve the convenience of the attitude detection implementation process.

The above disclosed embodiments suggest that the implementation of step S222 is not limited, and in one possible implementation, step S222 may include the following steps.

In step S2221, obtain the first continuous depth sub-information of at least one point on the connection via linear interpolation according to the discrete depth information corresponding to the connection.

In step S2222, a connection range corresponding to at least one connection is determined.

In step S2223, a second continuous depth sub-information of at least one point within the connection range corresponding to the connection is determined according to the first continuous depth sub-information.

In step S2224, obtain continuous depth sub-information corresponding to the connection according to the first continuous depth sub-information and/or the second continuous depth sub-information.

Here, the first continuous depth sub-information may be depth information located at a point on a connection or a connection line. As described in the disclosed embodiments above, in one possible implementation, at least one connecting line can be obtained by connecting the discrete depth information, and the discrete depth Since the information is connected discrete depth information, for each connecting line obtained, there are two points on the connecting line, and the positions of these two points on the connecting line are known. At the same time, the corresponding depth information is also known, so for this connecting line, the depth information corresponding to the remaining points on the connecting line are all the depth information of the two known points. Therefore, in one possible implementation, via step S2221, using the discrete depth information corresponding to the connecting line end points, the remaining points on the connecting line are calculated via linear interpolation. can be obtained from the first continuous depth sub-information of Here, whether to obtain the first continuous depth sub-information for each point on the connecting line or to obtain the first continuous depth sub-information for a part of the points on the connecting line can be flexibly determined according to needs. In one possible implementation, through the discrete depth information corresponding to the end point of the connecting line, the relational function between the first continuous depth sub-information on the connecting line and the position of the point can be obtained by linear interpolation, such as to obtain the position of the point of the first continuous depth sub-information regardless of whether it is all points on the connecting line or some of the points that need to obtain the first continuous depth sub-information. can be obtained by substituting it into the relational function.

The second continuous depth sub-information may be depth information of points within the connection range corresponding to the connection. As described in each disclosed embodiment above, the discrete depth information obtained may be the discrete depth information of a portion of the keypoints to be trained, still taking FIG. , if the discrete depth information to be acquired is the depth information Pw and Pe of the wrist keypoint and the hand-elbow keypoint, the connecting line obtained by connecting Pw and Pe completely covers the arm range of the training target. Since it may not be possible to fully reflect the continuous depth information of the arm when only the first continuous depth sub-information on the connection line is acquired, one possible implementation method is to extend the connection line in a suitable manner. Execute to determine a connecting range corresponding to the connecting line and obtain first continuous depth sub-information for points on the connecting line, and further obtain second continuous depth sub-information for remaining points within the connecting range. can be obtained.

The method of determining the connection range is not limited in the embodiments of the present invention. One possible implementation method is to take the connection line as the center and extend it to the boundary of the training object to obtain the connection range corresponding to the connection line. can be done. In one possible implementation, in order to reduce the workload and improve the efficiency of determining the connection range, a preset radius is set, the connection line is the center, and the connection line is extended to the preset radius. connection range can be obtained. This preset radius setting method can be flexibly selected according to actual cases, and is not limited to the embodiments of the present invention. , then, with the connecting line as the center line, one can construct a rectangle whose width is 2R and whose length is the same as that of the connecting line; On the side, two semicircles with radius R are constructed respectively, the value of R can be flexibly set according to the actual case, the semicircle with two radii R has a width of 2R Construct a connection range jointly with a rectangle. FIG. 3 shows a schematic diagram of determining the connection range according to an embodiment of the present invention, as shown in the figure, in one example, taking the connection line of Pw and Pe as the center line, according to the above scheme, the connection You can construct a connection range corresponding to .

After determining the connected range, the second continuous depth sub-information of the points included in the connected range can be determined according to the first continuous depth sub-information, where how to determine is described in each of the following disclosures. Reference can be made to the embodiments, which are not described in detail herein.

After obtaining the first continuous depth sub-information and the second continuous depth sub-information, the continuous depth sub-information can be obtained according to both, wherein the first continuous depth sub-information is the continuous depth sub-information, or Either the second continuous depth sub-information is the continuous depth sub-information, or the first continuous depth sub-information and the second continuous depth sub-information are jointly the continuous depth sub-information; Whether to use sub-information can be flexibly selected according to actual cases, and is not limited in the embodiments of the present invention.

As described in each of the disclosed embodiments above, in one possible implementation, the connections of the discrete depth information may be correspondences rather than substantial connections, in which case the discrete depths of the connections Although there is no connecting line between the information, the positional relationship based on the discrete depth information of the connection can refer to the implementation method of the above-disclosed embodiment, through linear interpolation, the different points in the coverage of both connections. Determining depth information, where the determination process can refer to the above disclosed embodiments and will not be repeated here.

obtaining first continuous depth sub-information of at least one point on the connection by linearly interpolating according to the discrete depth information corresponding to the connection; obtaining second continuous depth sub-information of at least one point within the connected range according to the information, further obtaining continuous depth information according to the first continuous depth sub-information and/or the second continuous depth sub-information; Get continuous depth sub-info for one connection. Through the above process, on the one hand, we can obtain the continuous depth sub-information on the connection more conveniently, and on the other hand, we can extend the coverage of the connection to obtain the continuous depth sub-information within the connection range corresponding to the connection. Sub-information can be obtained, thereby obtaining more comprehensive and accurate continuous depth information of the training object for training the neural network model, and further improving the accuracy of pose detection.

In one possible implementation, step S2223 may include the following steps.

In step S22231, if the connection range is within a preset range of discrete depth information corresponding to the connection, use the discrete depth information corresponding to the connection as second continuous depth sub-information for at least one point within the connection range. do.

In step S22232, if the connection range is outside the preset range of the discrete depth information corresponding to the connection, at least one Obtain the second continuous depth sub-information of the point.

Here, the preset range of discrete depth information corresponding to a connection can be a part of the range included in the connection range, where the size and definition method of the preset range are flexible depending on the connection range determination. can decide. As can be seen from the above-disclosed embodiments, in one possible implementation, a preset range can be used to distinguish between the cover area of the connecting line end point and the covering area of other points on the connecting line. That is, the second continuous depth sub-information corresponding to points within the preset range can be determined by the discrete depth information of the end point, and the second continuous depth sub-information corresponding to points outside the preset range can be determined by , can be determined by the first successive depth sub-information on the connecting line. In one possible realization method, the structure method of the connecting area can be as described in the above disclosed embodiment, i.e., as the connecting line and center line, one width is 2R, and the length and connecting line are the same, and at the two endpoints of the connecting lines, toward one side away from the rectangle, construct two semicircles with radii R, in this case the two radii are R A semicircle can be considered as a preset range of connection endpoints, and a remaining rectangle with a width of 2R can be considered as a non-preset range of connection endpoints.

As can be seen from step S22231, in one possible implementation, for points within the preset range, the discrete depth information of the connecting line end points corresponding to the preset range can be used as the second continuous depth sub-information. Taking FIG. 3 as an example, as can be seen from FIG. 3, for a semicircle with radius R constructed with Pw as the center, the second continuous depth sub-information of the points within this semicircle are all For a semicircle with radius R that is the same as the discrete depth information of Pw and similarly constructed with Pe as the center of the circle, the second continuous depth sub-information of the points within this semicircle are all the discrete depth information of Pe Same as information.

As can be seen from step S22232, in one possible implementation, for a point outside the preset range (where the point is P), the first continuous depth sub information can be used as the second continuous depth sub-information for the point P. Here, how to determine the point closest to the point P on the connection line, the determination method is not limited. , a vertical line can be drawn from the point P to the connecting line to obtain the vertical foot P′, and the first continuous depth sub-information of P′ can be used as the second continuous depth sub-information of the point P. can.

Through the above process, the second continuous depth sub-information of the preset range covered by the discrete depth information corresponding to the connection and the second continuous depth sub-information of the range of the connection cover are obtained respectively, and all connection ranges are obtained. can obtain the second continuous depth sub-information corresponding to the points in the second continuous depth sub-information, such a method for determining the second continuous depth sub-information is relatively simple, the amount of computation is small, and the second training data is It reduces the acquisition difficulty and workload, and improves the convenience and accuracy of the pose detection implementation process.

In one possible implementation, the target subject pose determined by each of the above disclosed embodiments can also be applied to different scenarios such as motion identification, human-computer interaction and autonomous driving. In one possible implementation, the pose of the target object determined by each disclosed embodiment above can be applied to the processing of the video data. Therefore, an embodiment of the present invention further proposes a video processing method.

FIG. 4 shows a flow chart of a video processing method according to an embodiment of the present invention, the method can be applied to a video processing device, and the implementation of the video processing device is the same as in the above disclosed embodiments. It may be the same as or different from the posture detection device, and its implementation can refer to the above disclosed embodiments and will not be repeated here.
In some possible embodiments, the video processing method may be implemented via a scheme in which a processor invokes computer readable instructions stored in memory.

As described in FIG. 4, the video processing method may include the following steps.

In step S21, image acquisition is performed for the current scenario to obtain an acquired video.

At step S22, at least two frames of target images containing the target object are selected from the acquired video.

In step S23, pose detection is performed on the target object in the target images of the at least two frames via the pose detection method in each of the disclosed embodiments above to obtain at least two images of the target object in the acquired video. determine posture.

Here, the implementation scheme of the target object is the same as each disclosed embodiment in pose detection above, and will not be repeated here. The current scenario can be any scenario, including the target subject, and is not limited to each disclosed example below. In one possible implementation, the current scenario can be a pedestrian detection scenario, an autonomous driving scenario, an object capture scenario in a classroom, and a company environment detection scenario.

The method of performing image acquisition for the current scenario can be flexibly determined depending on the actual scenario of the current scenario, for example, if the current scenario is a pedestrian detection scenario, the Image acquisition may be performed via imaging equipment, and if the current scenario is an autonomous driving scenario, image acquisition may be performed via imaging equipment deployed in the vehicle.

The implementation of video acquisition can be flexibly determined according to the actual case of image acquisition, and is not limited in the embodiments of the present invention.

After acquiring the video via step S21, a target image comprising at least two frames of the target object can be selected from the acquired video via step S22, where the realization of the target image also includes the above postures. It is the same as each disclosed embodiment in detection and will not be repeated here. The method of selecting a target image containing at least two frames of the target object from the acquired video can be flexibly determined according to actual cases, and is not limited to the following disclosed embodiments. In one possible implementation, target object detection is performed on at least a portion of the frames of the acquired video, and then target objects are selected randomly or based on criteria such as image quality from the detected frames. Therefore, it is possible to select at least a part of the frame from them as the target image.

After obtaining a target image containing at least two frames of the target object, via step S22, perform pose detection on the selected target image via the pose detection method according to any of the disclosed embodiments above. to determine at least two poses of the target object in the acquired video, where how the poses are determined can be flexibly determined according to the actual case of the target image, and the present invention Examples are not limiting.

Acquire an acquired video by performing image acquisition for the current scenario, select a target image containing at least two frames of the target object from the acquired video, and then perform image acquisition according to any of the above disclosed embodiments. Through the pose detection method, determining at least two poses of the target object in the acquired video is realized. Through the above process, the pose detection process realized based on continuous depth information can be applied to the video processing process, thereby more accurately determining multiple poses of the target object from the dynamic video. , facilitates effectively improving the accuracy of pose detection during video.

In one possible implementation, after acquiring at least two poses of the target object in the acquired video, the acquired multiple poses can be further processed, so that in one possible implementation, the present invention: A video processing method according to an embodiment may further include the following steps.

In step S24, successive poses of the target object are obtained according to the at least two poses of the target object and the time of the frames in the acquired video.

In step S25, the target object is tracked according to the continuous pose of the target object.

As described in the disclosed embodiments above, the pose of the target object within the acquired video can be determined according to the frame within the acquired video containing the target object, wherein the frame within the acquired video is time-preferred. Correspondingly, these frames sequentially prioritize the occurrences of the acquired poses according to their time in the acquired video, so as to determine the successive poses of the target object. can be done.

In some embodiments of the present invention, after determining the continuous pose of the target object, tracking can be implemented for the target object. Through the above process, the above pose detection process using continuous depth information can be used in the target object tracking process to improve tracking accuracy and achieve more efficient tracking.

Illustratively, in a city security monitoring scenario, image collection can be performed on a city via a city monitoring device to obtain collected video, and after the attitude detection device obtains the collected video, the collected video obtaining an image containing at least two frames of the target object from and through a first neural network model of at least two corresponding groups of continuous depth and position information of the target object in the image containing at least two frames of the target object and obtain at least two corresponding poses through a first neural network model according to at least two groups of continuous depth information and position information, and among the at least two poses, and the at least two poses, each pose According to the time of the frame corresponding to the continuous posture of the target object, further determine whether the behavior of the target object is dangerous behavior such as fighting, theft, robbery, etc., and determine whether the behavior of the target object is , automatically alerts and sends the collected video of the target and the determination of the dangerous behavior to the police station if it determines that the behavior is dangerous.

An exemplary application in a practical application scenario of the embodiments of the present application is described below.

3D human pose estimation is one of the fundamental tasks in the field of computer vision, and is widely applied in scenarios such as motion identification, human-computer interaction and autonomous driving to improve the accuracy of 3D human pose estimation. has become a matter of urgency.

FIG. 5 shows a schematic diagram according to one application of the present invention, as shown in the figure, an embodiment of the present invention proposes a pose detection method, which is to detect the three-dimensional pose of the human body in the target image. can be determined, and as shown in the figure, the process of pose detection in the application of the present invention is as follows.

As shown in the figure, in the application example of the present invention, the input target image is first initiated into the first neural network model (BackBone backbone neural network), and the first neural network model converts the input target image into can be processed to obtain three output results, which are respectively a two-dimensional heat map of the human body key points, a two-dimensional heat map of the human body and a depth interpolation map containing continuous depth information of the human body, and then: The three output results obtained are jointly input to the second neural network model (regression regression network) as input data, and through the processing of the second neural network model, the three-dimensional position of each continuous point in the human torso is obtained. Coordinates can be obtained.

Here, the structure of the first neural network model is constructed in the form of adding three prediction branches to a shallow network, the shallow network can perform feature extraction on the human body in the target image, and the obtained The feature extraction results are input into three prediction branches respectively, one prediction branch is used to predict a two-dimensional heat map outputting the human body keypoints, and one prediction branch is used to predict the two-dimensional heat map of the human torso. One prediction branch is used to output a depth interpolated map containing continuous depth information of the human torso. Therefore, the first neural network model is to create a training image containing the human body, a two-dimensional heat map of the human body key points, a two-dimensional heat map of the human torso, and a depth interpolation map containing continuous depth information of the human torso. , can be obtained by joint training, where a depth interpolated map containing continuous depth information of the human torso is obtained by any one of can be obtained by processing two schemes.

The structure of the second neural network model is configured by connecting the convolution layer and the pooling layer, where the second neural network model, in the process of processing the input data, The heatmap, the 2D heatmap of the human torso and the depth interpolated map containing the continuous depth information of the human torso are spliced along the dimension of the image channel, and then the spliced data is subjected to feature analysis via a convolutional layer. We can perform fusion and realize prediction through pooling layers, and finally output the 3D position coordinates of each continuous point in the human torso. Therefore, the second neural network model includes a two-dimensional heat map of the human body keypoints, a two-dimensional heat map of the human torso, a depth interpolated map containing continuous depth information of the human torso, and a three-dimensional position of each continuous point of the human torso. Coordinates can be obtained through joint training through these four data.

The image processing method of the embodiment of the present invention is not limited to being applied to the process of human body pose detection described above, but can be applied to pose detection of any target object. Note that it is not limiting.

The above-described method embodiments described in the present invention can be combined with each other to produce combined embodiments without violating the principle and logic, and due to space limitations, the present invention does not repeat Please understand. Persons skilled in the art can understand that in the above method of specific embodiments, the execution order of each step should be based on its functions and possibilities.

In addition, the present invention further provides a posture detection device, an electronic device, a computer-readable storage medium, and a program, all of which can implement any posture detection method provided in the present invention and correspond to The corresponding descriptions with reference to the technical solution and the description and the method part will not be repeated.

FIG. 6 shows a block diagram of an attitude detection device according to one embodiment of the present invention. The attitude detection device can be a terminal device, a server or other processing device, or the like. Here, the terminal equipment includes user equipment (UE), mobile equipment, user terminal, terminal, mobile phone, cordless phone, personal digital assistant (PDA), handheld equipment, computing equipment, in-vehicle equipment, It can be a wearable device or the like.
In some possible implementations, the pose detection apparatus can be implemented by a processor through calling computer readable instructions stored in memory.

As shown in FIG. 6, the attitude detection device 30
a target image acquisition unit 31 configured to acquire a target image;
an information acquisition unit 32 configured to acquire continuous depth information and position information of a target object in the target image according to the target image;
a pose determiner 33 configured to determine the pose of the target object according to the continuous depth information and the position information.

In one possible implementation, the information acquisition unit passes the target image through a first neural network model to acquire continuous depth information and position information of the target object in the target image, wherein the first neural network model receives the first obtained by training via training data and second training data, wherein the first training data is a training image including a training object and the second training data is a continuous depth of the training object information, and the location of the training target.

In one possible implementation, the pose determiner obtains the pose of the target subject through continuous depth information and position information through a second neural network model, which uses the second training data and the second configured to train via three training data, wherein the second training data includes continuous depth information of the training object and position information of the training object, and the third training data includes pose of the training object .

In one possible implementation, the second training data is generated according to third training data, the third training data includes a pose of a training object, and generating the second training data according to the third training data includes: obtaining discrete depth information for the training object and position information for the training object according to the pose of the training object in the third training data; and processing at least a portion of the discrete depth information to obtain continuous depth information for the training object. obtaining and generating second training data according to the continuous depth information of the training object and the position information of the training object.

In one possible implementation, processing at least a portion of the discrete depth information to obtain continuous depth information to be trained includes obtaining at least one connection corresponding to at least a portion of the discrete depth information. , determining continuous depth sub-information of at least one connection according to the discrete depth information corresponding to the connection; and statisticating the at least one continuous depth sub-information to obtain continuous depth information to be trained. include.

In one possible implementation, determining continuous depth sub-information of at least one connection according to the discrete depth information corresponding to the connection includes: obtaining first continuous depth sub-information of at least one point; determining a connection range corresponding to at least one connection; determining at least one within the connection range corresponding to the connection according to the first continuous depth sub-information; determining second continuous depth sub-information for one point; and obtaining continuous depth sub-information corresponding to a connection according to the first continuous depth sub-information and/or the second continuous depth sub-information, and at least one said connection. and obtaining continuous depth sub-information of .

In one possible implementation, determining a second continuous depth sub-information of at least one point in the connection range corresponding to the connection according to the first continuous depth sub-information is performed when the connection range is a discrete depth corresponding to the connection. using the discrete depth information corresponding to the connection as second continuous depth sub-information for at least one point within the connection range, if within the preset range of information; obtaining second continuous depth sub-information of at least one point within the connected range according to the first continuous depth sub-information that is closest to the point within the connected range within the connection if outside the preset range of information; including.

FIG. 7 shows a block diagram of a video processing device according to one embodiment of the present invention. The video processing device may be a terminal device, a server or other processing device, or the like. Here, the terminal equipment includes user equipment (UE), mobile equipment, user terminal, terminal, mobile phone, cordless phone, personal digital assistant (PDA), handheld equipment, computing equipment, in-vehicle equipment, It can be a wearable device or the like.
In some possible embodiments, the video processing apparatus can be implemented via a scheme in which a processor invokes computer readable instructions stored in memory.

As shown in FIG. 7, the video processing device 40:
an image acquisition unit 41 configured to perform image acquisition for the current scenario to obtain an acquired video;
a selection unit 42 configured to select at least two frames of a target image containing the target object from the acquired video;
Performing pose detection for a target object in at least two frames of target images, via a pose detection method according to any one of the above disclosed embodiments; a posture acquisition unit 43 configured to determine one posture.

In one possible implementation, the video processing unit 40 further obtains successive poses of the target object according to the at least two poses of the target object and the time of the frames in the acquired video; configured to track a subject;

An embodiment of the present invention further proposes a computer-readable storage medium on which computer program instructions are stored, said computer program instructions, when executed by a processor, realizing the above method. A computer-readable storage medium may be a non-volatile computer-readable storage medium.

An embodiment of the present invention further proposes an electronic device comprising a processor and a memory configured to store processor-executable instructions, wherein said processor comprises instructions stored by said memory to perform the above method.

An embodiment of the invention further provides a computer program product comprising computer readable code, and when the computer readable code is executed in a device, a processor in the device implements an image processing method according to any of the above embodiments. execute instructions for

Embodiments of the invention further provide another computer program product for use in storing computer readable instructions which, when executed, cause a computer to perform image processing according to any of the embodiments above. Execute the operation of the method.

An electronic device may be provided as a terminal, server, or other form of device.

In embodiments and other embodiments of the present invention, a "portion" may be a partial circuit, partial processor, partial program or software, etc., and of course may be a unit, modular or non-modular. may

FIG. 8 shows a block diagram of an electronic device 800 according to an embodiment of the invention. For example, electronic device 800 can be a terminal such as a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical equipment, fitness equipment, personal digital assistant, or the like.

Referring to FIG. 8, electronic device 800 includes processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814, and communication component 816. may include one or more components of

The processing component 802 generally controls the general operation of the electronic device 800, such as operations related to display, phone calls, data communications, camera operation and recording operations. The processing component 802 can comprise one or more processors 820 to execute instructions to complete all or part of the method steps. Additionally, processing component 802 can comprise one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 can comprise a multimedia module to facilitate interaction between multimedia component 808 and processing component 802 .

Memory 804 is configured to store various types of data to support operations on device 800 . Examples of these data include instructions for any application or method, contact data, phonebook data, messages, photos, videos, etc., running on electronic device 800 . Memory 804 can be static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM). , read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk, or any combination thereof.

Power component 806 provides power to various components of electronic device 800 . Power components 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and allocating power for electronic device 800 .

Multimedia component 808 includes a screen that provides an output interface between electronic device 800 and a user. In some examples, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen for receiving input signals from the user. A touch panel includes one or more touch sensors for detecting touches, swipes, and gestures on the touch panel. The touch sensor can not only sense the boundaries of a touch or swipe operation, but also the duration and pressure associated with the touch or swipe action. In some examples, multimedia component 808 includes one front camera and/or one rear camera. When electronic device 800 is in an operational mode, such as photography mode or video mode, the front and/or rear cameras may receive external multimedia data. Each front and rear camera may be a fixed optical lens system or may have a focal length and optical zoom capability.

Audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 comprises one microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in operational modes such as call mode, recording mode and voice recognition mode. be done. The received audio signal can be stored in memory 804 or transmitted via communication component 816 . In some examples, audio component 810 also includes a speaker configured to output an audio signal.

I/O interface 812 provides an interface between processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, and the like. These buttons may include, but are not limited to, home button, volume button, start button, lock button.

Sensor component 814 comprises one or more sensors for providing status assessments of aspects to electronic device 800 . For example, the sensor component 814 can detect the on/off state of the electronic device 800, the relative positions of components such as the display and keypad of the electronic device 800, and the sensor component 814 can sense the electronic device 800 or electronic Changes in the position of one component of device 800, presence or absence of contact between the user and electronic device 800, orientation or acceleration/deceleration of electronic device 800, and changes in temperature of electronic device 800 can also be detected. Sensor component 814 can comprise a proximity sensor and is configured to detect the presence of nearby objects without physical contact. Sensor component 814 may also comprise an optical sensor, such as a CMOS or CCD image sensor, used for imaging applications. In some examples, the sensor component 814 may further include an acceleration sensor, gyroscope sensor, magnetic sensor, pressure sensor, or temperature sensor.

Communications component 816 is configured to facilitate wired or wireless communications between electronic device 800 and other devices. Electronic device 800 may access wireless networks based on communication standards such as WiFi, 2G or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component 816 further comprises a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, electronic device 800 includes one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processors (DSPDs), It can be implemented by a programmable logic device (PLD), field programmable gate array (FPGA), controller, microcontroller, microprocessor, or other electronic device.

An exemplary embodiment further provides a non-volatile computer-readable storage medium, such as memory 804, containing computer program instructions, which are executed by processor 820 of electronic device 800 to perform the above method. can be completed.

FIG. 9 shows a block diagram of an electronic device 1900 according to an embodiment of the invention. For example, electronic device 1900 can be provided as a server. Referring to FIG. 9, electronic device 1900 includes a processing component 1922 and includes one or more processors and memory resources represented by memory 1932 to provide instructions, such as application programs, to be executed by processing component 1922. used to remember. An application program stored in memory 1932 may comprise one or more modules, each module corresponding to a set of instructions. Additionally, the processing component 1922 is configured to execute instructions to perform the methods described above.

The electronic device 1900 further includes one power component 1926 configured to perform power management of the electronic device 1900 and one wired or wireless network interface configured to connect the electronic device 1900 to a network. 1950 and one input/output (I/O) interface 1958 . Electronic device 1900 can be operated based on an operating system such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like, stored in memory 1932 .

An exemplary embodiment also provides a volatile computer-readable storage medium, such as memory 1932, containing computer program instructions, which are executed by processing component 1922 of electronic device 1900 to perform the method described above. can be completed.

The invention can be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium loaded with computer-readable program instructions used to implement various aspects of the present invention on a processor.

A computer-readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction-executing device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory) ), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy discs, mechanical encoding devices such as punch cards on which instructions are stored or Including groove protruding structures, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, includes radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or wires. It is not interpreted as a transient signal per se, such as an electrical signal transmitted through a

The computer readable program instructions described herein can be downloaded from a computer readable storage medium to various computing/processing devices or distributed over networks such as the Internet, local area networks, wide area networks and/or wireless networks. can be downloaded to an external computer or external storage device. A network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards said computer-readable program instructions for storage on a computer-readable storage medium in each computing/processing device. be.

Computer program instructions used to perform the operations of the present disclosure may be component instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, or one or more of can be source code or object code written in any combination of programming languages, including object-oriented programming languages such as Smalltalk, C++, and conventional programming languages such as the "C" language or similar programming languages. including procedural programming languages. Computer-readable program instructions can be executed entirely on a user's computer, partially on a user's computer, as a stand-alone package, partially on a user's computer, partially on a remote computer, or entirely Can run on a remote computer or server. In the remote computer scenario, the remote computer can access the user's computer or can access the remote computer over any type of network, including a local area network (LAN) or a wide area network (WAN). over the Internet using an Internet Service Provider). In some embodiments, electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized through the use of status information in computer readable program instructions. and the electronic circuitry can execute computer readable program instructions to implement various aspects of the present disclosure.

Various aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, a proprietary computer, or other programmable data processing apparatus, thereby creating a device, by which these instructions may be read by the computer or other programmable data processing apparatus. When executed by the processor of the processing unit, it implements the specified functions/actions of one or more blocks in the flowchart illustrations and/or block diagrams. These computer readable program instructions may be stored on a computer readable storage medium, and these instructions cause the computer, programmable data processing device, and/or other apparatus to operate in a particular manner; A computer-readable medium on which is stored includes various aspects of instructions that implement the specified functions/actions of one or more blocks in the flowcharts and/or block diagrams.

The computer readable program instructions can also be loaded into a computer, other programmable data processing device, or other equipment, and cause the computer, other programmable data processing device, or other equipment to perform a series of operational steps to cause the computer to To generate a process of implementation that causes instructions to be executed by a computer, other programmable data processing device, or other machine to perform the specified functions/actions of one or more blocks in the flowchart illustrations and/or block diagrams. Realize

The process diagrams and block diagrams in the figures illustrate possible architectures, functionality, and operation of systems, methods and computer program products according to several embodiments of the present invention. In this regard, each block in a flowchart or block diagram can represent a portion of a module, program segment, or instruction, wherein said module, program segment, or portion of instruction is one or more Contains executable instructions used to implement specified logic functions. In some alternative implementations, the marked functions of the blocks may occur out of the order marked in the figures. For example, depending on the functionality involved, two consecutive blocks can actually be executed essentially in parallel, sometimes in reverse order. Each block in the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, may be implemented by, or may be, a dedicated hardware-based system that performs the specified functions or actions. Note that it can also be implemented using a combination of software and computer instructions.

The computer program product can be specifically realized through hardware, software, or a combination thereof. In one exemplary embodiment, said computer program product is specifically embodied as a computer storage medium, and in another exemplary embodiment, said computer program product is specifically embodied as a software development kit ( It is embodied as a software product such as SDK (Software Development Kit).

Although embodiments of the present invention have been described above, the above description is illustrative only and is not exhaustive and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of each described embodiment. The choice of terminology used herein is such that it best describes the principle, practical application, or technical improvement in the market of each embodiment, or allows those of ordinary skill in the art to understand each implementation disclosed herein. The intention is to make the example understandable.

In an implementation of the present invention, through obtaining the target image and the continuous depth information and position information of the target object within the target image, the pose of the target object is determined according to the continuous depth information and the position information. Through the above process, the continuous depth information of the target object can be used to predict the pose of the target object. It can accurately detect and improve the accuracy and effectiveness of attitude detection.

Claims (15)

A posture detection method comprising:
obtaining a target image;
obtaining continuous depth and position information of a target object within the target image according to the target image;
determining a pose of the target object according to the continuous depth information and the position information. Obtaining continuous depth information and location information of a target object within the target image according to the target image;
passing the target image through a first neural network model to obtain continuous depth and position information of a target object in the target image;
said first neural network model is obtained by training via first training data and second training data;
the first training data is a training image including a training target;
the second training data includes continuous depth information of the training object and location information of the training object;
The posture detection method according to claim 1. Determining a pose of the target object according to the continuous depth information and the position information includes:
obtaining a pose of the target object from the continuous depth information and the position information through a second neural network model;
training the second neural network model via second training data and third training data;
the second training data includes continuous depth information for a training target and location information for the training target;
wherein the third training data includes the posture of the training target;
The attitude detection method according to claim 1 or 2. the second training data is generated according to third training data, the third training data including a posture of the training object;
generating the second training data according to the third training data;
obtaining discrete depth information of the training object and position information of the training object according to the pose of the training object in the third training data;
processing at least a portion of the discrete depth information to obtain continuous depth information for the training target;
generating the second training data according to the continuous depth information of the training object and the location information of the training object;
The attitude detection method according to claim 2 or 3. processing the at least a portion of the discrete depth information to obtain continuous depth information for the training object;
obtaining at least one connection corresponding to at least a portion of the discrete depth information;
determining continuous depth sub-information for at least one of said connections according to discrete depth information corresponding to said connections;
statistics of the at least one continuous depth sub-information to obtain continuous depth information for the training object;
The posture detection method according to claim 4. Determining continuous depth sub-information for at least one connection according to discrete depth information corresponding to the connection;
obtaining first continuous depth sub-information of at least one point on said connection via linear interpolation according to discrete depth information corresponding to said connection;
determining a connection range corresponding to at least one of said connections;
determining second continuous depth sub-information for at least one point within a connection range corresponding to said connection according to said first continuous depth sub-information;
obtaining continuous depth sub-information corresponding to the connections according to the first continuous depth sub-information and/or the second continuous depth sub-information to obtain continuous depth sub-information for the at least one connection; including,
The posture detection method according to claim 5. Determining second continuous depth sub-information for at least one point within a connection range corresponding to said connection according to said first continuous depth sub-information;
If the connected range is within a preset range of discrete depth information corresponding to the connected, then using the discrete depth information corresponding to the connected as second continuous depth sub-information for at least one point within the connected range. and
If the connection range is outside the preset range of discrete depth information corresponding to the connection, at least one within the connection range according to a first continuous depth sub-information that is closest to a point within the connection range within the connection. obtaining second consecutive depth sub-information for one point;
The attitude detection method according to claim 6. A video processing method comprising:
performing image acquisition for the current scenario to obtain an acquired video;
selecting at least two frames of a target image containing a target object from the acquired video;
Performing pose detection on the target object in the target images of at least two frames via a pose detection method according to any one of claims 1 to 7, wherein the target object in the acquired video determining at least two poses of . The video processing method comprises:
obtaining successive poses of the target object according to at least two poses of the target object and time of frames in the acquired video;
tracking the target object according to a continuous pose of the target object;
A video processing method according to claim 8. A posture detection device,
a target image acquisition unit configured to acquire a target image;
an information acquisition unit configured to acquire continuous depth information and location information of a target object within the target image according to the target image;
a pose determination unit configured to determine a pose of the target object according to the continuous depth information and the position information. A video processing device,
an image acquisition unit configured to perform image acquisition for the current scenario and obtain an acquired video;
a selection unit configured to select at least two frames of a target image containing a target object from the acquired video;
Performing pose detection on the target object in the target images of at least two frames via a pose detection method according to any one of claims 1 to 7, wherein the target object in the acquired video a pose acquisition unit configured to determine at least two poses of the video processing apparatus. an electronic device,
a processor;
a memory configured to store processor-executable instructions;
10. The electronic device, wherein the processor is configured to invoke instructions stored by the memory to perform a posture detection method according to any one of claims 1-9. A computer readable storage medium storing computer program instructions, comprising:
10. The computer readable storage medium implementing the pose detection method of any one of claims 1 to 9 when the computer program instructions are executed by a processor. A computer program product, comprising a computer program or instructions, said computer program or instructions, when executed on a computer, causing said computer to perform an attitude detection method according to any one of claims 1 to 9. , said computer program product. 10. A posture according to any one of claims 1 to 9, comprising a computer program comprising computer readable code, said computer readable code being executed in an electronic device and being executed by a processor within said electronic device. Said computer program implementing the detection method.

Images