US20210074005A1

US20210074005A1 - Image processing method and apparatus, image device, and storage medium

Info

Publication number: US20210074005A1
Application number: US17/102,364
Authority: US
Inventors: Fubao XIE; Wentao Liu
Original assignee: Beijing Sensetime Technology Development Co Ltd
Current assignee: Beijing Sensetime Technology Development Co Ltd
Priority date: 2019-01-18
Filing date: 2020-11-23
Publication date: 2021-03-11
Also published as: SG11202011596WA; KR20210011424A; JP2021525431A; JP7109585B2; SG11202011595QA; JP7061694B2; US11468612B2; WO2020181900A1; US20210035344A1; KR20210011984A; CN111460875A; CN111460872A; US11538207B2; JP2021524113A; US20210074003A1; US11741629B2; US20210074004A1; JP2021518960A; KR20210011425A; KR20210011985A

Abstract

Embodiments of the present disclosure disclose an image processing method and apparatus, an image device, and a storage medium. The image processing method includes: obtaining an image; obtaining features of at least two parts of a target based on the image; determining, according to the features of the at least two parts and a first movement constraint condition of a connection portion, movement information of the connection portion, where the connection portion connects two of the at least two parts; and controlling the movement of the connection portion in a controlled model according to the movement information of the connection portion.

Description

CROSS REFERENCE TO RELATED APPLICATION

The present application is a continuation of International Application No. PCT/CN2020/072549, filed on Jan. 16, 2020, which claims priority to Chinese Patent Application No. 201910049830.6, filed on Jan. 18, 2019, and entitled “IMAGE PROCESSING METHOD AND APPARATUS, IMAGE DEVICE, AND STORAGE MEDIUM”, and to Chinese Patent Application No. 201910363433.6, filed on Apr. 30, 2019, and entitled “IMAGE PROCESSING METHOD AND APPARATUS, IMAGE DEVICE, AND STORAGE MEDIUM”, all of which are incorporated herein by reference in their entirety.

TECHNICAL FIELD

The present disclosure relates to the field of information technologies, and in particular, to an image processing method and apparatus, an image device, and a storage medium.

BACKGROUND

With the development of information technologies, users can perform online teaching and online anchoring through video recording, and somatosensory games and the like become possible. However, in some cases, for example, in somatosensory games, users are required to wear special somatosensory devices to detect activities of their own limbs and the like, so as to control game characters. However, when performing online teaching or online anchoring, the face, limbs or the like of a user are completely exposed online; on the one hand, this may involve a user privacy issue, and on the other hand, this may also involve an information security issue. In order to solve such as privacy or security issue, a face image can be covered by means of mosaic, etc., but this will affect the video effect.

SUMMARY

In view of this, embodiments of the present disclosure are expected to provide an image processing method and apparatus, an image device, and a storage medium.
In a first aspect, the present disclosure provides an image processing method, including:
obtaining an image; obtaining features of at least two parts of a target based on the image; determining, according to the features of the at least two parts and a first movement constraint condition of a connection portion, movement information of the connection portion, where the connection portion connects two of the at least two parts; and controlling the movement of the connection portion in a controlled model according to the movement information of the connection portion.
Based on the foregoing solutions, controlling the movement of the connection portion in the controlled model according to the movement information of the connection portion includes: determining, according to the type of the connection portion, a control mode of controlling the connection portion; and controlling the movement of the connection portion in the controlled model according to the control mode and the movement information of the connection portion.
Based on the foregoing solutions, determining, according to the type of the connection portion, the control mode of controlling the connection portion includes: in the case that the connection portion is a first-type connection portion, the control mode is a first-type control mode, where the first-type control mode is used for directly controlling the movement of a connection portion in the controlled model corresponding to the first-type connection portion.
Based on the foregoing solutions, determining, according to the type of the connection portion, the control mode of controlling the connection portion includes: in the case that the connection portion is a second-type connection portion, the control mode is a second-type control mode, where the second-type control mode is used for indirectly controlling the movement of a connection portion in the controlled model corresponding to the second-type connection portion, and the indirect control is achieved by controlling a part in the controlled model corresponding to the part other than the second-type connection portion.
Based on the foregoing solutions, controlling the movement of the connection portion in the controlled model according to the control mode and the movement information of the connection portion includes: in the case that the control mode is the second-type control mode, decomposing the movement information of the connection portion to obtain first-type rotation information of the connection portion that the connection portion rotates under the traction of a traction portion; adjusting movement information of the traction portion according to the first-type rotation information; and controlling the movement of the traction portion in the controlled model according to the adjusted movement information of the traction portion, to indirectly control the movement of the connection portion.
Based on the foregoing solutions, the method further includes: decomposing the movement information of the connection portion, to obtain second-type rotation information of the second-type connection portion rotating with respect to the traction portion; and controlling, in the controlled model, the rotation of the connection portion with respect to the traction portion by using the second-type rotation information.
Based on the foregoing solutions, the second-type connection portion includes: a wrist; and an ankle.
Based on the foregoing solutions, in the case that the second-type connection portion is a wrist, the traction portion corresponding to the wrist includes: an upper arm and/or a forearm; and in the case that the second-type connection portion is an ankle, the traction portion corresponding to the ankle includes: a thigh and/or a calf.
Based on the foregoing solutions, the first-type connection portion includes a neck connecting a head and a trunk.
Based on the foregoing solutions, determining, according to the features of the at least two parts and the first movement constraint condition of the connection portion, the movement information of the connection portion includes: determining respective orientation information of the two parts according to the features of the two parts connected by the connection portion; determining candidate orientation information of the connection portion according to the respective orientation information of the two parts; and determining the movement information of the connection portion according to the candidate orientation information and the first movement constraint condition.
Based on the foregoing solutions, determining the candidate orientation information of the connection portion according to the respective orientation information of the two parts includes: determining first candidate orientation information and second candidate orientation information of the connection portion according to the respective orientation information of the two parts.
Based on the foregoing solutions, determining the movement information of the connection portion according to the candidate orientation information and the first movement constraint condition includes: selecting target orientation information within an orientation change constraint range from the first candidate orientation information and the second candidate orientation information; and determining the movement information of the connection portion according to the target orientation information.
Based on the foregoing solutions, determining the respective orientation information of the two parts according to the features of the two parts includes: obtaining, for each of the two parts, a first key point and a second key point; obtaining a first reference point for each of the two parts, where the first reference point refers to a first predetermined key point in the target; generating a first vector based on the first key point and the first reference point; generating a second vector based on the second key point and the first reference point; and determining the orientation information for each of the two parts based on the first vector and the second vector.
Based on the foregoing solutions, determining the orientation information for each of the two parts based on the first vector and the second vector includes: performing, for each part, cross product on the two vectors of the part to obtain a normal vector of a plane where the part is located; and taking the normal vector as the orientation information of the part.
Based on the foregoing solutions, the method further includes: determining movement information of the at least two parts based on the features. Determining, according to the features of the at least two parts and the first movement constraint condition of the connection portion, the movement information of the connection portion includes: determining the movement information of the connection portion according to the movement information of the at least two parts.
Based on the foregoing solutions, determining the movement information of the connection portion according to the movement information of the at least two parts includes: obtaining a third 3-Dimensional (3D) coordinate of the connection portion with respect to a second reference point, where the second reference point refers to a second predetermined key point in the at least two parts; and obtaining absolute rotation information of the connection portion according to the third 3D coordinate; and controlling the movement of the connection portion in the controlled model according to the movement information of the connection portion includes: controlling the movement of the connection portion in the controlled model based on the absolute rotation information.
Based on the foregoing solutions, controlling the movement of the connection portion in the controlled model based on the absolute rotation information further includes: decomposing the absolute rotation information according to a traction hierarchy relationship among multiple connection portions in the target to obtain relative rotation information; and controlling the movement of the connection portion in the controlled model according to the movement information of the connection portion includes: controlling the movement of the connection portion in the controlled model based on the relative rotation information.
Based on the foregoing solutions, controlling the movement of the connection portion in the controlled model based on the absolute rotation information further includes: correcting the relative rotation information according to a second constraint condition; and controlling the movement of the connection portion in the controlled model based on the relative rotation information includes: controlling the movement of the connection portion in the controlled model based on the corrected relative rotation information.
Based on the foregoing solutions, the second constraint condition includes: a rotatable angle of the connection portion.
In a second aspect, the present disclosure provides an image processing apparatus, including:
a first obtaining module, configured to obtain an image; a second obtaining module, configured to obtain features of at least two parts of a target based on the image; a first determination module, configured to determine, according to the features of the at least two parts and a first movement constraint condition of a connection portion, movement information of the connection portion, where the connection portion connects two of the at least two parts; and a control module, configured to control the movement of the connection portion in a controlled model according to the movement information of the connection portion.
In a third aspect, the present disclosure provides an image device, including: a memory; and a processor, connected to the memory, and configured to execute computer-executable instructions on the memory to implement the image processing method according to any one of the foregoing items.
In a fourth aspect, the present disclosure provides a non-volatile computer storage medium, where the computer storage medium stores computer-executable instructions, and after the computer-executable instructions are executed by a processor, the image processing method according to any one of the foregoing items can be implemented.
In the technical solutions provided by the embodiments of the present disclosure, the movement information of the connection portion is obtained according to the features of the at least two parts and the first movement constraint condition of the connection portion. In this way, for the connection portion whose movement information is inconvenient to directly obtain, the movement information of the connection portion can also be obtained precisely, and thus the movement of the connection portion corresponding to the controlled model can be controlled. When the controlled model is used to simulate the movement of a user for live video streaming, the movement of the connection portion in the controlled model can be controlled more precisely, so that the controlled model can precisely simulate the movements of acquisition objects such as users, thereby protecting the privacy of the user during the live video streaming.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic flowchart of an image processing method provided by the embodiments of the present disclosure.

FIG. 1B is a schematic flowchart of an image processing method provided by another embodiment of the present disclosure.

FIG. 2 is a schematic flowchart of an image processing method provided by yet another embodiment of the present disclosure.

FIG. 3A to FIG. 3C are schematic diagrams of changes in hand movement of a collected user simulated by a controlled model provided by the embodiments of the present disclosure.

FIG. 4A to FIG. 4C are schematic diagrams of changes in trunk movement of an acquired user simulated by a controlled model provided by the embodiments of the present disclosure.

FIG. 5A to FIG. 5C are schematic diagrams of foot movement of an acquired user simulated by a controlled model provided by the embodiments of the present disclosure.

FIG. 6 is a schematic structural diagram of an image processing apparatus provided by the embodiments of the present disclosure.

FIG. 7A is a schematic diagram of key points of a skeleton provided by the embodiments of the present disclosure.

FIG. 7B is a schematic diagram of key points of a skeleton provided by the embodiments of the present disclosure.

FIG. 8 is a schematic diagram of a skeleton provided by the embodiments of the present disclosure.

FIG. 9 is a schematic diagram of a local coordinate system of different bones of a human body provided by the embodiments of the present disclosure.

FIG. 10 is a schematic structural diagram of an image device provided by the embodiments of the present disclosure.

DETAILED DESCRIPTION

The technical solutions of the present disclosure are further described in detail below with reference to the accompanying drawings and specific embodiments of the specification.
As shown in FIG. 1A, the embodiments provide an image processing method, including the following steps.
At step S110, an image is obtained.
At step S120, features of at least two parts of a target are obtained based on the image.
At step S130, according to the features of the at least two parts and a first movement constraint condition of a connection portion, movement information of the connection portion is determined, where the connection portion connects two of the at least two parts.
At step S140, the movement of the connection portion in a controlled model is controlled according to the movement information of the connection portion.
The image processing method provided by the embodiments can drive the movement of the controlled model through image processing.
The image processing method provided by the embodiments can be applied to an image device. The image device may be any of electronic devices capable of performing image processing, for example, electronic devices for performing image acquisition, image display, and image processing. The image device includes, but is not limited to, various terminal devices, such as a mobile terminal and/or a fixed terminal, and may also include various servers capable of providing image services. The mobile terminal includes portable devices easy to carry by a user, such as a mobile phone or a tablet computer, and may also include a device worn by the user, such as a smart bracelet, a smart watch, or smart glasses. The fixed terminal includes a fixed desktop computer, etc.
In the embodiments, the image obtained in step S110 may be a two-dimensional (2D) image or a three-dimensional (3D) image. The 2D image may include: an image acquired by a monocular or multiocular camera, such as a red green blue (RGB) image. The approach of obtaining the image may include: acquiring the image by using a camera of the image device; and/or receiving an image from an external device; and/or reading the image from a local database or a local memory.
The 3D image may be a 3D image obtained by detecting 2D coordinates from a 2D image, and then using a conversion algorithm from 2D coordinates to 3D coordinates. The 3D image may also be an image acquired using a 3D camera.
In step S110, the obtained image may be one image frame or multiple image frames. For example, when the obtained image is one image frame, the subsequently obtained movement information may reflect the movement of the connection portion in the current image with respect to the corresponding connection portion of an initial coordinate system (also referred to as a camera coordinate system). For another example, when multiple image frames are obtained, the subsequently obtained movement information may reflect the movement of the connection portion in the current image with respect to the corresponding connection portions in the previous image frames, or the subsequently obtained movement information may also reflect the movement of the connection portion in the current image with respect to the corresponding connection portion of the camera coordinate system. The present application does not limit the number of the obtained images
Step S120 may include: obtaining the features of the at least two parts of the target by detecting the image, where the two parts are different parts on the target. The two parts may be continuously distributed on the target, and may also be distributed on the target at an interval.
For example, if the target is a person, the at least two parts may include at least two of the following: head, trunk, limbs, upper limbs, lower limits, hands, feet, or the like. The connection portion may be a neck connecting the head and the trunk, a right hip connecting the right leg and the trunk, a wrist connecting the hand and the forearm, or the like. In some other embodiments, the target is not limited to humans, but may also be various movable living bodies or non-living bodies such as animals.
In the embodiments, the features of at least two parts are obtained, and the features may be features characterizing the spatial structure information, position information, or movement states of the two parts in various forms.
The features of the two parts include, but are not limited to, various image features. For example, the image features may include color features and/or optical flow features obtained using an optical flow algorithm. The color features include, but are not limited to, RGB features and/or gray-scale features. In the embodiments, a deep learning model such as a neural network may be used to detect the image, so as to obtain the features.
After the features are obtained, the precise movement information of the connection portion can be obtained based on the features, the connection relationship between the two parts and the connection portion, and the first movement constraint condition that the movement of the connection portion needs to satisfy.
For a case where one connection portion connects three or more parts, the connection portion may be split into a plurality of sub connection portions; then, according to the method provided by the present disclosure, for each sub connection portion, the movement information thereof is calculated respectively, and then the movement information of the sub connection portions is merged to obtain the movement information of the connection portion. For the sake of simplicity, the present disclosure is described below in the case that one connection portion connects two parts in the present disclosure.
The controlled module may be a model corresponding to the target. For example, if the target is a person, the controlled model is a human body model. If the target is an animal, the controlled model may be a corresponding animal body model. If the target is a vehicle, the controlled model may be a vehicle model.
In the embodiments, the controlled model is a model for the category to which the target belongs. The model may be predetermined, and may further have multiple styles. The style of the controlled model may be determined based on a user instruction. The controlled model may include a variety of styles, such as, a simulated real-person style, an anime style, an Internet celebrity style, different temperament styles, and a game style. Different temperament styles may be a literary style or a rock style. Under the game style, the controlled model may be a game character.
For example, in the process of network teaching, some teachers are not willing to expose their faces and bodies, and think that these are privacy. If a video is recorded directly, the face and body, etc. of the teacher would necessarily be exposed. In the embodiments, a movement image of the teacher may be obtained through image acquisition, etc., and then a virtual controlled model may be controlled to move through feature extraction and movement information acquisition. In this way, on the one hand, limb movement teaching is completed by using own limb movement to make the controlled model simulate the movement of the teacher, and on the other hand, the movement of the controlled model is used for teaching, and the face and body of the teacher thus do not need to be directly exposed in the teaching video, thereby protecting the privacy of the teacher.
For another example, in a road surveillance video, if the video of vehicles is directly acquired, once the video is exposed to the network, all vehicle information of some specific users is exposed, but without monitoring, situations where responsibility cannot be determined in the event of a traffic accident may occur. If the method of the embodiments is used, a surveillance video is obtained by using a vehicle model to simulate the real vehicle movement, the license plate information of the vehicles and/or the overall outlines of the vehicles are reserved in the surveillance video, but the brands, models, colors, and condition, etc. of the vehicles can all be hidden, thereby protecting the privacy of the user.
In the technical solutions provided by the embodiments of the present disclosure, the movement information of the connection portion is obtained according to the features of the at least two parts. For the connection portion whose movement information is inconvenient to directly obtain, the movement information of the connection portion can also be obtained precisely, and thus the movement of the connection portion corresponding to the controlled model is controlled. Thus, when the controlled model is used to simulate the movement of the target for live video broadcast, the movement of the connection portion in the controlled model can be controlled precisely, so that the controlled model can precisely simulate the movements of acquisition objects such as users, thereby protecting the privacy of the user during the live video broadcast.
Further, as shown in FIG. 1B, the embodiments further provide an image processing method. On the basis of FIG. 1A, the method further includes steps S150 and S160.
At step S150, movement information of the at least two parts is determined based on the features.
At step S160, according to the movement information of the at least two parts, the movement of the parts of the controlled model is controlled.
After the features of the at least two parts are obtained, the movement information of the at least two parts can be obtained. The movement information represents movement changes and/or expression changes, etc. of the two parts at two adjacent moments.
After the movement information of the at least two parts is obtained, the movement information of the connection portion may be obtained according to the connection relationship between the connection portion and the two parts, and the corresponding first movement constraint condition.
The information form of the movement information of the at least two parts includes, but is not limited to, at least one of the following: the coordinates of the key points corresponding to the parts. The coordinates include, but are not limited to: 2D coordinates and 3D coordinates. The coordinates can represent the changes of the key points corresponding to the parts with respect to a reference position, and thus can represent the movement state of the corresponding parts.
The movement information may be represented in various information forms, such as vectors, arrays, one-dimensional values, and matrices.
In the embodiments, the movement information of the two parts is further obtained. The movement of the parts of the controlled model is controlled by means of the movement information of the two parts, so that each part in the controlled model can simulate the target to move.
In some embodiments, as shown in FIG. 2, on the basis of FIG. 1B, step S120 may include the following steps.
At step S121, a first-type feature of a first-type part of the target is obtained based on the image.
At step S122, a second-type feature of a second-type part of the target is obtained based on the image, where the type of the second-type feature is different from that of the first-type feature.
In the embodiments, the first-type feature and the second-type feature are features also representing the spatial structure information, position information and/or movement states of the corresponding parts, but are different types of features.
Different types of features have different characteristics, and would implement higher precision when applied to the different types of parts. In some embodiments, the first-type feature of the first-type part and the second-type feature of the second-type part are respectively obtained based on the image. The obtaining subjects for obtaining the first-type feature and the second-type feature are different; for example, the features are obtained using different deep learning models or deep learning modules. The obtaining logic of the first-type feature is different from that of the second-type feature.
The first-type part and the second-type part are different types of parts. The different types of parts may be distinguished through the movable amplitudes of the different types of parts, or distinguished through the movement fineness of the different types of parts.
In the embodiments, the first-type part and the second-type part may be two types of parts with a relatively large difference in the maximum amplitudes of movement. For example, the first-type part may be a head. The five sense organs of the head can move, but the movements of the five sense organs of the head are all relatively small; the entire head can also move, for example, nodding or shaking the head, but the movement amplitude is relatively small with respect to the movement amplitude of the limbs or the trunk.
The second-type part may be the upper limbs, lower limbs, or four limbs, and the movement amplitudes of the limbs are all very large. If the movement states of the two types of parts are represented by the same feature, the problems such as a decrease in precision or an increase in the complexity of the algorithm due to fitting the movement amplitude of a certain part may be caused.
Here, according to the characteristics of different types of parts, different types of features are used to obtain movement information. Compared with the related mode of using the same type of features to represent the same type of parts, the information precision of at least one type of part may be increased, thereby improving the precision of the movement information.
In some embodiments, step S121 may include: obtaining expression features of the head based on the image.
In the embodiments, the first-type part is a head. The head includes a face. The expression features include, but are not limited to, at least one of the following: the movement of eyebrows, the movement of mouth, the movement of nose, the movement of eyes, and the movement of cheeks. The movement of eyebrows may include: raising eyebrows and slouching eyebrows. The movement of mouth may include: opening the mouth, closing the mouth, twitching the mouth, pouting, grinning, snarling, etc. The movement of nose may include: nose contraction generated by inhalation into the nose and nose extension movement accompanied with blowing outward. The movement of eyes may include, but is not limited to: the movement of eye sockets and/or the movement of eyeballs. The sizes and/or shapes of the eye sockets would be changed by the movement of the eye sockets, for example, both the shapes and sizes of the eye sockets of squinting, glaring and smiling eyes would be changed. The movement of eyeballs may include: the positions of the eyeballs in the eye sockets, for example, a change in the line of sight of the user may cause the eyeballs to be in different positions in the eye sockets, and the movement of the left and right eyeballs together may reflect different emotional states of the user. For the movement of cheeks, some users may have dimples or pear vortexes when they laugh, and the shape of their cheeks may also change accordingly.
In some embodiments, the movement of the head is not limited to the expression movement, and therefore, the first-type feature is not limited to the expression features and further includes hair movement features such as the hair movement of the head. The first-type feature may further include: the entire movement feature of the head, such as nodding and/or shaking the head.
In some embodiments, step S121 further includes: obtaining an intensity coefficient of the expression feature based on the image.
The intensity coefficient in the embodiments may correspond to an expression amplitude of a facial expression. For example, the face is set with multiple expression bases. One expression base corresponds to one expression action. The intensity coefficient here may be used for representing the intensity of the expression action, for example, the intensity may be the amplitude of the expression action.
In some embodiments, the greater the intensity coefficient, the higher the intensity. For example, the higher the intensity coefficient, the greater the amplitude of the open mouth expression base, and the greater the amplitude of the pout expression base, and so on. For another example, the greater the intensity coefficient, the higher the eyebrow height of the eyebrow-raising expression base.
Through the introduction of the intensity coefficient, the controlled model can not only simulate the current action of the target, but also precisely simulate the intensity of the current expression of the target, thereby implementing the precise migration of the expression. In such way, if the method is applied to a somatosensory game scene, the controlled object is a game character. By using this method, the game character can not only be controlled by the limb movements of the user, but also accurately simulate the expression features of the user. In such a game scene, the simulation of the game scene is improved, and the game experience of the user is improved.
In some embodiments, step S160 may include: controlling an expression change of the head of the controlled model based on the expression features; and controlling the intensity of the expression change of the controlled model based on the intensity coefficient.
In some embodiments, obtaining the expression features of the target based on the image includes: obtaining mesh information of the first-type part based on the image.
In the embodiments, when the target is a person, in step S120, through mesh detection, etc., mesh information representing the expression change of the head is obtained, and the change of the controlled model is controlled based on the mesh information. The mesh information includes, but is not limited to: quadrilateral mesh information and/or triangular patch information. The quadrilateral mesh information indicates information of longitude and latitude lines; the triangular patch information is information of a triangular patch formed by connecting three key points.
For example, the mesh information is formed by a predetermined number of face key points of a face surface; the intersections between the latitude and longitude lines in a mesh represented by the quadrilateral mesh information may be the positions of the face key points; position changes of the intersections of the mesh are the expression changes. Thus, the expression features and intensity coefficient obtained based on the quadrilateral mesh information can accurately control the expression of the face of the controlled model. Similarly, the vertices of the triangular patch corresponding to the triangular patch information include the face key points. The changes in the positions of the key points are the expression changes. The expression features and intensity coefficient obtained based on the triangular patch information can be used for precise control of the expression of the face of the controlled model.
In some embodiments, obtaining the intensity coefficient of the expression feature based on the image includes: obtaining the intensity coefficient representing each subpart in the first-type part based on the image.
For example, the five sense organs of the face, i.e., the eyes, the eyebrows, the nose, the mouth, and the ears, respectively correspond to at least one expression base, and some correspond to multiple expression bases; one expression base corresponds to one type of expression action of one of the five sense organs, and the intensity coefficient just represents the amplitude of the expression action.
In some implementations, step S122 may include: obtaining the image, position information of the key points of the second-type part of the target based on the image.
The position information may be represented by the position information of the key points of the target. The key points may include: bracket key points and outer contour key points. If a person is taken as an example, the bracket key points may include the skeleton key points of the human body, and the contour key points may be the key points of the outer contour of the body surface of the human body.
The position information may be represented by coordinates, for example, be represented with 2D coordinates and/or 3D coordinates of a predetermined coordinate system. The predetermined coordinate system includes, but is not limited to: an image coordinate system where the image is located. The position information may be the coordinates of the key points, and are obviously different from the above-mentioned mesh information. Because the second-type part is different from the first-type part, the movement change of the second-type part can be more precisely represented with the position information.
In some embodiments, step S150 may include: determining the movement information of at least two parts of the second-type part based on the position information.
If a person is taken as an example of the target, the second-type part includes, but is not limited to: the trunk and/or four limbs; the trunk and/or upper limbs, or the trunk and/or lower limbs.
Furthermore, step S122 may specifically include: obtaining first coordinates of the bracket key points of the second-type part of the target based on the image; and obtaining second coordinates based on the first coordinates.
Both the first coordinates and the second coordinates are coordinates representing the bracket key points. If a person or an animal is taken as an example of the target, the bracket key points here are skeleton key points.
The first coordinates and the second coordinates may be different types of coordinates. For example, the first coordinates are 2D coordinates in a 2D coordinate system, and the second coordinates are 3D coordinates in a 3D coordinate system. The first coordinates and the second coordinates may also be the same type of coordinates. For example, the second coordinates are coordinates obtained after performing correction on the first coordinates. In this case, the first coordinates and the second coordinates are the same type of coordinates. For example, both the first coordinates and the second coordinates are 3D coordinates or 2D coordinates.
In some embodiments, obtaining the first coordinates of the bracket key points of the second-type part of the target based on the image includes: obtaining first 2D coordinates of the bracket key points of the second-type part based on a 2D image. Obtaining the second coordinates based on the first coordinates includes: obtaining first 3D coordinates corresponding to the first 2D coordinates based on the first 2D coordinates and a transforming relationship from 2D coordinates to 3D coordinates.
In some embodiments, obtaining the first coordinates of the bracket key points of the second-type part of the target based on the image includes: obtaining second 3D coordinates of the bracket key points of the second-type part of the target based on a 3D image. Obtaining the second coordinates based on the first coordinates includes: obtaining third 2D coordinates based on the second 3D coordinates.
For example, a 3D image is directly obtained in step S110, and the 3D image includes: a 2D image and a depth image corresponding to the 2D image. The 2D image provides coordinate values of the bracket key points in an xoy plane, and depth values in the depth image provide the coordinates of the bracket key points on a z-axis. The z-axis is perpendicular to the xoy plane.
In some embodiments, obtaining the third 2D coordinates based on the second 3D coordinates includes: adjusting the 3D coordinates of the bracket key points corresponding to a shaded portion of the second-type part in the 3D image based on the second 3D coordinates so as to obtain the third 2D coordinates.
In the embodiments, a 3D model is used to first extract the second 3D coordinates from the 3D image, and then the shading of different parts in the target is used and taken into account. Correct third 2D coordinates of different parts of the target in a 3D space can be obtained through correction, thereby ensuring the subsequent control precision of the controlled model.
In some embodiments, step S150 may include: determining a quaternion of the second-type part based on the position information.
In some embodiments, the movement information of the at least two parts is not limited to being represented by the quaternion, but can also be represented by coordinate values in different coordinate systems, such as the coordinate values in the Euler coordinate system or Lagrange coordinate system. The quaternion may be used to precisely represent the spatial position and/or rotation in each direction of the second-type part.
In some embodiments, the quaternion is used as the movement information of the at least two parts and/or the movement information of the connection portion; in specific implementation, the movement information is not limited to the quaternion, and may also be indicated with the coordinate values in various coordinate systems with respect to a reference point, for example, the quaternion may be replaced with Euler coordinates or Lagrange coordinates.
In some embodiments, step S120 may include: obtaining first position information of the bracket key points of a first part in the second-type part; and obtaining second position information of the bracket key points of a second part in the second-type part.
The second-type part may at least include two different parts. Thus, the controlled model can simultaneously simulate the movements of at least two parts of the target.
In some embodiments, step S150 may include: determining the movement information of the first part according to the first position information; and determining the movement information of the second part according to the second position information.
In some embodiments, step S160 may include: controlling the movement of a part of the controlled model corresponding to the first part according to the movement information of the first part; and controlling the movement of a part of the controlled model corresponding to the second part according to the movement information of the second part.
In some other embodiments, the first part is the trunk; and the second part is the upper limbs, lower limbs, or four limbs.
In some embodiments, step S140 further includes: determining, according to the type of the connection portion, a control mode of controlling the connection portion; and controlling the movement of the connection portion in the controlled model according to the control mode and the movement information of the connection portion.
The connection portion may connect another two parts. For example, taking a person as an example, the neck, wrist, ankle, or waist is a connection portion for connecting two parts. The movement information of these connection portions may be inconvenient to detect or depend to a certain extent on other adjacent parts. In the embodiments, the control mode would be determined according to the type of the connection portion.
For example, the lateral rotation of the wrist is, for example, rotation performed by taking an extension direction from the upper arm to the hand as the axis, and the lateral rotation of the wrist is caused by the rotation of the upper arm. For another example, the lateral rotation of the ankle is, for example, rotation performed by taking the extension direction of the calf as the axis, and the rotation of the ankle is also directly driven by the calf.
Moreover, the rotation of the connection portion, such as the neck, determines the orientation of the face and the orientation of the trunk.
In some other embodiments, determining, according to the type of the connection portion, the control mode of controlling the connection portion includes: in the case that the connection portion is a first-type connection portion, determining to use a first-type control model, where the first-type control mode is used for directly controlling the movement of a connection portion in the controlled model corresponding to the first-type connection portion.
In the embodiments, the first-type connection portion is driven by its own rotation rather than other parts.
In some other embodiments, the connection portion further includes a second-type connection portion other than the first-type connection portion. The movement of the second-type connection portion here is not limited to itself, but is driven by other parts.
In some other embodiments, determining, according to the type of the connection portion, the control mode of controlling the connection portion includes: in the case that the connection portion is a second-type connection portion, determining to use a second-type control model, where the second-type control mode is used for indirectly controlling the second-type connection portion by controlling a part in the controlled model other than the second-type connection portion.
The part other than the second-type connection portion includes, but is not limited to, a part directly connected to the second-type connection port, or a part indirectly connected to the second-type connection portion. For example, during the lateral rotation of the wrist, the entire upper limb may be moving, and thus the shoulder and the elbow are both rotating. In such a way, the rotation of the wrist may be indirectly controlled by controlling the lateral rotation of the shoulder and/or elbow.
In some embodiments, controlling the movement of the connection portion in the controlled model according to the control mode and the movement information of the connection portion includes: in the case that the control mode is the second-type control mode, decomposing the movement information of the connection portion to obtain first-type rotation information of the connection portion that the connection portion rotates under the traction of a traction portion; adjusting movement information of the traction portion according to the first-type rotation information; and controlling the movement of the traction portion in the controlled model according to the adjusted movement information of the traction portion, to indirectly control the movement of the connection portion.
In the embodiments, the traction portion is a part directly connected to the second-type connection portion. Taking the wrist being the second-type connection portion as an example, the traction portion is the elbow above the wrist or even an arm. Taking the ankle being the second-type connection portion as an example, the traction portion is the knee above the ankle or even the root of the thigh.
For example, the lateral rotation of the wrist along a linear direction from the shoulder and the elbow to the wrist may be a rotation driven by the shoulder or elbow. However, during the detection of the movement information, the lateral rotation is caused by the movement of the wrist, so the lateral rotation information of the wrist should be essentially assigned to the elbow or shoulder. The movement information of the elbow or shoulder is adjusted through this transfer assignment, and the adjusted movement information is used to control the movement of the elbow or shoulder in the controlled model. Thus, with reference to the effect of the image, the lateral rotation corresponding to the elbow or shoulder would be reflected by the wrist of the controlled model, thereby implementing the precise simulation of the movement of the target by the controlled model.
In some embodiments, the method further includes: decomposing the movement information of the connection portion, to obtain second-type rotation information of the second-type connection portion rotating with respect to the traction portion; and controlling, in the controlled model, the rotation of the connection portion with respect to the traction portion by using the second-type rotation information.
The first-type rotation information is information obtained directly according to the features of the image by an information model for extracting the rotation information, and the second-type rotation information is rotation information obtained by adjusting the first-type rotation information. In the embodiments, first, the movement information of the second-type connection portion with respect to the predetermined posture can be known through the features of the second-type connection portion, for example, 2D coordinates or 3D coordinates. The movement information of the connection portion includes, but is not limited to, the rotation information. In some embodiments, the movement information of the connection portion further includes: translation information.
In some embodiments, the second-type connection portion includes: a wrist; and an ankle.
In some other embodiments, if the second-type connection portion is a wrist, the traction portion corresponding to the wrist includes: an upper arm and/or a forearm; and if the second-type connection portion is an ankle, the traction portion corresponding to the ankle includes: a thigh and/or a calf.
In some embodiments, the first-type connection portion includes a neck connecting the head and the trunk.
In some other embodiments, determining, according to the features of the at least two parts and the first movement constraint condition of the connection portion, the movement information of the connection portion includes: determining orientation information of the two parts according to the features of the two parts connected by the connection portion; determining candidate orientation information of the connection portion according to the orientation information of the two parts; and determining the movement information of the connection portion according to the candidate orientation information and the first movement constraint condition.
In some embodiments, determining the candidate orientation information of the connection portion according to the orientation information of the two parts includes: determining first candidate orientation information and second candidate orientation information of the connection portion according to the orientation information of the two parts.
Two included angles may be formed between the orientation information of the two parts. In the embodiments, the included angle satisfying the first movement constraint condition is taken as the movement information of the connection port.
For example, two included angles are formed between the orientation of the face and the orientation of the trunk, and the sum of the two included angles is 180 degrees. The two included angles are assumed to be a first included angle and a second included angle respectively. Moreover, the first movement constraint condition for the neck connecting the face and the trunk is between −90 degrees and 90 degrees, thus angles exceeding 90 degrees are excluded according to the first movement constraint condition. In this way, the abnormality of rotation angles clockwisely or counterclockwisely exceeding 90 degrees, for example, 120 degrees and 180 degrees, during the simulation of the movement of the target by the controlled model can be reduced. If such a situation where the limit value of the first constraint condition is exceeded, the limit value corresponding o the first constraint condition is substituted for the abnormal value. If the first movement constraint condition is between −90 degrees and 90 degrees, the first movement constraint condition corresponds to two limit angles: one is −90 degrees and the other is 90 degrees.
When the rotation angle exceeds the range of −90 degrees to 90 degrees, the detected rotation angle is modified as the maximum angle defined by the first movement constraint condition, i.e., the limit value. For example, if a rotation angle exceeding 90 degrees is detected, the detected rotation angle is modified as a limit angle closer to the detected rotation angle, for example, 90 degrees.
In some embodiments, determining the movement information of the connection portion according to the candidate orientation information and the first movement constraint condition includes: selecting target orientation information within an orientation change constraint range from the first candidate orientation information and the second candidate orientation information; and determining the movement information of the connection portion according to the target orientation information.
The target orientation information here is information satisfying the first movement constraint condition.
For example, taking the neck as an example, if the face faces right, the corresponding orientation of the neck may be 90 degrees toward the right or 270 degrees toward the left. However, according to the physiological structure of the human body, the orientation of the neck of the human body cannot be changed by turning left by 270 degrees to make the neck face right. In this case, the orientation of the neck is: 90 degrees toward the right and 270 degrees toward the left, which are both candidate orientation information. The orientation information of the neck needs to be further determined, and needs to be determined according to the aforementioned first movement constraint condition. In this example, 90 degrees toward right of the neck is the target orientation information of the neck, and according to the 90 degrees toward the right of the neck, it is obtained that the current movement information of the neck is rotating 90 degrees toward the right.
In some embodiments, determining the orientation information of the two parts according to the features of the two parts includes: obtaining a first key point and a second key point for each of the two parts; obtaining a first reference point for each of the two parts, where the first reference point refers to a first predetermined key point in the target; generating a first vector based on the first key point and the first reference point, and generating a second vector based on the second key point and the first reference point; and determining the orientation information for each of the two parts based on the first vector and the second vector.
If the first part of the two parts is the shoulder of the human body, the first reference point of the first part may be the waist key point of the target or the midpoint of the key points of the two crotches. If the second part of the two parts is a human face, the first reference point of the second part may be the connection point of the neck connected to the human face and the shoulder.
In some embodiments, determining the orientation information for each of the two parts based on the two vectors includes: performing, for each part, cross product on the two vectors of the part to obtain a normal vector of a plane where the part is located; and taking the normal vector as the orientation information of the part.
Another vector can be obtained by cross-product calculation, and the vector is the normal vector of a plane where the connection portion is located. If the normal vector is determined, the orientation of the plane where the part is located is also determined; thus, it may be equivalent to determining the rotation angle of the connection portion with respect to a reference plane, i.e., equivalent to determining the movement information of the connection portion.
In some embodiments, the method further includes: determining movement information of the at least two parts based on the features; and determining, according to the features of the at least two parts and the first movement constraint condition of the connection portion, the movement information of the connection portion includes: determining the movement information of the connection portion according to the movement information of the at least two parts.
In some embodiments, determining the movement information of the connection portion according to the movement information of the at least two parts includes: obtaining a third 3D coordinate of the connection portion with respect to a second reference point, where the second reference point refers to a second predetermined key point in the at least two parts; and obtaining absolute rotation information of the connection portion according to the third 3D coordinate; and controlling the movement of the connection portion in the controlled model according to the movement information of the connection portion includes: controlling the movement of the connection portion in the controlled model based on the absolute rotation information.
In some embodiments, the second reference point may be one of the skeleton key points of the target. Taking the target being a person as an example, the second reference point may be a key point of a part connected by the first-type connection portion. For example, taking the neck as an example, the second reference point may be the key point of the shoulder connected by the neck.
In some other embodiments, the second reference point may be the same as the first reference point, for example, both the first reference point and the second reference point may be a root node of the human body, and the root node of the human body may be the midpoint of a connection line of the two key points of the crotches of the human body. The root node includes, but is not limited to, a key point 0 shown in FIG. 7B. FIG. 7B is a schematic diagram of a skeleton of a human body, and FIG. 7B includes totally 17 skeleton key points numbered 0 to 16.
In some other embodiments, controlling the movement of the connection portion in the controlled model based on the absolute rotation information includes: decomposing the absolute rotation information according to a traction hierarchy relationship among multiple connection portions in the target to obtain relative rotation information; and controlling the movement of the connection portion in the controlled model according to the movement information of the connection portion includes: controlling the movement of the connection portion in the controlled model based on the relative rotation information.
For example, the following is an example of a hierarchical relationship: the first hierarchy: pelvis; the second hierarchy: waist; the third hierarchy: thighs (e.g. the left thigh and the right thigh); the fourth hierarchy: calves (e.g., the left calf and the right calf); and the fifth hierarchy: feet.
For another example, the following is another hierarchical relationship: the first hierarchy: chest; the second hierarchy: neck; and the third hierarchy: head.
Furthermore, for example, the following is still another hierarchical relationship: the first hierarchy: clavicles, which correspond to the shoulders; the second hierarchy: upper arms; the third hierarchy: forearms (also referred to as lower arms); and the fourth hierarchy: hands.
The hierarchical relationship decreases in sequence from the first hierarchy to the fifth hierarchy. The movement of a part of a high hierarchy affects the movement of a part of a low hierarchy. Therefore, the hierarchy of a traction portion is higher than that of a connection portion.
During determining the movement information of the connection portion, movement information of key points corresponding to a part of each hierarchy is first obtained, and then the movement information of the key points of the part of the low hierarchy with respect to the key points of the part of the high hierarchy (i.e., the relative rotation information) is determined based on the hierarchical relationship.
For example, if a quaternion is used for representing movement information, the relative rotation information may be represented by the following calculation formula (1): rotation quaternions of each key point with respect to the camera coordinate system are {Q₀, Q₁, . . . , Q₁₈}, and then a rotation quaternion q_iof each key point with respect to a parent key point is calculated:
q _i =Q _parent(i) ⁻¹ ·Q _i (1)
where the parent key point parent(i) is the key point of the previous hierarchy of the current key point i; Q₁is a rotation quaternion of the current key point i with respect to the camera coordinate system; and Q_parent(i) ⁻¹is an inverse rotation parameter of the key point of the previous hierarchy. For example, if Q_parent(i)is a rotation parameter of the key point of the previous hierarchy, and the rotation angle is 90 degrees, the rotation angle of Q_parent(i) ⁻¹is −90 degrees.
In some embodiments, controlling the movement of the connection portion in the controlled model based on the absolute rotation information further includes: correcting the relative rotation information according to a second constraint condition; and controlling the movement of the connection portion in the controlled model based on the relative rotation information includes: controlling the movement of the connection portion in the controlled model based on the corrected relative rotation information.
In some embodiments, the second constraint condition includes: a rotatable angle of the connection portion.
In some embodiments, the method further includes: performing posture defect correction on the movement information of the connection portion to obtain corrected movement information of the connection portion. Controlling the movement of the connection portion in the controlled model according to the movement information of the connection portion includes: controlling the movement of the connection portion in the controlled model by using the corrected movement information of the connection portion.
For example, some users may have problems of non-standard body shapes, uncoordinated walking, etc. In order to reduce the occurrence of relatively weird actions due to direct simulation by the controlled model, in the embodiments, posture defect correction may be performed on the movement information of the connection portion to obtain the corrected movement information of the connection portion.
In some embodiments, the method further includes: performing posture defect correction on the movement information of the at least two parts to obtain corrected movement information of the at least two parts. Step S160 may include: controlling the movement of corresponding parts of the controlled model by using the corrected movement information of the at least two parts.
In some embodiments, the posture defect correction includes at least one of the following: synchronization defects of upper limbs and lower limbs; movement defects of looped legs; movement defects caused by toe-out feet; or movement defects caused by toe-in feet.
In some embodiments, the method further includes: obtaining a posture defect correction parameter according to difference information between the body shape of the target and a standard body shape, where the posture defect correction parameter is used for the correction of the movement information of the at least two parts and/or the movement information of the connection portion.
For example, before controlling the controlled model by using the image including the target, the body shape of the target is detected first, and then the detected body shape is compared with the standard body shape to obtain the difference information; and posture defect correction is performed through the difference information.
A prompt of maintaining a predetermined posture is output on a display interface, and after seeing the prompt, the user maintains the predetermined posture. In this way, an image device can acquire the image of the user who maintains the predetermined posture; and then, whether the user maintains the predetermined posture standardly enough is determined through image detection, so as to obtain the difference information.
For example, some people have toe-out feet, while the normal standard standing posture should be that the connection lines of the toes and heels of the feet are parallel to each other. During obtaining the movement information of the at least two parts corresponding to the features of the target and/or the movement information of the connection portion, the correction of such non-standard body shape (i.e., the posture defect correction) would be considered during the control of the controlled model. The predetermined posture may include, but is not limited to, an upright standing posture of the human body.
In some other embodiments, the method further includes: correcting the ratios of different parts of a standard model according to the ratio relationship of different parts of the target, to obtain the corrected controlled model.
There may be differences between the ratio relationships of the parts of different targets. For example, professional models have a greater leg to head ratio than ordinary people. For another example, some people have relatively full buttocks, so the gap between their two crotches may be larger than that of ordinary people.
The standard model may be a mean model based on a large amount of human body data. In order to enable the controlled model to more precisely simulate the movement of the target, in the embodiments, the ratios of the different parts of the standard model re corrected according to the ratios of the different parts of the target to obtain the corrected controlled model. For example, taking the target being a person as an example, the corrected parts include, but are not limited to: crotches and/or legs.
As shown in FIG. 3A, FIG. 3B, and FIG. 3C, the small image in the upper left corner of the image is the acquired image, and the lower right corner is the controlled model of the human body. The hand of the user is moving. From FIG. 3A to FIG. 3B, and then from FIG. 3B to FIC. 3C, the hand of the user is moving, and the hand of the controlled model is also following the movement. The hand movement of the user in FIG. 3A to FIG. 3C is changed in sequence from making a first to stretching out the palm and to extending the index finger, and the controlled model simulates the changes of the gesture of the user from making a first to stretching out the palm and to extending the index finger.
As shown in FIG. 4A, FIG. 4B, and FIG. 4C, the small image in the upper left corner of the image is the acquired image, and the lower right corner is the controlled model of the human body. The trunk of the user is moving. From FIG. 4A to FIG. 4B, and then from FIG. 4B to FIC. 4C, the trunk of the user is moving, and the trunk of the controlled model is also following the movement. From FIG. 4A to FIG. 4C, the user changes from pushing the crotch to the right of the image, to pushing the crotch to the left of the image, and finally to standing upright. The controlled model also simulates the user to make trunk movement.
As shown in FIG. 5A, FIG. 5B, and FIG. 5C, the small image in the upper left corner of the image is the acquired image, and the lower right corner is the controlled model of the human body. From FIG. 5A to FIG. 5C, the user strides toward the right side of the image, strides toward the left of the image, and finally stands upright. The controlled model also simulates the user to make foot movement.
In addition, in FIG. 4A to FIG. 4C, the controlled model also simulates the expression changes of the user.
As shown in FIG. 6, the embodiments provide an image processing apparatus, including the following modules.
A first obtaining module 110 is configured to obtain an image.
A second obtaining module 120 is configured to obtain features of at least two parts of a target based on the image.
A first determination module 130 is configured to determine, according to the features of the at least two parts and a first movement constraint condition of a connection portion, movement information of the connection portion, where the connection portion connects two of the at least two parts.
A control module 140 is configured to control the movement of the connection portion in a controlled model according to the movement information of the connection portion.
In some embodiments, the control module 140 is specifically configured to determine, according to the type of the connection portion, a control mode of controlling the connection portion; and control the movement of the connection portion in the controlled model according to the control mode and the movement information of the connection portion.
In some embodiments, the control module 140 is specifically configured to: in the case that the connection portion is a first-type connection portion, the control mode is a first-type control mode, where the first-type control mode is used for directly controlling the movement of a connection portion in the controlled model corresponding to the first-type connection portion.
In some embodiments, the control module 140 is specifically configured to: in the case that the connection portion is a second-type connection portion, the control mode is a second-type control mode, where the second-type control mode is used for indirectly controlling the movement of a connection portion in the controlled model corresponding to the second-type connection portion, and the indirect control is achieved by controlling a part in the controlled model corresponding to the part other than the second-type connection portion.
In some embodiments, the control module 140 is specifically configured to: in the case that the control mode is the second-type control mode, decompose the movement information of the connection portion to obtain first-type rotation information of the connection portion that the connection portion rotates under the traction of a traction portion; adjust movement information of the traction portion according to the first-type rotation information; and control the movement of the traction portion in the controlled model according to the adjusted movement information of the traction portion, to indirectly control the movement of the connection portion.
In some embodiments, the apparatus further includes: a decomposition module, configured to decompose the movement information of the connection portion, to obtain second-type rotation information of the second-type connection portion rotating with respect to the traction portion; and the control module 140 is further configured to control, in the controlled model, the rotation of the connection portion with respect to the traction portion by using the second-type rotation information.
In some embodiments, the second-type connection portion includes: a wrist; and an ankle.
In some embodiments, if the second-type connection portion is a wrist, the traction portion corresponding to the wrist includes: an upper arm and/or a forearm; and if the second-type connection portion is an ankle, the traction portion corresponding to the ankle includes: a thigh and/or a calf.
In some embodiments, the first-type connection portion includes a neck connecting the head and the trunk.
In some embodiments, the apparatus further includes: an orientation determination module, configured to determine respective orientation information of the two parts according to the features of the two parts connected by the connection portion; a second determination module, configured to determine candidate orientation information of the connection portion according to the respective orientation information of the two parts; and a selection module, configured to determine the movement information of the connection portion according to the candidate orientation information and the first movement constraint condition.
In some embodiments, the second determination module is configured to determine first candidate orientation information and second candidate orientation information of the connection portion according to the respective orientation information of the two parts.
In some embodiments, the selection module is specifically configured to: select target orientation information within an orientation change constraint range from the first candidate orientation information and the second candidate orientation information; and determine the movement information of the connection portion according to the target orientation information.
In some embodiments, the orientation determination module is specifically configured to: obtain a first key point and a second key point for each of the two parts; obtain a first reference point for each of the two parts, where the first reference point refers to a first predetermined key point in the target; generate a first vector based on the first key point and the first reference point, and generate a second vector based on the second key point and the first reference point; and determine the orientation information for each of the two parts based on the first vector and the second vector.
In some embodiments, the orientation control module is specifically configured to: perform, for each part, cross product on the two vectors of the part to obtain a normal vector of a plane where the part is located; and take the normal vector as the orientation information of the part.
In some embodiments, the apparatus further includes: a third determination module, configured to determine movement information of the at least two parts based on the features; and the first determination module 130 is specifically configured to determine the movement information of the connection portion according to the movement information of the at least two parts.
In some embodiments, the first determination module 130 is specifically configured to: obtain a third 3D coordinate of the connection portion with respect to a second reference point, where the second reference point refers to a second predetermined key point in the at least two parts; and obtain absolute rotation information of the connection portion according to the third 3D coordinate. The control module 140 is specifically configured to control the movement of the connection portion in the controlled model based on the absolute rotation information.
In some embodiments, the first determination module 130 is specifically configured to: decompose the absolute rotation information according to a traction hierarchy relationship among multiple connection portions in the target to obtain relative rotation information; and the control module 140 is specifically configured to control the movement of the connection portion in the controlled model based on the relative rotation information.
In some embodiments, the apparatus further includes: a correction module, configured to correct the relative rotation information according to a second constraint condition. The control module 140 is specifically configured to control the movement of the connection portion in the controlled model based on the corrected relative rotation information.
In some embodiments, the second constraint condition includes: a rotatable angle of the connection portion.
The following provides several specific examples with reference to any of the foregoing embodiments.

Example 1

The present example provides an image processing method, and the method includes the following steps.
An image is acquired, where the image includes a target, and the target includes, but is not limited to, a human body.
Face key points of the human body are detected, where the face key points may be contour key points of a face surface.
Trunk key points and/or limb key points of the human body are detected. The trunk key points and/or limb key points here may all be 3D key points, and are represented by 3D coordinates. The 3D coordinates may include 3D coordinates obtained by detecting 2D coordinates from a 2D image, and then using a conversion algorithm from 2D coordinates to 3D coordinates. The 3D coordinates may also be 3D coordinates extracted from a 3D image acquired using a 3D camera. The limb key points here may include: upper limb key points and/or lower limb key points. Taking a hand as an example, hand key points of the upper limb key points include, but are not limited to: wrist joint key points, metacarpophalangeal joint key points, finger joint key points, and fingertip key points. The positions of these key points can reflect the movements of the hand and fingers.
According to the face key points, mesh information of a face is generated. An expression base corresponding to a current expression of the target is selected according to the mesh information, and the expression of the controlled model is controlled according to the expression base. The expression intensity of the controlled model corresponding to each expression base is controlled according to an intensity coefficient reflected by the mesh information.
Quaternions are converted according to the trunk key points and/or the limb key points. The trunk movement of the controlled model is controlled according to the quaternion corresponding to the trunk key points; and/or, the limb movement of the controlled model is controlled according to the quaternion corresponding to the limb key points.
For example, the face key points may include: 106 key points. The trunk key points and/or the limb key points may include: 14 key points or 17 key points. The details are shown in FIG. 7A and FIG. 7B. FIG. 7A shows a schematic diagram including 14 skeleton key points; and FIG. 7B shows a schematic diagram including 17 skeleton key points.
FIG. 7B may be a schematic diagram of 17 key points generated based on the 14 key points shown in FIG. 7A. The 17 key points in FIG. 7B are equivalent to the key points shown in FIG. 7A, with addition of a key point 0, a key point 7, and a key point 9, where the 2D coordinates of the key point 9 may be preliminarily determined based on the 2D coordinates of a key point 8 and a key point 10, and the 2D coordinates of the key point 7 may be determined according to the 2D coordinates of the key point 8 and the 2D coordinates of the key point 0. The key point 0 may be a reference point provided by the embodiments of the present disclosure, and the reference point may serve as the foregoing first reference point and/or second reference point.
The controlled model in the present example may be a game character in a game scene, a teacher model in an online education video in an online teaching scene, and a virtual anchor in a virtual anchor scene. In short, the controlled model is determined according to application scenarios. If the application scenarios are different, the models and/or appearances of the controlled model are different.
For example, in conventional platform teaching scenes of mathematics and physics, the teacher model may be more sedate in clothing, such as a suit. For another example, for sports teaching scenes such as yoga or gymnastics, the controlled model may be wearing sportswear.

Example 2

The present example provides an image processing method, and the method includes the following steps.
An image is acquired, where the image includes a target, and the target includes, but is not limited to, a human body.
Trunk key points and limb key points of the human body are detected. The trunk key points and/or limb key points here may all be 3D key points, and are represented by 3D coordinates. The 3D coordinates may include 3D coordinates obtained by detecting 2D coordinates from a 2D image, and then using a conversion algorithm from 2D coordinates to 3D coordinates. The 3D coordinates may also be 3D coordinates extracted from a 3D image acquired using a 3D camera. The limb key points here may include: upper limb key points and/or lower limb key points. Taking a hand as an example, hand key points of the upper limb key points include, but are not limited to: wrist joint key points, metacarpophalangeal joint key points, finger joint key points, and fingertip key points. The positions of these key points can reflect the movements of the hand and fingers.
The trunk key points are converted into a quaternion representing the trunk movement, and the quaternion may be called a trunk quaternion. The trunk key points are converted into a quaternion representing the trunk movement, and the quaternion may be called a trunk quaternion.
The trunk movement of the controlled model is controlled using the trunk quaternion. The limb movement of the controlled model is controlled using a limb quaternion.
The trunk key points and the limb key points may include: 14 key points or 17 key points. The details are shown in FIG. 7A or FIG. 7B.
The controlled model in the present example may be a game character in a game scene, a teacher model in an online education video in an online teaching scene, and a virtual anchor in a virtual anchor scene. In short, the controlled model is determined according to application scenarios. If the application scenarios are different, the models and/or appearances of the controlled model are different.
For example, in conventional platform teaching scenes of mathematics and physics, the teacher model may be more sedate in clothing, such as a suit. For another example, for sports teaching scenes such as yoga or gymnastics, the controlled model may be wearing sportswear.

Example 3

The present example provides an image processing method, and the method includes the following steps.
An image is obtained, where the image includes a target, and the target may be a human body.
A 3D posture of the target in a 3D space is obtained according to the image, where the 3D posture may be represented by 3D coordinates of skeleton key points of the human body.
Absolute rotation parameters of joints of the human body in a camera coordinate system are obtained, where the absolute rotation position may be determined by the coordinates in the camera coordinate system.
Coordinate directions of the joints are obtained according to the coordinates. Relative rotation parameters of the joints are determined according to a hierarchical relationship. Determining the relative rotation parameters may specifically include: determining the positions of key points of the joints with respect to the root node of the human body. The relative rotation parameters may be used for quaternion representation. The hierarchical relationship here may be a traction relationship among the joints. For example, the movement of the elbow joint causes the movement of the wrist joint to some extent, and the movement of the shoulder joint also causes the movement of the elbow joint, and so on. The hierarchical relationship may be predetermined according to the joints of the human body.
The rotation of the controlled model is controlled using the quaternions.
For example, the following is an example of a hierarchical relationship: the first hierarchy: pelvis; the second hierarchy: waist; the third hierarchy: thighs (e.g. the left thigh and the right thigh); the fourth hierarchy: calves (e.g., the left calf and the right calf); and the fifth hierarchy: feet.
For another example, the following is another hierarchical relationship: the first hierarchy: chest; the second hierarchy: neck; and the third hierarchy: head.
Furthermore, for example, the following is still another hierarchical relationship: the first hierarchy: clavicles, which correspond to the shoulders; the second hierarchy: upper arms; the third hierarchy: forearms (also referred to as lower arms); and the fourth hierarchy: hands.
The hierarchical relationship decreases in sequence from the first hierarchy to the fifth hierarchy. The movement of a part of a high hierarchy affects the movement of a part of a low hierarchy. Therefore, the hierarchy of a traction portion is higher than that of a connection portion.
During determining the movement information of the connection portion, movement information of key points of a part of each hierarchy is first obtained, and then the movement information of the key points of the part of the low hierarchy with respect to the key points of the part of the high hierarchy (i.e., the relative rotation information) is determined based on the hierarchical relationship.
For example, if a quaternion is used for representing movement information, the relative rotation information may be represented by a calculation formula: rotation quaternions of each key point with respect to the camera coordinate system are {Q₀, Q₁, . . . , Q₁₈}, and then a rotation quaternion q_iof each key point with respect to a parent key point is calculated according to formula (1).
The above-mentioned control of the movement of each joint of the controlled model by using a quaternion may include: controlling the movement of each joint of the controlled model by using q_i.
In a further image processing method, the method further includes: converting the quaternion into a first Euler angle; transforming the first Euler angle to obtain a second Euler angle within a constraint condition, where the constraint condition may be used for angle limitation of the first Euler angle; and obtaining a quaternion corresponding to the second Euler angle, and then controlling the rotation of the controlled model by using the quaternion. By obtaining the quaternion corresponding to the second Euler angle, the second Euler angle may be directly converted into a quaternion.
Taking a human body as an example, the key points of 17 joints can be detected through human body detection, and moreover, two key points are set corresponding to the left and right hands, and therefore, there are 19 key points in total. FIG. 7B is a schematic diagram of a skeleton having 17 key points. FIG. 8 is a schematic diagram of a skeleton having 19 key points. The bones shown in FIG. 8 may correspond to the 19 key points, which respectively refer to the following bones: pelvis, waist, left thigh, left calf, left foot; right thigh, right calf, right foot, chest, neck, head, left clavicle, right clavicle, right upper arm, right forearm, right hand, left upper arm, left forearm, and left hand.
First, coordinates of the 17 key points in an image coordinate system may be obtained by detecting the key points of joints of a human body in an image, and may specifically be as follows: S={(x₀,y₀,z₀), . . . , (x₁₆,y₁₆,z₁₆)}. (x_i,y_i,z_i) may be the coordinates of an i^thkey point, and the value of i is from 0 to 16.
The coordinates of the key points of 19 joints in respective local coordinate systems may be defined as follows: A={(p₀,q₀), . . . , (p₁₈,q₁₈)}. p_irepresents the 3D coordinates of a node i in a local coordinate system, is generally a fixed value carried by the original model, and does not need to be modified and migrated. q_iis a quaternion, represents the rotation of a bone controlled by the node i in a parent node coordinate system thereof, and may also be considered as the rotation of a local coordinate system of the current node and a local coordinate system of a parent node.
The process of calculating the quaternions of the key points corresponding to all the joints may be as follows: determining coordinate axis directions of a local coordinate system of each node. For each bone, the direction pointing from a child node to a parent node is an x-axis; a rotation axis that causes a maximum rotatable angle of the bone is a z-axis, and if the rotation axis cannot be determined, a direction that the human body faces is taken as a y-axis. Reference may be made to FIG. 9 for details.
In the present example, a left-handed coordinate system is used for description, and during specific implementation, a right-handed coordinate system may also be used.


Serial numbers of
nodes in a skeleton	Using key points in a skeleton having
having 19 points	17 points for calculation

0	Take (1-7) × (1-4) as the y-axis, and (7-0)
	as the x-axis
1	Take the maximum default value of 3D coordinates
2	Take (14-11) × (14-7) as the y-axis, and (8-7)
	as the x-axis
3	Take the maximum default value of 3D coordinates
4	Take (10-8) as the x-axis, and (9-10) × (9-8)
	as the z-axis
5	Take (11-8) as the x-axis, and (12-11) × (11-8)
	as the y-axis
6	Take (12-11) as the x-axis, and (11-12) × (12-13)
	as the z-axis
7	Take (13-12) as the x-axis, and (11-12) × (12-13)
	as the z-axis; note: the node changes after the
	quaternions of the hands are added subsequently.
9	Take (5-4) as the x-axis, and (5-6) × (5-4)
	as the z-axis
10	Take (6-5) as the x-axis, and (5-6) × (5-4)
	as the z-axis
12	Take (14-8) as the x-axis, and (8-14) × (14-15)
	as the y-axis
13	Take (15-14) as the x-axis, and (14-15) × (15-16)
	as the z-axis
14	Take (16-15) as the x-axis, and (14-15) × (15-16)
	as the z-axis; note: the node changes after the
	quaternion of the hands are added subsequently.
16	Take (2-1) as the x-axis, and (2-3) × (2-1)
	as the z-axis
17	Take (3-2) as the x-axis, and (2-3) × (2-1)
	as the z-axis

In the table above, (i-j) represents a vector where i points to j, and x represents a cross product. For example, (1-7) represents a vector where a first key point points to a seventh key point.
In the table above, nodes 8, 15, 11, and 18 are the four key points of the hands and feet. Because the calculation of the quaternions of the four key points can be determined only when specific postures are used, the four key points are not included in this table. In addition, in the table above, the serial numbers of the nodes of the skeleton having 19 points are shown in FIG. 8, and for the serial numbers of the key points of the skeleton having 17 points, reference may be made to FIG. 7B.
The process of solving the first Euler angle is as follows.
After a local rotation quaternion q_iof the joints is calculated, it is first converted into an Euler angle, where the order of x-y-z is used by default.
Let q_i=(q0,q1,q2,q3), q0 is a real component; and q2,q2,q3 are all imaginary components. Thus, the calculation formulas of the Euler angle are represented by formulas (2)-(4):
X=a tan 2(2*(q0*q1−q2*q3),1−2*(q1*q1+q2*q2)) (2)
Y=a sin(2*(q1*q3+q0*q2)) and the value of Y is between −1 and 1 (3)
Z=a tan 2(2*(q0*q3−q1*q2),1−2*(q2*q2+q3*q3)) (4)
where X is an Euler angle in a first direction; Y is an Euler angle in a second direction; and Z is an Euler angle in a third direction. Any two of the first direction, the second direction, and the third direction are perpendicular.
Three angles (X, Y, Z) may be then limited. If the angles exceed the range, the angles are defined within boundary values to obtain corrected second Euler angles (X′, Y′, Z′). A new rotation quaternion q_i′ of the local coordinate system is restored.
In another further image processing method, the method further includes: performing posture optimization adjustment on second Euler angles. For example, adjustment is performed on some of second Euler angles so that they can be adjusted into posture-optimized Euler angles based on a preset rule, thereby obtaining third Euler angles. Thus, the obtaining of quaternions corresponding to the second Euler angles may include: converting the third Euler angles into quaternions for controlling a controlled model.
In still another further image processing method, the method further includes: converting a second Euler angle into a quaternion, and then performing posture optimization processing on the converted quaternion. For example, adjustment is performed based on a preset rule to obtain an adjusted quaternion, and the controlled model is controlled according to the finally adjusted quaternion.
In some embodiments, the adjustment performed on the second Euler angle or the quaternion obtained by conversion from the second Euler angle may be adjustment based on a preset rule, and may also be optimization adjustment performed by a deep learning model. There are various implementation modes, which are not limited in the present application.
In addition, a yet another image processing method may further include preprocessing. For example, the width of the crotch and/or shoulder of the controlled model is modified according to the size of the acquired human body so as to correct the overall posture of the human body. The standing posture of the human body, upright standing correction and abdomen lifting correction may be performed. Some people lift their abdomens when standing, and abdomen lifting correction makes the controlled model not simulate the abdomen lifting action of the user. Some people hunch when standing, and hunching correction makes the controlled model not simulate the hunching action, etc. of the user.

Example 4

The present example provides an image processing method, and the method includes the following steps.
An image is obtained, where the image includes a target, and the target may include at least one of a human body, human upper limbs, or human lower limbs.
A coordinate system of a target joint is obtained according to position information of the target joint in an image coordinate system. According to position information of a limb part in the image coordinate system, a coordinate system of a limb part capable of causing the target joint to move is obtained.
The rotation of the target joint with respect to the limb part is determined based on the coordinate system of the target joint and the coordinate system of the limb part to obtain rotation parameters, where the rotation parameters include self-rotation parameters of the target joint and rotation parameters under the traction of the limb part.
Limitation is performed on the rotation parameters under the traction of the limb part by using first angle limitation to obtain final traction rotation parameters. The rotation parameters of the limb part are corrected according to the final traction rotation parameters. Relative rotation parameters are obtained according to a coordinate system of the limb part and the corrected rotation parameters of the limb part; and second angle limitation is performed on the relative rotation parameters to obtain limited relative rotation parameters.
A quaternion is obtained according to the limited rotation parameters. The movement of the target joint of a controlled model is controlled according to the quaternion.
For example, if processing is performed on a human upper limb, a coordinate system of a hand in the image coordinate system is obtained, and a coordinate system of a lower arm and a coordinate system of an upper arm are obtained. In this case, the target joint is a wrist joint. The rotation of the hand with respect to the lower arm is decomposed into self-rotation and rotation under traction. The rotation under traction is transferred to the lower arm; specifically, the rotation under traction is assigned to the rotation of the lower arm in a corresponding direction. The maximum rotation of the lower arm is limited with the first angle limitation of the lower arm. Then, the rotation of the hand with respect to the corrected lower arm is determined to obtain a relative rotation parameter. Second angle limitation is performed on the relative rotation parameter to obtain the rotation of the hand with respect to the lower arm.
During performing processing on a human lower limb, a coordinate system of a foot in the image coordinate system is obtained, and a coordinate system of a calf and a coordinate system of a thigh are obtained. In this case, the target joint is an ankle joint. The rotation of the foot with respect to the calf is decomposed into self-rotation and rotation under traction. The rotation under traction is transferred to the calf; specifically, the rotation under traction is assigned to the rotation of the calf in a corresponding direction. The maximum rotation of the calf is limited with the first angle limitation of the calf. Then, the rotation of the foot with respect to the corrected calf is determined to obtain a relative rotation parameter. Second angle limitation is performed on the relative rotation parameter to obtain the rotation of the foot with respect to the calf.

Example 5

The neck controls the orientation of the head, and the face, human body, and human hands are separate components. The rotation of the neck is very important for the ultimate formation of the face, human body, and human hands into an integral whole.
The orientation of a human body may be calculated according to key points of the human body. Moreover, the orientation of a face can be calculated according to key points of the face. The relative position of the two orientations is the rotation angle of the neck. An angle problem of a connection portion is to be solved. The angle problem of the connection portion is solved through relative calculation. For example, if the body is 0 degree and the face is 90 degrees, to control a controlled model, it only focuses on part angles and angle changes of the head and the body. The angle of the neck of the controlled model needs to be calculated to control the head of the controlled model.
In the present example, the current orientation of the face of a user is determined based on an image, and then the rotation angle of the neck is calculated. Since the rotation of the neck has a range, for example, assuming that the neck can rotate 90 degrees at most, if the calculated rotation angle exceeds this range (−90 degrees to 90 degrees), the boundary of the range is taken as the rotation angle of the neck (such as, −90 degrees or 90 degrees).
The orientation of the body or face may be calculated using 3D key points. The specific calculation of the orientation may be: two vectors, which are not on a straight line, in a plane where the face or body is located, are performed cross product to obtain a normal vector of the plane, and the normal vector is the orientation of the face or body. The orientation may be taken as the orientation of a connection portion (the neck) between the body and the face.
As shown in FIG. 10, the embodiments of the present application provide an image device, including: a memory 1002, configured to store information; and a processor 1001, connected to the memory 1002 and configured to execute computer executable instructions stored on the memory 1002 so as to implement the image processing method provided by one or more of the foregoing technical solutions, for example, the image processing methods shown in FIG. 1A, FIG. 1B and/or FIG. 2.
The memory 1002 may be any type of memory, such as a random access memory, a read-only memory, and a flash memory. The memory 1002 may be configured to store information, for example, store computer executable instructions, etc. The computer executable instructions may be various program instructions, for example, a target program instruction and/or a source program instruction, etc.
The processor 1001 may be any type of processor, for example, a central processing unit, a microprocessor, a digital signal processor, a programmable array, an application-specific integrated circuit, an image processor, etc.
The processor 1001 may be connected to the memory 1002 by means of a bus. The bus may be an integrated circuit bus, etc.
In some embodiments, the terminal device may further include: a communication interface 1003. The communication interface 1003 may include a network interface, for example, a local area network interface, a transceiver antenna, etc. The communication interface is also connected to the processor 1001 and can be configured to transmit and receive information.
In some embodiments, the terminal device further includes a human-machine interaction interface 1005. For example, the human-machine interaction interface 1005 may include various input and output devices, for example, a keyboard and a touch screen, etc.
In some embodiments, the image device further includes: a display 1004. The display may display various prompts, acquired face images and/or various interfaces.
The embodiments of the present application provide a non-volatile computer storage medium. The computer storage medium stores computer executable codes. After the computer executable codes are executed, the image processing method provided by one or more of the foregoing technical solutions can be implemented, for example, the image processing methods shown in FIG. 1A, FIG. 1B, and/or FIG. 2.
It should be understood that the disclosed device and method in the embodiments provided in the present application may be implemented by other modes. The device embodiments described above are merely exemplary. For example, the unit division is merely logical function division and may be actually implemented by other division modes. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections among the components may be implemented by means of some interfaces. The indirect couplings or communication connections between the devices or units may be implemented in electronic, mechanical, or other forms.
The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, may be located at one position, or may be distributed on a plurality of network units. A part of or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
In addition, the functional units in the embodiments of the present disclosure may be integrated into one processing module, or each of the units may exist as an independent unit, or two or more units are integrated into one unit, and the integrated unit may be implemented in the form of hardware, or may also be implemented in the form of a hardware and software functional unit.
A person of ordinary skill in the art may understand that: all or some steps of implementing the forgoing method embodiments may be achieved by a program by instructing related hardware; the foregoing program may be stored in a computer-readable storage medium; when the program is executed, steps including the foregoing method embodiments are performed; moreover, the above-mentioned non-volatile storage medium includes various media capable of storing program codes such as a mobile storage device, an Read-Only Memory (ROM), a magnetic disk, or an optical disk.
The descriptions above only specific implementations of this disclosure. However, the scope of protection of this disclosure is not limited thereto. Within the technical scope disclosed by this disclosure, any variation or substitution that can be easily conceived of by those skilled in the art should all fall within the scope of protection of this disclosure. Therefore, the scope of protection of the present disclosure should be defined by the scope of protection of the claims.

Claims

1. An image processing method, comprising:

obtaining an image;

obtaining features of at least two parts of a target based on the image;

determining, according to the features of the at least two parts and a first movement constraint condition of a connection portion, movement information of the connection portion, wherein the connection portion connects two of the at least two parts; and

controlling movement of the connection portion in a controlled model according to the movement information of the connection portion.

2. The method according to claim 1, wherein controlling the movement of the connection portion in the controlled model according to the movement information of the connection portion comprises:

determining, according to a type of the connection portion, a control mode of controlling the connection portion; and

controlling the movement of the connection portion in the controlled model according to the control mode and the movement information of the connection portion.

3. The method according to claim 2, wherein determining, according to the type of the connection portion, the control mode of controlling the connection portion comprises:

in response to that the connection portion is a first-type connection portion, the control mode is a first-type control mode, wherein the first-type control mode is used for directly controlling movement of a connection portion in the controlled model corresponding to the first-type connection portion.

4. The method according to claim 2, wherein determining, according to the type of the connection portion, the control mode of controlling the connection portion comprises:

in response to that the connection portion is a second-type connection portion, the control mode is a second-type control mode, wherein the second-type control mode is used for indirectly controlling movement of a connection portion in the controlled model corresponding to the second-type connection portion, and the indirect control is achieved by controlling a part in the controlled model corresponding to a part other than the second-type connection portion.

5. The method according to claim 4, wherein controlling the movement of the connection portion in the controlled model according to the control mode and the movement information of the connection portion comprises:

in response to that the control mode is the second-type control mode, obtaining, by decomposing the movement information of the connection portion, first-type rotation information of the connection portion that the connection portion rotates under traction of a traction portion;

adjusting movement information of the traction portion according to the first-type rotation information; and

controlling movement of the traction portion in the controlled model according to the adjusted movement information of the traction portion, to indirectly control the movement of the connection portion.

6. The method according to claim 5, further comprising:

obtaining, by decomposing the movement information of the connection portion, second-type rotation information of the second-type connection portion rotating with respect to the traction portion; and

controlling, in the controlled model, rotation of the connection portion with respect to the traction portion by using the second-type rotation information.

7. The method according to claim 4, wherein the second-type connection portion comprises:

a wrist; and

an ankle.

8. The method according to claim 5, wherein

in response to that the second-type connection portion is a wrist, the traction portion corresponding to the wrist comprises: an upper arm and/or a forearm; and

in response to that the second-type connection portion is an ankle, the traction portion corresponding to the ankle comprises: a thigh and/or a calf.

9. The method according to claim 3, wherein

the first-type connection portion comprises a neck connecting a head and a trunk.

10. The method according to claim 1, wherein determining, according to the features of the at least two parts and the first movement constraint condition of the connection portion, the movement information of the connection portion comprises:

determining respective orientation information of the two parts according to the features of the two parts connected by the connection portion;

determining candidate orientation information of the connection portion according to the respective orientation information of the two parts; and

determining the movement information of the connection portion according to the candidate orientation information and the first movement constraint condition.

11. The method according to claim 10, wherein determining the candidate orientation information of the connection portion according to the respective orientation information of the two parts comprises:

determining first candidate orientation information and second candidate orientation information of the connection portion according to the respective orientation information of the two parts.

12. The method according to claim 11, wherein determining the movement information of the connection portion according to the candidate orientation information and the first movement constraint condition comprises:

selecting target orientation information within an orientation change constraint range from the first candidate orientation information and the second candidate orientation information; and

determining the movement information of the connection portion according to the target orientation information.

13. The method according to claim 10, wherein determining the respective orientation information of the two parts according to the features of the two parts comprises:

obtaining, for each of the two parts, a first key point and a second key point;

obtaining a first reference point for each of the two parts, wherein the first reference point refers to a first predetermined key point in the target;

generating a first vector based on the first key point and the first reference point;

generating a second vector based on the second key point and the first reference point; and

determining the orientation information for each of the two parts based on the first vector and the second vector.

14. The method according to claim 13, wherein determining the orientation information for each of the two parts based on the first vector and the second vector comprises: performing, for each part, cross product on the two vectors of the part to obtain a normal vector of a plane where the part is located; and

taking the normal vector as the orientation information of the part.

15. The method according to claim 1, further comprising:

determining movement information of the at least two parts based on the features; and

wherein determining, according to the features of the at least two parts and the first movement constraint condition of the connection portion, the movement information of the connection portion comprises:

determining the movement information of the connection portion according to the movement information of the at least two parts.

16. The method according to claim 15, wherein

determining the movement information of the connection portion according to the movement information of the at least two parts comprises:

obtaining a third 3-Dimensional (3D) coordinate of the connection portion with respect to a second reference point, wherein the second reference point refers to a second predetermined key point in the at least two parts; and

obtaining absolute rotation information of the connection portion according to the third 3D coordinate; and

controlling the movement of the connection portion in the controlled model according to the movement information of the connection portion comprises:

controlling the movement of the connection portion in the controlled model based on the absolute rotation information.

17. The method according to claim 16, wherein

controlling the movement of the connection portion in the controlled model based on the absolute rotation information further comprises:

obtaining, by decomposing the absolute rotation information according to a traction hierarchy relationship among a plurality of connection portions in the target, relative rotation information; and

controlling the movement of the connection portion in the controlled model based on the relative rotation information.

18. The method according to claim 17, wherein

correcting the relative rotation information according to a second constraint condition, wherein the second constraint condition comprises a rotatable angle of the connection portion; and

controlling the movement of the connection portion in the controlled model based on the relative rotation information comprises:

controlling the movement of the connection portion in the controlled model based on the corrected relative rotation information.

19. An image device, comprising:

a memory storing computer executable instructions; and

a processor coupled to the memory

wherein the processor is configured to:

obtain an image;

obtain features of at least two parts of a target based on the image;

determine, according to the features of the at least two parts and a first movement constraint condition of a connection portion, movement information of the connection portion, wherein the connection portion connects two of the at least two parts; and

control movement of the connection portion in a controlled model according to the movement information of the connection portion.

20. A non-volatile computer storage medium storing computer-executable instructions that are executed by a processor to:

obtain an image;

obtain features of at least two parts of a target based on the image;