WO2021169839A1 - 一种基于骨骼关键点的动作还原方法以及装置 - Google Patents

一种基于骨骼关键点的动作还原方法以及装置 Download PDF

Info

Publication number
WO2021169839A1
WO2021169839A1 PCT/CN2021/076723 CN2021076723W WO2021169839A1 WO 2021169839 A1 WO2021169839 A1 WO 2021169839A1 CN 2021076723 W CN2021076723 W CN 2021076723W WO 2021169839 A1 WO2021169839 A1 WO 2021169839A1
Authority
WO
WIPO (PCT)
Prior art keywords
key point
bone key
bone
target
point
Prior art date
Application number
PCT/CN2021/076723
Other languages
English (en)
French (fr)
Inventor
孙继强
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2021169839A1 publication Critical patent/WO2021169839A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30164—Workpiece; Machine component

Definitions

  • This application relates to the field of robot actions, and in particular to a method and device for restoring actions based on bone key points.
  • robots are increasingly entering the lives and work of ordinary people, and various types of robot products have appeared in the direction of housework, entertainment, education, etc.
  • robots that look like humans or cartoons have body movements.
  • Head movements and facial expressions need to be pre-arranged according to machine movements to control the robot to complete specific actions, so as to bring a better user experience in the process of human-computer interaction.
  • the animation production process of different robots is different and will depend on the hardware form and structure of the robot. Therefore, the more freedom of the robot, the more complicated it will be.
  • existing robot animation production schemes generally obtain multiple frames of two-dimensional planar images corresponding to a series of actions through a piece of target object's action video (such as the actions of human limbs); Posture: According to preset constraints and algorithms, each frame of human posture is restored on the corresponding robot in a time-series manner, so as to generate a series of corresponding robot actions.
  • computer vision technology is generally used in the image or video to determine the position of each bone key point of the target object, and then restore the action according to the position change of the joint point; but For the actions of multi-degree-of-freedom robots, it is troublesome to restore actions from two-dimensional images.
  • the shoulder joint of a real person can move in three directions, while the robot may only be able to move in a few directions due to the limitation of the motor volume.
  • the two-dimensional image lacks depth information, and it is difficult to restore the true position of a specific bone key point (the specific bone key point corresponds to the corresponding structural position of the robot) in space based on the two-dimensional coordinates. Therefore, the distortion of the action is caused, and the completion of the robot action is more directly reduced.
  • the embodiments of the present application provide a device and method for restoring actions based on bone key points, which can effectively restore the corresponding actions in the three-dimensional space according to the two-dimensional planar image and the constraint conditions of the preset actions.
  • the embodiments of the present application provide a method of motion restoration based on bone key points, which can be applied to robot motion generation, and the method may include:
  • the target plane is used to indicate the relative position relationship of the bone key point of the target object mapped from the three-dimensional space to the two-dimensional plane;
  • the spatial position of the second bone key point is determined according to the coordinate position of the second bone key point and the target curve.
  • the motion parameters are obtained according to a series of motions on the two-dimensional plane projection, and the robot motion is deduced inversely, so as to realize the motion trajectory of the robot arm in space.
  • select the bone key points associated with the motion for example, the bone key points of the shoulder joint point and the wrist point are used as the first bone key point and the second bone key point.
  • a two-dimensional coordinate system is established; for example, taking the shoulder joint as the coordinate origin, preset multiple possible positions of the second bone key points.
  • the robot arm moves 360° at the maximum amplitude
  • the motion trajectory of the wrist position is a circle
  • the center of the circle corresponds to the position of the arm shoulder (ie the shoulder joint point); due to the influence of the machine structure, the second The dimensional shape can be an irregular closed curve.
  • the robot arm moves 360°, and multiple sets of closed curves can be obtained, that is, one or more irregular closed curves with the first bone key point as the center point. Each point on the curve may be the position of the wrist point.
  • the positions of the two bone key points on the frame of image and the target position of the second bone key point (ie, the shoulder joint point) on the target curve are determined.
  • Each point on the curve corresponds to the angle value of the steering gear rotation at the corresponding structural position of one or more robots (that is, a type of steering gear control signal).
  • the method before acquiring the coordinate position of the first bone key point on the target plane, the method further includes: acquiring image information of the target object; and determining the target object based on the image information
  • the first bone key point and the second bone key point of, the first bone key point and the second bone key point are adjacent bone key points of the target object.
  • the image information including the target object is first acquired, and then the first bone key point and the second bone key point among the multiple bone key points of the target object are determined from the image information.
  • the image information is an image, and the first bone key point and the second bone key point are extracted from the image according to a preset bone key point recognition algorithm, and the relationship between the two is determined.
  • the image information is a video; the first bone key point and the second bone key point are adjacent bone key points on the same joint in the same frame of video;
  • the image information determining the first bone key point and the second bone key point of the target object includes: determining multiple bone key points of the target object in the same frame of video; from the multiple bone key points Determine the first bone key point and the second bone key point adjacent to the first bone key point on the same joint.
  • a pair of related bone key points is found among the determined multiple bone key points; and according to the body structure of the target object and the preset extraction position of the bone key points, the multiple bone key points
  • the relevant positions of the required first bone key point and the second bone key point are determined from the points.
  • the relatively fixed position of the bone key points among the plurality of bone key points has a small change in position in each frame of the video image.
  • the determining multiple key bone points of the target object in the same frame of video includes: extracting each frame of video pictures from the video of the target object, and obtaining all The target plane corresponding to each frame of video picture; according to a preset bone key point recognition algorithm, a plurality of bone key points of the target object in the target plane corresponding to each frame of video picture are determined.
  • a video is obtained by shooting a target object (such as a human body); then according to extracting multiple frames of video images from the video; obtaining corresponding multiple video images in a time sequence, and mapping each video image to a two-dimensional On the target plane, in order to restore the data processing of the video action according to the image action.
  • the method further includes: sequentially determining the positions in each frame of video pictures according to the arrangement order of each frame of video pictures.
  • the spatial position of the second bone key point is used to generate the movement track of the second bone key point.
  • the spatial position of the second bone key point in each frame of video picture is determined, the spatial position of the second bone key point in each continuous video image is restored to a coherent action to determine the second bone key point. The movement path of the bone key point.
  • the determining the target curve to which the second bone key point belongs according to the coordinate position of the second bone key point on the target plane includes: on the target plane A two-dimensional plane coordinate system is established to determine the two-dimensional coordinates of the second bone key point on the target plane; according to the two-dimensional coordinates, the target curve to which the second bone key point belongs is determined.
  • the accurate target position of the second bone key point on the curve is determined through the two-dimensional coordinates of the first bone key point and the second bone key point, which improves the accuracy of the image restoration action.
  • the distance relationship between the first bone point and the second bone point is expressed by a relative distance; the relative distance is between the first bone key point and the second bone key point The ratio of the actual distance on the image information to the maximum distance that can be presented on the image information between the first bone key point and the second bone key point.
  • the embodiment of the present application provides a method of expressing distance; that is, expressing the distance between a point and a point by a relative distance, which can reduce the error of the distance between the points.
  • the embodiments of the present application provide a device for restoring a motion based on bone key points, which can be applied to the motion generation of a robot, and the device may include:
  • a coordinate acquisition unit configured to acquire the coordinate position of the first bone key point on a target plane;
  • the target plane is used to indicate the relative position relationship of the bone key point of the target object from the three-dimensional space to the two-dimensional plane;
  • the curve preset unit is used to determine one or more curves on the target plane with the coordinate position of the first bone key point as the center point, and the curve is used to indicate that the second bone key point is mapped to the All possible positions on the target plane; the coordinate position of each point on the curve corresponds to one or more angle information of the second bone key point in the three-dimensional space;
  • the target curve determining unit is configured to determine the target curve to which the second bone key point belongs according to the coordinate position of the second bone key point on the target plane, and the target curve is the one or more curves one of the;
  • the spatial position determining unit is configured to determine the spatial position of the second bone key point according to the coordinate position of the second bone key point and the target curve.
  • the device further includes an image information acquisition unit and a bone key point unit; the image information acquisition unit is configured to acquire the coordinates of the first bone key point on the target plane. Before the position, obtain the image information of the target object; the bone key point unit is used to determine the first bone key point and the second bone key point of the target object based on the image information, the first bone key point The point and the second bone key point are adjacent bone key points of the target object.
  • the image information is a video; the first bone key point and the second bone key point are adjacent bone key points on the same joint in the same frame of video; the device further It includes a multi-skeletal keypoint unit and a target bone keypoint unit; the multi-skeletal keypoint unit is used to determine multiple bone keypoints of the target object in the same frame of video; the target bone keypoint unit, It is used to determine the first bone key point from the plurality of bone key points and the second bone key point adjacent to the first bone key point on the same joint.
  • the multi-skeletal keypoint unit is specifically configured to: extract each frame of video picture from the video of the target object, and obtain the target plane corresponding to each frame of video picture; According to a preset bone key point recognition algorithm, multiple bone key points of the target object in the target plane corresponding to each frame of the video picture are determined.
  • the device further includes a trajectory determining unit, configured to: after determining the spatial position of the second bone key point, according to the sequence of each frame of video pictures, sequentially The spatial position of the second bone key point in each frame of video picture is determined to generate a motion track of the second bone key point.
  • a trajectory determining unit configured to: after determining the spatial position of the second bone key point, according to the sequence of each frame of video pictures, sequentially The spatial position of the second bone key point in each frame of video picture is determined to generate a motion track of the second bone key point.
  • the target curve determining unit is specifically configured to: establish a two-dimensional plane coordinate system on the target plane, and determine the second bone key point on the target plane. Dimensional coordinates; according to the two-dimensional coordinates, determine the target curve to which the second bone key point belongs.
  • the device further includes a distance unit for expressing the distance relationship between the first bone point and the second bone point by a relative distance; the relative distance is the first bone key point The actual distance from the second bone key point on the image information and the maximum distance between the first bone key point and the second bone key point that can be presented on the image information Compare.
  • an embodiment of the present application provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, the computer program includes program instructions, and the program instructions when executed by a processor The processor is caused to execute the method described in the first aspect.
  • embodiments of the present application provide a computer program product containing instructions, which when run on a processor, cause the processor to execute the method described in the first aspect.
  • an embodiment of the present application provides an electronic device, which may include: the device for restoring actions based on bone key points as described in the first aspect, and a device coupled to the outside of the device for restoring actions based on bone key points.
  • Discrete device may include: the device for restoring actions based on bone key points as described in the first aspect, and a device coupled to the outside of the device for restoring actions based on bone key points.
  • an embodiment of the present application provides a terminal, the terminal includes a processor, and the processor is configured to support the terminal to perform a corresponding function in the method for restoring an action based on bone key points provided in the first aspect.
  • the terminal may also include a memory, which is used for coupling with the processor and stores necessary program instructions and data for the terminal.
  • the terminal may also include a communication interface for the terminal to communicate with other devices or communication networks.
  • FIG. 1 is a schematic diagram of an application scenario of robot motion generation provided by an embodiment of the present application
  • FIG. 2 is a schematic diagram of a system architecture corresponding to a method for restoring actions based on bone key points according to an embodiment of the present application
  • Fig. 3 is a schematic diagram of a flow of robot action generation provided by an embodiment of the present application.
  • FIG. 4 is a schematic diagram of a method for restoring actions based on bone key points according to an embodiment of the present application
  • Fig. 5 is a schematic front view of an irregular closed curve provided by an embodiment of the present application.
  • Fig. 6 is a schematic side view of the curve shown in Fig. 5 provided by an embodiment of the present application.
  • FIG. 7 is a movement trajectory diagram of a second bone key point provided by an embodiment of the present application.
  • FIG. 8 is a schematic diagram of another action restoration method based on bone key points according to an embodiment of the present application.
  • FIG. 9 is a schematic diagram of a method for mapping three-dimensional point coordinates according to an embodiment of the present application.
  • FIG. 10 is a schematic diagram of a two-dimensional mapping plane provided by an embodiment of the present application.
  • FIG. 11 is a schematic diagram of multiple bone key points of a target object provided by an embodiment of the present application.
  • FIG. 12 is a schematic diagram of a point-to-point distance representation method provided by an embodiment of the present application.
  • FIG. 13 is a mapping relationship between a robot and a human body movement provided by an embodiment of the present application.
  • FIG. 14 is a schematic structural diagram of a motion restoration device based on bone key points provided by an embodiment of the present application.
  • FIG. 15 is a schematic structural diagram of a motion restoration device based on bone key points according to an embodiment of the present application.
  • FIG. 16 is a schematic structural diagram of a device provided by an embodiment of the present application.
  • component used in this specification are used to denote computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution.
  • the component may be, but is not limited to, a process, a processor, an object, an executable file, an execution thread, a program, and/or a computer running on a processor.
  • the application running on the computing device and the computing device can be components.
  • One or more components may reside in processes and/or threads of execution, and components may be located on one computer and/or distributed among two or more computers.
  • these components can be executed from various computer readable media having various data structures stored thereon.
  • the component can be based on, for example, a signal having one or more data packets (e.g. data from two components interacting with another component in a local system, a distributed system, and/or a network, such as the Internet that interacts with other systems through a signal) Communicate through local and/or remote processes.
  • a signal having one or more data packets (e.g. data from two components interacting with another component in a local system, a distributed system, and/or a network, such as the Internet that interacts with other systems through a signal) Communicate through local and/or remote processes.
  • Degree of freedom according to mechanical principles, the mechanism has the number of independent motion parameters that must be given when determining the motion (that is, the number of independent generalized coordinates that must be given in order to determine the position of the mechanism), which is called Degree of freedom of mechanism (degree of freedom of mechanism), the number of which is often denoted by F.
  • Skeleton key point data is a kind of data that uses key points to describe the actions of the human body.
  • SDK Software Development Kit
  • Software development tools include, in a broad sense, a collection of related documents, examples, and tools that assist in the development of a certain type of software.
  • a software development kit is a collection of development tools used by software engineers to create application software for specific software packages, software frameworks, hardware platforms, operating systems, etc.
  • SDK is used to develop applications under the Windows platform. SDK.
  • Image recognition refers to the use of computers to process, analyze, and understand images to identify targets and objects in various patterns. It is a practical application of deep learning algorithms. At present, image recognition technology is generally divided into face recognition and product recognition. Face recognition is mainly used in security inspection, identity verification and mobile payment; product recognition is mainly used in the circulation of goods, especially unmanned shelves, smart retail cabinets, etc. People in the retail field. The traditional image recognition process is divided into four steps: image acquisition ⁇ image preprocessing ⁇ feature extraction ⁇ image recognition.
  • AI Artificial Intelligence
  • the following exemplarily enumerate the application scenarios of the bone key point-based motion restoration method in the present application, which may include the following application scenarios of robot motion generation.
  • FIG. 1 is a schematic diagram of an application scenario for generating robot actions according to an embodiment of the present application.
  • the application scenario includes a camera device (a camera is taken as an example in the embodiment of the present application), a target object, a terminal, and a robot.
  • the target object may be a human, an animal, or other non-living things (for example, a robot or a mechanical structure), which is not limited in the embodiment of the present application.
  • the camera is used to photograph the movement changes of the target object over a period of time; for example, when the target object is a human body, photograph the movement of the limbs of the human body within ten minutes.
  • the target object is an animal, the running situation of the animal within half an hour is photographed.
  • the camera can be combined with other hardware devices to pre-process the captured data to a certain extent, such as removing the video content of the target object without movement, so as to improve the efficiency of subsequent video processing.
  • the terminal is used for receiving the video data sent by the camera; further processing the video data, and then transmitting the motion parameters of the corresponding robot obtained according to the human motion in the video to the corresponding robot. For example, map each frame of video image received to a two-dimensional plane; determine the movement of the human body in each frame of the image through the two-dimensional coordinate system; then, according to the coordinates of the human movement and preset constraints, set it on the corresponding robot Restore a series of actions.
  • the aforementioned data processing, receiving, and sending processes may also be completed through the server. The embodiment of the application does not limit this.
  • the robot is used to receive various control parameters sent by the terminal, and complete the posture of each frame of the image according to the various parameters in a certain time sequence to finally restore a series of actions in the action video of the target object.
  • the human body ie the target object
  • a camera a device that runs an algorithm program (such as a camera driver, an AI image processing program, a human body motion recognition program, a robot motion conversion program, etc.), smart robot.
  • the camera collects human movements, and the data is processed to extract key points of human bones, generally including shoulders, elbows, wrists, neck, head, thigh roots, knees, feet and other key parts, and output as key bone points on a two-dimensional plane Coordinates, the human body motion is extracted through the processing of the human body motion recognition program, combined with the characteristics of different robot hardware, the human body motion is converted into the robot motion, and the human body motion is finally converted into the robot motion.
  • an animation file is generated and applied to the robot.
  • FIG. 1 is only an exemplary implementation in the embodiment of the present application, and the application scenario in the embodiment of the present application includes but is not limited to the above application scenario.
  • FIG. 2 is a schematic diagram of a system architecture corresponding to a bone key point-based motion restoration method provided by an embodiment of the present application.
  • the bone key point-based motion restoration method proposed in this application can be applied to the system architecture.
  • the system architecture includes a camera drive module, an AI image processing module, a human motion recognition module, a robot motion conversion module, and a motion fine-tuning module. in,
  • the camera driver module is used to adapt the camera hardware, use the camera's SDK to complete the image data processing, and realize the collection of human movements.
  • the AI image processing module is used to complete image processing based on machine learning methods. Through the training and tuning of the model, the recognition and tracking of human facial expressions, torso, limbs and even fingers are realized; in the examples of this application, only the human body is used. The data of the torso is described by 14 key points.
  • the restoration of actions in the embodiments of the present application is not limited to human trunks such as limbs, and may also include facial expressions and the like.
  • the human body action recognition module is used to describe the human body's actions in a specific method according to the plane two-dimensional coordinate data, combine multiple images to recognize the body's posture changes, apply filter algorithms and key frame extraction algorithms to obtain relatively coherent action sequences.
  • the robot motion conversion module is used to convert motion data into robot motions according to the characteristics of the robot's hardware structure and degree of freedom.
  • the action fine-tuning module is used to obtain the action sequence of the robot after completing the input of the human body movement, and can support the adjustment of the action.
  • the action fine-tuning module provides Support friendly.
  • the embodiments of this application can quickly complete the generation of robot movements according to the movements of the human body, greatly shorten the time of animation production, reduce the operation difficulty of users, and improve the efficiency of animation production. Because it is directly recorded human movements, it is coordinated, smooth, and smooth. Naturally, it is consistent with human movements.
  • Fig. 3 is a schematic diagram of a robot motion generation process provided by an embodiment of the present application; as shown in Fig. 3, the human body motion is captured by a camera and AI image processing capabilities are used to extract the key human bones from the image. Point information, using the coordinates of the two-dimensional plane to describe the key points of the bones; in the process of changing the body posture, some key points are related to each other. In order to better reflect these characteristics, use the direction and amplitude to describe these relationships , It can realize the recognition of human movements; combined with the robot's hardware structure and degree of freedom characteristics, the robot can make the same movements.
  • the movable range of each limb of the robot is divided into two maintenances: direction and amplitude, and multiple closed-loop orbits can be obtained. These orbits are sequentially distributed in space. The recognized human movements and these orbits will produce Intersection point, the curve passing through all intersection points in space is the motion trajectory of the robot limbs. Due to the lack of information in the two-dimensional space and the user may adjust the desired action, the action fine-tuning module provides fine-tuning capabilities, and finally generates an action file for use on the robot.
  • system architecture in FIG. 2 is only an exemplary implementation in the embodiment of the present application, and the system architecture in the embodiment of the present application includes but is not limited to the above system architecture.
  • FIG. 4 is a schematic diagram of an action restoration method based on bone key points provided by an embodiment of the present application.
  • the action restoration method based on bone key points can be applied to an action restoration system based on bone key points (including the above-mentioned system). Architecture), and is suitable for the application scenario shown in Figure 1 above. The description will be made from a single side of the terminal with reference to FIG. 4 below.
  • the method may include the following steps S401 to S404.
  • Step S401 Obtain the coordinate position of the first bone key point on the target plane.
  • the target plane is obtained by mapping the posture and shape of the target object in space, and multiple bone key points of the target object are extracted according to the recognition algorithm.
  • Select a bone key point as the first bone key point for example, the bone key point corresponding to the shoulder joint
  • specify the coordinate position of the first bone key point on the target plane such as x-coordinate data and y-coordinate data.
  • the target plane is used to indicate the relative position relationship of the bone key points of the target object mapped from the three-dimensional space to the two-dimensional plane.
  • Step S402 Determine one or more curves on the target plane with the coordinate position of the first bone key point as the center point.
  • one or more curves are preset.
  • the curve can be a closed or semi-closed curve; according to the limitations of the machine structure and the body structure of the target object, the curve can be a regular or irregular curve.
  • the curve is used to indicate that the second bone key point is mapped to all possible positions on the target plane; the coordinate position of each point on the curve corresponds to the position of the second bone key point in the three-dimensional space One or more angle information.
  • one or more irregularly closed curves with the first bone key point as the center point on the target plane are determined. For example, after mapping a three-dimensional video image to a two-dimensional plane (ie, a target plane), one or more irregular closed curves are determined on the two-dimensional plane. Each position on the curve corresponds to one or more steering gear control signals of the robot.
  • the curve includes all possible positions of the second bone key point; where each point on the curve represents the position of the second bone key point (the wrist point is taken as an example for illustration in this application). Location.
  • the drawing of the curve can determine one or more curves according to the preset joint length and the position of the shoulder joint point. Since the conversion of a two-dimensional image into a three-dimensional space requires certain constraints, the following constraints can be referred to.
  • the embodiment of the present application may assume that the arm of the robot moves on the front side of the body.
  • FIG. 5 is a schematic front view of an irregular closed curve provided by an embodiment of the present application; from the front view of the robot in FIG. To move 360°; that is, the trajectory of the wrist is a circle (the center of the circle corresponds to the shoulder of the left arm), and the shape of the trajectory on the plane is an irregular closed curve. It is understandable that the motion trajectory of the wrist is affected by the structure of the robot and does not present a circular trajectory.
  • the center point is the first bone key point (that is, the bone key point corresponding to the left shoulder), and the curve with the center point as the core is the set of all possible positions of the second bone key point (that is, the bone key point corresponding to the wrist).
  • the maximum amplitude cannot be reached at some angles.
  • the embodiments of the present application do not limit the specific shape of the curve.
  • FIG. 6 is a schematic side view of the curve shown in FIG. 5 provided by an embodiment of the present application; the curve shown in FIG. 6 and the curve shown in FIG. 5 are views of the same curve at different angles.
  • the motion trajectory of the robot arm on the front side of the body is divided into two dimensions: the ratio of the direction and the amplitude.
  • the position of each point on the curve corresponds to the degree of freedom of the point (that is, the direction of movement of the point).
  • the degree of freedom can restore the two-dimensional coordinates of the combined point to the corresponding action in the video in the three-dimensional space.
  • the degree of freedom of the arm is generally 3-4, and the following takes 3 degrees of freedom as an example.
  • a 360° angle of movement is divided into N angle segments; please refer to Table 1 for specific data, as shown below:
  • Angle_1 represents angle data
  • Value_1_1, Value_1_2, and Value_1_3 represent motion data on three preset degrees of freedom under the angle.
  • Figure 7 is a motion trajectory diagram of a second bone key point provided by an embodiment of the present application; as shown in FIG. 7, a circle of irregular curves centered on a central dot is a motion curve in the direction of 360° under the same relative amplitude. , Divided according to different degrees of fineness, you can get a set of curves. Among them, there is an intersection point between a curve and each curve, that is, the direction of the motion data.
  • the angle between point 1 and the y axis is 1', which corresponds to the aforementioned Angle_1; then when the robot corresponds to point 1 with three steering gears, point 1 corresponds to three degrees of freedom and the corresponding motion parameters, so that the robot’s The corresponding structure can be moved to the position of point 1.
  • the first bone key point and the second bone key point are two associated bone key points on the same joint in the same frame of video image.
  • the target position of the second bone key point on the target curve is determined according to the two-dimensional coordinates of the first bone key point and the second bone key point on the target plane.
  • the target curve is one of the one or more irregularly closed curves.
  • the coordinates of the first bone key point and the second bone key point are determined according to the set two-dimensional coordinates. For example, a certain frame of video image in the video is mapped to a two-dimensional target plane, and several bone key points of the target object are extracted. Taking the shoulder joint point and the wrist point as examples, the position of two points is determined on the target plane, and the coordinate data of the two points can be obtained, namely the abscissa data and the ordinate data.
  • the target of the second bone key point on the target curve is determined according to the two-dimensional coordinates of the first bone key point and the second bone key point on the target plane Location, including:
  • the two-dimensional coordinates of the first bone key point and the second bone key point on the target plane determine the distance between the first bone key point and the second bone key point, and the second bone key point The direction of the second bone key point relative to the first bone key point;
  • the target position of the second bone key point on the target curve is determined according to the distance.
  • the distance between the first bone key point and the second bone key point is expressed as the actual distance between the first bone key point and the second bone key point And the ratio of the maximum distance between the first bone key point and the second bone key point.
  • Step S403 Determine the target curve to which the second bone key point belongs according to the coordinate position of the second bone key point on the target plane.
  • the coordinate position of the second bone key point on the target plane is determined, it is determined according to the coordinate data (ie, the coordinate position) that the second bone key point falls on a certain curve among the preset curves.
  • the curve centered on the first bone key point is generally densely distributed in multiple groups, so the second bone key point must fall on one of the curves, and this curve is the target curve.
  • the target curve is one of the one or more curves.
  • the spatial position of the second bone key point is determined according to one or more steering gear control signals of the robot corresponding to the target position. For example, after determining the target position of the point, according to the preset steering gear control signal corresponding to the target position, the steering gear corresponding to the position is controlled to rotate or move according to the parameters, so that the point moves to the specified spatial position.
  • the target position corresponds to 2 degrees of freedom (forward degrees of freedom and right degrees of freedom), and each degree of freedom specifies the rotation angle of the steering gear; then control the forward steering gear to rotate a certain angle, and then Control the steering gear to the right to rotate a certain angle, and finally make the point reach the target.
  • the second bone key point After determining the specific target position of the second bone key point in the target curve according to the coordinate data of the second bone key point, the second bone key point can be restored from the degree of freedom data corresponding to the point at the target position of the curve.
  • the spatial location It is equivalent to determining the three-dimensional coordinate data according to the z-axis coordinate data of the point on the curve after the x-axis and y-axis coordinates of the point are known; the position of the point is restored by passing the control parameters to the robot.
  • the second bone key point control the operation of the steering gear of the robot corresponding structure; the second bone key point is on the target surface; the target surface is the target surface.
  • One of the one or more irregular closed curved surfaces; each position on each of the one or more irregular closed curved surfaces corresponds to one or more steering gear control signals.
  • one or more irregular closed curves with the first bone key point as the center point are determined as one or more preset position trajectories of the second bone key point;
  • the bone key point and the second bone key point are two associated bone key points in the same frame of video image; according to the position information of the first bone key point and the second bone key point, the first bone key point is calculated The distance between a bone key point and the second bone key point, and the direction of the second bone key point relative to the first bone key point; when the second bone key point is relative to the first bone In the direction of the key point, the position of the second bone key point on the target position trajectory is determined according to the distance, and the target position trajectory is one of the one or more preset position trajectories.
  • the position information is the point coordinates of the bone key points; the method further includes: determining multiple bone key points of the target object in the same frame of video image; The first bone key point and the second bone key point associated with the first bone key point are determined in the bone key points; according to the first bone key point, the second bone key point and the coordinates System, determine the point coordinates of the first bone key point and the second bone key point.
  • the method further includes: according to the multiple frames of time-sequenced video images, The position information of the second bone key point in each frame of the video image in the multiple frames of time-series video images is sequentially determined, and the motion sequence of the second bone key point is generated.
  • restore the second bone key point on the machine structure Location changes.
  • the one or more is determined according to multiple distance values that can be reached between the first bone key point and the second bone key point.
  • An irregular closed curve An action sequence corresponding to the target object is generated, and each action in the action sequence includes position information of the multiple bone key points.
  • the action video is a video that captures the movement of the target object over a period of time; collects the human body action video, extracts the position information of the key points of the human skeleton in each frame, and forms a whole body skeleton key point action sequence.
  • acquiring an image set of the target object the image set including a plurality of images of different forms of the target object; and determining the form of the target object corresponding to each image in the image set.
  • the action video image collection
  • the position of the key point and the change of the key point are extracted from the action data
  • the plane image of a series of actions is determined; according to the preset constraint conditions and the corresponding plane image, determine the original The action that matches the action.
  • Step S404 Determine the spatial position of the second bone key point according to the coordinate position of the second bone key point and the target curve.
  • the angle information can be a steering gear control signal set to restore the spatial position of the point (that is, a certain position on the robot structure, such as the wrist) in the video (for example, the position requires steering gear A, steering gear B, and steering gear).
  • the steering gear control signal is the control signal of the three steering gears, such as the angle of rotation).
  • the steering gear at the corresponding position is controlled according to the control parameter so that the second bone key point appears at the target spatial position.
  • the embodiment of the present application uses the two-dimensional plane coordinate data of the key points of the human skeleton to extract the direction and the ratio of the current amplitude of the projection in this direction to the maximum amplitude, describe the characteristics of multiple interrelated points, and recognize human movements; at the same time,
  • the action of the robot can be divided in the same way to form multiple spatial tracks, and then the posture of the human body is reversed back to the action of the robot, and the transformation from the human action to the robot action is finally completed.
  • the movements of the robot are directly obtained from the movements of the human body; because the two-dimensional plane coordinates cannot express the front and rear movements of the human body, it can be adjusted if necessary.
  • the obtained robot motion is fine-tuned to compensate for the missing part of the information, and finally the robot animation is output.
  • the robot animation is output.
  • it has improved production efficiency, lowered costs, and lowered technical thresholds. Realize the direct conversion of human actions into robot actions. This technology is applied to robots, which can imitate humans to do the same actions.
  • FIG. 8 is a schematic diagram of another action restoration method based on bone key points provided by an embodiment of the present application.
  • the action restoration method based on bone key points can be applied to an action restoration system based on bone key points (including the above System architecture), and is suitable for the application scenario shown in Figure 1 above.
  • the description will be made from a single side of the terminal with reference to FIG. 8.
  • the method may include the following steps S801 to S807; optional steps may include step S801, step S802, and step S807.
  • Step S801 Obtain image information of the target object.
  • the action video of the target object can be captured by the camera device to obtain the video action of the target object, and then the video is extracted frame by frame; when the image information is a picture, it can be directly obtained from the picture. Extract the key points of the bones.
  • the embodiment of the present application does not limit the method of obtaining image information.
  • Step S802 Determine the first bone key point and the second bone key point of the target object based on the image information.
  • the image information when the image information is a video, first extract each frame of video picture from the video frame by frame, then map the video picture to a two-dimensional target plane, and then combine the preset algorithm to determine the first bone key point and Information about the key points of the second bone (such as two-dimensional coordinates).
  • the image information is a picture
  • the target object contained in the picture is directly mapped to the target plane by dimensionality reduction; and then the first bone key point and the second bone key point contained in the target plane are determined according to the recognition algorithm.
  • each frame of video image corresponds to a different action of a target object.
  • extract multiple bone key points of the object according to a preset recognition algorithm.
  • the multiple bone key points in the image ie, bone key points
  • are mapped to a two-dimensional plane ie, a target two-dimensional plane.
  • FIG. 9 is a schematic diagram of a method for mapping three-dimensional point coordinates provided by an embodiment of the present application; as shown in FIG.
  • the motion of the human body and the motion of the robot are motions in a three-dimensional space
  • the embodiment of the present application is based on
  • the camera captures human movements and generates coordinate data of key bone points on a two-dimensional plane. Projecting the movement of key parts of the limbs in a three-dimensional space onto a two-dimensional plane will lose one-dimensional information. Observing the change of the projection in the direction perpendicular to the projection plane, it is impossible to distinguish the change information in the direction perpendicular to the plane.
  • Figure 9 shows the two line segments AB and AC in the three-dimensional space, where point A (0, 0, 0), point B (5, 5, 5), point C (5, 5, -5), in the three-dimensional coordinate system Two line segments can be distinguished in. Please refer to FIG.
  • FIG. 10 is a schematic diagram of a two-dimensional mapping plane provided by an embodiment of the present application; as shown in FIG. 10, the projection of the key points of the human bones on the plane cannot be distinguished whether they are on the front side of the body or behind the body. side. For example, if the arm extends 45° to the front of the body, and the arm extends 45° to the back of the body, the projection on the plane is the same.
  • Step S803 Obtain the coordinate position of the first bone key point on the target plane.
  • each position on the curve corresponds to one or more steering gear control signals of the robot, and the curve includes all possible positions of the second bone key point.
  • the first bone key point and the second bone key point associated with the first bone key point on the same joint are determined from the plurality of bone key points.
  • two bone key points related to each other on a joint are determined from multiple bone key points of the target object; for example, a shoulder joint point and a wrist point.
  • FIG. 11 is a schematic diagram of multiple bone key points of a target object provided by an embodiment of the present application.
  • 14 human bone key points are extracted through image recognition.
  • a two-dimensional plane coordinate system is established with a point on the plane as the origin, and (x, y) is used to represent each point. The meaning of each point is shown in Table 4.
  • Serial number meaning Serial number meaning 0 head 7 Right wrist 1 neck 8 Root of left leg 2 Left shoulder 9 Left knee
  • point 2 is the position of the shoulder.
  • Elbow point 3 and wrist point 4 move relative to point 2 as the center, and point 4 will also move around point 3 as the center.
  • the plane coordinate system can easily express the absolute position of the key points of the human bones, but for the interrelated points like the arm, it is not very intuitive to describe clearly. Therefore, for the relationship between the points on the plane, you can use the direction and distance. To describe. For example, point 2, point 3, and point 4 on the left arm, using 2 as the reference point, and using direction and distance to describe other points, then point 3 can be described as:
  • point 2 is the position of the shoulder.
  • Elbow point 3 and wrist point 4 move relative to point 2 as the center, and point 4 will also move around point 3 as the center.
  • the point coordinates of the first bone key point and the second bone key point are determined according to the first bone key point, the second bone key point, and the coordinate system.
  • the absolute position of the human bone key point can be easily expressed by the plane coordinate system, but it is not very intuitive for the interrelated points like the arm.
  • the description is clear, so the relationship between points on the plane can be described by direction and distance. For example, point 2, point 3, and point 4 on the left arm, using 2 as the reference point and using direction and distance to describe other points, then point 3 can be described as:
  • Point 4 can be described as:
  • point 4 can be described as:
  • the angle calculation method may include: selecting a unit vector on a coordinate axis, and calculating the angle through vector operations.
  • n is a unit vector on the coordinate axis.
  • the coordinates of the two points a and b are (x a , y a ), (x b , y b ).
  • each set of data select a number of points with a relatively fixed position of the human torso, and take the average length of the multiple lines between the points as the reference length value of this set of data; please refer to Figure 12, which is the reference length of this set of data.
  • the embodiment provides a schematic diagram of a point-to-point distance representation method; as shown in FIG. 12, each joint point has multiple points, and each point is described in combination with the content of Table 5; as shown in Table 5:
  • Table 5 lists the start and end points of the 6 line segments on the torso (the 6 dashed lines in the figure correspond to the 6 line segments).
  • the description method can be described by angle and relative distance.
  • the description method is as follows:
  • the point coordinates are two-dimensional coordinates
  • the coordinate system is a two-dimensional coordinate system
  • the multiple bone key points of each frame are mapped to the plane where the two-dimensional coordinate system is located in order of shooting time to obtain multiple frames of time-series video images; each frame of the multiple frames of time-series video images
  • the image corresponds to a pose of the target object.
  • the human body motion is mapped to the target plane.
  • the human body motion recognition is based on the following constraints: the coordinate data of the plane cannot distinguish the motion in the front and back direction of the arm, and the default motion direction is the front side of the body. Constraint 2.
  • the line segment 34 is affected by the line segment 23 and does not accurately express the spatial information.
  • points 2 and 4 are used to describe the arm directly, and the information of point 3 is ignored, that is, for the arm movement process, only the shoulders,
  • the position of the wrist can directly describe the position of the wrist with the shoulder as the reference point.
  • the two dimensions of the ratio of the direction and the amplitude can be extracted on the plane to describe:
  • Step S804 Determine one or more curves on the target plane with the coordinate position of the first bone key point as the center point.
  • the target curve is one of the one or more irregularly closed curves.
  • Step S805 Determine the target curve to which the second bone key point belongs according to the coordinate position of the second bone key point on the target plane.
  • Step S806 Determine the spatial position of the second bone key point according to the coordinate position of the second bone key point and the target curve.
  • FIG. 13 is a mapping relationship between the robot and the human motion provided by the embodiment of the present application.
  • human body movements can be mapped on a two-dimensional plane through coordinates, and nodes can be described according to the relative relationship between points.
  • the human body movement is restored.
  • the action in the three-dimensional space can also be mapped to a two-dimensional plane.
  • the two-dimensional plane lacks depth information, and it is impossible to distinguish whether the limbs are on the front side of the body or the back side of the body. In order to solve the problem 1, it is assumed that the movements of the limbs are all on the front side. After the movement is recorded, it can be adjusted through the fine-tuning function.
  • the range of motion of the robot is limited by hardware, and its degree of freedom is different from that of the human body. It cannot be as flexible as the human body. When the human motion projection is reversed to the robot motion, part of the motion cannot be fully displayed.
  • the mapping is done by the ratio of the amplitude, using the following formula:
  • the ratio of the amplitude is the ratio of the relative amplitude of the current action to the maximum relative amplitude that the action can reach.
  • the formula is as follows:
  • the maximum relative amplitude that can be achieved at a certain angle is:
  • the entity is parallel to the projection plane, standing upright, and the maximum relative amplitude data of the arm extending in a certain direction.
  • Step S807 According to the sequence of each frame of video pictures, sequentially determine the spatial position of the second bone key point in each frame of video picture to generate a motion track of the second bone key point.
  • the video containing the target object is decomposed into several frames of pictures.
  • pictures with the same posture or action can be eliminated, and the action of the target object in each frame of the left picture is different.
  • determine the spatial position of the second bone key point in each frame of the picture for example, obtain the steering gear control parameters given in order to restore the spatial position.
  • the spatial positions of the second bone key points on each picture are restored one by one, so that the robot makes a continuous and continuous action.
  • the position information of the first bone key point in the multiple frames of time-series video images remains unchanged.
  • FIG. 14 is a schematic structural diagram of an apparatus for restoring actions based on bone key points according to an embodiment of the present application, which may include a coordinate acquisition unit 1401, a curve preset unit 1402, a target curve determination unit 1403, and a spatial position determination unit.
  • Unit 1404 image information acquisition unit 1405, bone key point unit 1406, multiple bone key point unit 1407, target bone key point unit 1408, and trajectory determination unit 1409.
  • the optional units further include an image information acquisition unit 1405, a bone key point unit 1406, a multiple bone key point unit 1407, a target bone key point unit 1408, and a trajectory determination unit 1409.
  • the coordinate acquiring unit 1401 is configured to acquire the coordinate position of the first bone key point on a target plane; the target plane is used to indicate the relative position relationship of the bone key point of the target object from the three-dimensional space to the two-dimensional plane;
  • the curve preset unit 1402 is used to determine one or more curves on the target plane with the coordinate position of the first bone key point as the center point, and the curve is used to indicate that the second bone key point is mapped to All possible positions on the target plane; the coordinate position of each point on the curve corresponds to one or more angle information of the second bone key point in the three-dimensional space;
  • the target curve determining unit 1403 is configured to determine the target curve to which the second bone key point belongs according to the coordinate position of the second bone key point on the target plane, and the target curve is the one or more One of the curves
  • the spatial position determining unit 1404 is configured to determine the spatial position of the second bone key point according to the coordinate position of the second bone key point and the target curve.
  • the device further includes an image information acquisition unit 1405 and a bone key point unit 1406;
  • the image information acquisition unit 1405 is configured to acquire the first bone key point in the target plane Before the coordinate position on the above, obtain the image information of the target object;
  • the bone key point unit 1406 is used to determine the first bone key point and the second bone key point of the target object based on the image information, the The first bone key point and the second bone key point are adjacent bone key points of the target object.
  • the image information is a video; the first bone key point and the second bone key point are adjacent bone key points on the same joint in the same frame of video; the device further It includes a multi-skeletal keypoint unit 1407 and a target bone keypoint unit 1408; the multi-skeletal keypoint unit is used to determine multiple bone keypoints of the target object in the same frame of video; the target bone keypoint The unit is used to determine the first bone key point and the second bone key point adjacent to the first bone key point on the same joint from the plurality of bone key points.
  • the multi-skeletal keypoint unit 1407 is specifically configured to: extract each frame of video picture from the video of the target object, and obtain the target plane corresponding to each frame of video picture ; According to a preset bone key point recognition algorithm, determine multiple bone key points of the target object in the target plane corresponding to each frame of the video picture.
  • the device further includes a trajectory determining unit 1409, configured to: after determining the spatial position of the second bone key point, according to the sequence of each frame of video picture, The spatial position of the second bone key point in each frame of the video picture is sequentially determined to generate a motion track of the second bone key point.
  • a trajectory determining unit 1409 configured to: after determining the spatial position of the second bone key point, according to the sequence of each frame of video picture, The spatial position of the second bone key point in each frame of the video picture is sequentially determined to generate a motion track of the second bone key point.
  • the target curve determining unit 1403 is specifically configured to: establish a two-dimensional plane coordinate system on the target plane, and determine the second bone key point on the target plane. Dimensional coordinates; according to the two-dimensional coordinates, determine the target curve to which the second bone key point belongs.
  • the relative distance is the actual distance on the image information between the first bone key point and the second bone key point, and the actual distance between the first bone key point and the first bone key point and The ratio of the maximum distance between the second bone key points that can be presented on the image information.
  • bone key point-based motion restoration device described in the embodiment of the present application can refer to the description of the related method of bone key point-based motion restoration in the aforementioned device embodiment, and will not be repeated here.
  • the embodiments of the present application provide an electronic device, the device for restoring motion based on bone key points as described in the first aspect, and a discrete device coupled to the outside of the device for restoring motion based on bone key points.
  • An embodiment of the present application provides a terminal, the terminal includes a processor, and the processor is configured to support the terminal to perform a corresponding function in the method for restoring an action based on bone key points provided in the first aspect.
  • the terminal may also include a memory, which is used for coupling with the processor and stores necessary program instructions and data for the terminal.
  • the terminal may also include a communication interface for the terminal to communicate with other devices or communication networks.
  • FIG. 15 is a schematic structural diagram of a device for motion restoration based on bone key points provided by an embodiment of the application, as shown in FIG. 15
  • the device 14 for restoring actions based on bone key points can be implemented with the structure of FIG.
  • the device may also include general components such as an antenna and a power supply, which are not described in detail here.
  • the storage component 1501 may include one or more storage units, and each unit may include one or more memories.
  • the storage component can be used to store programs and various data, and can complete the programs or data at high speed and automatically during the operation of the general equipment. Access.
  • a physical device with two stable states can be used to store information, and the two stable states are represented as "0" and "1", respectively.
  • the aforementioned storage component may be read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), or other types that can store information and instructions.
  • the type of dynamic storage device can also be electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), CD-ROM (Compact Disc Read-Only Memory, CD-ROM), or other optical disk storage, optical disc Storage (which can include compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store desired program codes in the form of instructions or data structures and can Any other medium accessed by the computer, but not limited to this.
  • the memory can exist independently and is connected to the processor through a bus.
  • the memory can also be integrated with the processor.
  • the processing component 1502 may also be referred to as a processor, a processing unit, a processing board, a processing module, a processing device, and so on.
  • the processing component can be a central processing unit (CPU), a network processor (NP), or a combination of CPU and NP, or a microprocessor, application-specific integrated circuit (ASIC) ), or one or more integrated circuits used to control the execution of the program above.
  • CPU central processing unit
  • NP network processor
  • ASIC application-specific integrated circuit
  • the communication component 1503 which may also be called a transceiver, or a transceiver, may be used to communicate with other devices or a communication network, and may include a unit for wireless, wired, or other communication methods.
  • the processing component 1502 is used to call the data of the storage component 1501 to perform the following operations: obtain the first bone key point on the target plane Coordinate position; the target plane is used to indicate the relative position relationship of the bone key points of the target object from the three-dimensional space to the two-dimensional plane; determine the coordinate position of the first bone key point on the target plane as the center point One or more curves, the curve is used to indicate that the second bone key point is mapped to all possible positions on the target plane; the coordinate position of each point on the curve corresponds to the second bone key One or more angle information of a point in a three-dimensional space; according to the coordinate position of the second bone key point on the target plane, the target curve to which the second bone key point belongs is determined, and the target curve is the target curve One of the one or more curves; determining the spatial position of the second bone key point according to the coordinate position of the second bone key point and the target curve.
  • FIG. 16 is a schematic structural diagram of a device provided by an embodiment of the present application.
  • the device 16 may include a processor 1601 and a memory 1602; processing The device 1601 is used to support the device to execute the corresponding function of any one of the foregoing method embodiments; the memory 1602 is used to store the program instructions and data of the device.
  • the device is a 16-bit chip system
  • the chip system executes the method described in any one of the foregoing method embodiments; the chip system may also include other external discrete devices.
  • the apparatus 16 is a type of terminal equipment, reference may be made to the related description of the equipment described in FIG. 15, which will not be repeated here.
  • the disclosed device may be implemented in other ways.
  • the device embodiments described above are only illustrative, for example, the division of the above-mentioned units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or integrated. To another system, or some features can be ignored, or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical or other forms.
  • the units described above as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • the functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.
  • the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of the present application essentially or the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , Including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc., specifically a processor in a computer device) execute all or part of the steps of the foregoing methods of the various embodiments of the present application.
  • the aforementioned storage media may include: U disk, mobile hard disk, magnetic disk, optical disk, read-only memory (Read-Only Memory, abbreviation: ROM) or Random Access Memory (Random Access Memory, abbreviation: RAM), etc.
  • U disk mobile hard disk
  • magnetic disk magnetic disk
  • optical disk read-only memory
  • Read-Only Memory abbreviation: ROM
  • Random Access Memory Random Access Memory

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)
  • Manipulator (AREA)

Abstract

一种基于骨骼关键点的动作还原方法，可以应用于机械领域或者动作识别领域的机器人动作生成，该方法可以包括获取三维空间中目标对象映射在目标平面上第一骨骼关键点的坐标位置(S401)；确定该目标平面上以第一骨骼关键点为中心点的一个或多个曲线，该曲线上的每一个点的坐标位置对应第二骨骼关键点在三维空间中的一个或多个角度信息(S402)；根据映射到目标平面上第二骨骼关键点的坐标位置，确定第二骨骼关键点所属的目标曲线(S403)；根据第二骨骼关键点的坐标位置和目标曲线，确定第二骨骼关键点的空间位置(S404)。采用该方法有效地根据二维平面图像和空间角度信息，通过相关机器人还原目标对象的动作。

Description

一种基于骨骼关键点的动作还原方法以及装置
本申请要求于2020年02月29日提交中国专利局、申请号为202010132407.5、申请名称为“一种基于骨骼关键点的动作还原方法以及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及机器人动作领域，尤其涉及一种基于骨骼关键点的动作还原方法以及装置。
背景技术
当下机器人越来越多地走进普通人的生活和工作中，在家务、娱乐、教育等方向出现了各种各样形态的机器人产品；其中，外观模拟人或卡通形象的机器人，肢体的动作、头部的动作、面部的表情，需要根据预先编排好的机器动作，以控制机器人完成特定的行动，从而在人机交互过程中，带来较好的用户体验。不同机器人的动画制作的过程有所不同，会依赖于机器人的硬件形态、结构，因此机器人的自由度越多会越复杂。
目前，现有机器人动画制作方案一般通过一段目标对象的动作视频(例如人体四肢的动作)，获取一系列动作对应的多帧二维平面图像；根据多张二维平面图像获取每一帧图像中人体的姿态；根据预设的约束条件和算法，将每一帧人体姿态有时序地在相应的机器人上还原，以达到生成一系列对应的机器人动作。为了能够尽可能地还原二维图像对应的三维动作，一般会在图像或视频中运用计算机视觉技术，确定目标对象的各个骨骼关键点的位置，然后根据关节点的位置变化情况来还原动作；但是，对于多自由度机器人的动作而言，从二维影像中进行动作的还原比较麻烦。例如，真人的肩关节是可以三向活动的，而机器人受到电机体积的制约可能只能实现较少方向的活动。并且，二维影像缺少了深度信息，难以仅仅根据二维坐标还原特定骨骼关键点(该特定骨骼关键点对应机器人相应的结构位置)在空间中的真实位置。因此造成动作的失真，更直接的降低了机器人动作的完成度。
因此，如何有效地根据二维图像还原三维空间里的机器人动作，成为亟待解决的问题。
发明内容
本申请实施例提供了一种基于骨骼关键点的动作还原装置以及方法，实现了根据二维平面图像和预设动作的约束条件，有效地还原三维空间里的相应动作。
第一方面，本申请实施例提供了一种基于骨骼关键点的动作还原方法，可以应用于机器人的动作生成，所述方法可包括：
获取所述第一骨骼关键点在目标平面上的坐标位置；所述目标平面用于指示目标对象的骨骼关键点从三维空间映射到二维平面的相对位置关系；
确定所述目标平面上以所述第一骨骼关键点的坐标位置为中心点的一个或多个曲线，所述曲线用于指示所述第二骨骼关键点映射到所述目标平面上的所有可能的位置；所述曲线上的每一个点的坐标位置对应所述第二骨骼关键点在三维空间中的一个或多个角度信息；
根据所述第二骨骼关键点在所述目标平面上的坐标位置，确定所述第二骨骼关键点所属的目标曲线，所述目标曲线为所述一个或多个曲线中的一个；
根据所述第二骨骼关键点的坐标位置和所述目标曲线，确定所述第二骨骼关键点的空间位置。
本申请实施例中，根据二维平面投影上的一系列动作获取运动参数，而反推出机器人动作，从而实现机器人手臂在空间中运动轨迹。首先，针对某一帧视频图像中的多个骨骼关键点，选择关联运动的骨骼关键点；例如，肩关节点和手腕点的骨骼关键点作为第一骨骼关键点和第二骨骼关键点。在该帧图像中，设立二维坐标系；例如，以肩关节为坐标原点，预设第二骨骼关键点的多个可能出现的位置。例如，在二维平面上，机器人手臂按最大幅度来运动360°，手腕位置的动作轨迹是一个圆，圆心为对应手臂肩膀(即肩关节点)的位置；由于受到机器结构的影响，其二维形状可以是一个不规则的闭合曲线。那么按照最小幅度到最大幅度之间，机器人的手臂都运动360°，可以得到多组封闭曲线，即以第一骨骼关键点为中心点的一个或多个不规则封闭的曲线。曲线上每一个点都可能是手腕点的位置。然后根据第一骨骼关键点和第二骨骼关键点的坐标，确定在该帧图像上两个骨骼关键点的位置，以及第二骨骼关键点(即肩关节点)在目标曲线的目标位置。每个曲线上的点都对应一个或多个机器人相应结构位置上舵机旋转的角度数值(即舵机控制信号的一种)。在确定了第二骨骼关键点的坐标之后，结合该点对应的舵机控制信号，可以还原该帧图像中手腕点在空间中的位置。因此，在准确确定了点二维坐标后，根据与该点对应的舵机控制信号(例如手腕在身前与二维平面之间的深度数据)可以有效还原其在空间中的位置。
在一种可能的实现方式中，所述获取所述第一骨骼关键点在目标平面上的坐标位置之前，还包括：获取所述目标对象的图像信息；基于所述图像信息确定所述目标对象的第一骨骼关键点和第二骨骼关键点，所述第一骨骼关键点和所述第二骨骼关键点为所述目标对象的相邻骨骼关键点。本申请实施例，首先获取包含目标对象的图像信息，然后从图像信息中确定目标对象的多个骨骼关键点中的第一骨骼关键点和第二骨骼关键点。例如，图像信息为一张图像，从图像中根据预设的骨骼关键点识别算法将第一骨骼关键点和第二骨骼关键点提取出来，并确定两者之间的关系。
在一种可能的实现方式中，所述图像信息为视频；所述第一骨骼关键点和所述第二骨骼关键点为同一帧视频中同一关节上相邻的骨骼关键点；所述基于所述图像信息确定所述目标对象的第一骨骼关键点和第二骨骼关键点，包括：在所述同一帧视频中确定所述目标对象的多个骨骼关键点；从所述多个骨骼关键点中确定所述第一骨骼关键点，以及与所述第一骨骼关键点在所述同一个关节上相邻的所述第二骨骼关键点。本申请实施例中，通过在确定的多个骨骼关键点中，找到相互关联的一对骨骼关键点；并根据目标对象的身体结构以及预设的骨骼关键点的提取位置，从多个骨骼关键点中确定出所需要第一骨骼关键点和第二骨骼关键点的相关位置。其中，多个骨骼关键点中位置相对固定的骨骼关键点在每一帧视频图像中位置变化较小。
在一种可能的实现方式中，所述在所述同一帧视频中确定所述目标对象的多个骨骼关键点，包括：从所述目标对象的视频中提取每一帧视频图片，并获取所述每一帧视频图片 对应的目标平面；根据预设的骨骼关键点识别算法，确定所述每一帧视频图片对应的目标平面中所述目标对象的多个骨骼关键点。本申请实施例，对目标对象进行拍摄(例如人体)得到视频；然后根据从视频中提取多帧视频图像；获取对应的有时间顺序的多张视频图像，将每一张视频图像映射到二维目标平面上，以便于根据图像动作还原视频动作的数据处理。
在一种可能的实现方式中，所述确定所述第二骨骼关键点的空间位置之后，还包括：根据所述每一帧视频图片的排列顺序，依次确定所述每一帧视频图片中所述第二骨骼关键点的空间位置，以生成所述第二骨骼关键点的运动轨迹。本申请实施例中，在确定了每一帧视频图片中第二骨骼关键点的空间位置后，根据第二骨骼关键点在各个连续视频图像中的空间位置还原成连贯的动作，以确定第二骨骼关键点的运动路径。
在一种可能的实现方式中，所述根据所述第二骨骼关键点在所述目标平面上的坐标位置，确定所述第二骨骼关键点所属的目标曲线，包括：在所述目标平面上建立二维平面坐标系，确定所述第二骨骼关键点在所述目标平面上的二维坐标；根据所述二维坐标，确定所述第二骨骼关键点所属的所述目标曲线。本申请实施例中，通过第一骨骼关键点和第二骨骼关键点的二维坐标，确定了准确的第二骨骼关键点在曲线上的目标位置，提高了针对该图像还原动作的准确度。
在一种可能的实现方式中，通过相对距离表示所述第一骨骼点和第二骨骼点的距离关系；所述相对距离为所述第一骨骼关键点和所述第二骨骼关键点之间在所述图像信息上的实际距离，与所述第一骨骼关键点和所述第二骨骼关键点之间在所述图像信息上能够呈现的最大距离之比。本申请实施例，提供了一种表示距离的方法；即以相对距离来表示点与点的距离，能够减小点之间距离的误差。
第二方面，本申请实施例提供了一种基于骨骼关键点的动作还原装置，可以应用于机器人的动作生成，所述装置可包括：
坐标获取单元，用于获取所述第一骨骼关键点在目标平面上的坐标位置；所述目标平面用于指示目标对象的骨骼关键点从三维空间映射到二维平面的相对位置关系；
曲线预设单元，用于确定所述目标平面上以所述第一骨骼关键点的坐标位置为中心点的一个或多个曲线，所述曲线用于指示所述第二骨骼关键点映射到所述目标平面上的所有可能的位置；所述曲线上的每一个点的坐标位置对应所述第二骨骼关键点在三维空间中的一个或多个角度信息；
目标曲线确定单元，用于根据所述第二骨骼关键点在所述目标平面上的坐标位置，确定所述第二骨骼关键点所属的目标曲线，所述目标曲线为所述一个或多个曲线中的一个；
空间位置确定单元，用于根据所述第二骨骼关键点的坐标位置和所述目标曲线，确定所述第二骨骼关键点的空间位置。
在一种可能的实现方式中，所述装置还包括图像信息获取单元和骨骼关键点单元；所述图像信息获取单元，用于在所述获取所述第一骨骼关键点在目标平面上的坐标位置之前，获取所述目标对象的图像信息；所述骨骼关键点单元，用于基于所述图像信息确定所述目标对象的第一骨骼关键点和第二骨骼关键点，所述第一骨骼关键点和所述第二骨骼关键点为所述目标对象的相邻骨骼关键点。
在一种可能的实现方式中，所述图像信息为视频；所述第一骨骼关键点和所述第二骨骼关键点为同一帧视频中同一关节上相邻的骨骼关键点；所述装置还包括多骨骼关键点单元和目标骨骼关键点单元；所述多骨骼关键点单元，用于在所述同一帧视频中确定所述目标对象的多个骨骼关键点；所述目标骨骼关键点单元，用于从所述多个骨骼关键点中确定所述第一骨骼关键点，以及与所述第一骨骼关键点在所述同一个关节上相邻的所述第二骨骼关键点。
在一种可能的实现方式中，所述多骨骼关键点单元，具体用于：从所述目标对象的视频中提取每一帧视频图片，并获取所述每一帧视频图片对应的目标平面；根据预设的骨骼关键点识别算法，确定所述每一帧视频图片对应的目标平面中所述目标对象的多个骨骼关键点。
在一种可能的实现方式中，所述装置还包括轨迹确定单元，用于：在所述确定所述第二骨骼关键点的空间位置之后，根据所述每一帧视频图片的排列顺序，依次确定所述每一帧视频图片中所述第二骨骼关键点的空间位置，以生成所述第二骨骼关键点的运动轨迹。
在一种可能的实现方式中，所述目标曲线确定单元，具体用于：根在所述目标平面上建立二维平面坐标系，确定所述第二骨骼关键点在所述目标平面上的二维坐标；根据所述二维坐标，确定所述第二骨骼关键点所属的所述目标曲线。
在一种可能的实现方式中，所述装置还包括距离单元，用于通过相对距离表示所述第一骨骼点和第二骨骼点的距离关系；所述相对距离为所述第一骨骼关键点和所述第二骨骼关键点之间在所述图像信息上的实际距离，与所述第一骨骼关键点和所述第二骨骼关键点之间在所述图像信息上能够呈现的最大距离之比。
第三方面，本申请实施例提供了一种计算机可读存储介质，其特征在于，所述计算机存储介质存储有计算机程序，所述计算机程序包括程序指令，所述程序指令当被处理器执行时使所述处理器执行第一方面所述的方法。
第四方面，本申请实施例提供了一种包含指令的计算机程序产品，当其在处理器上运行时，使得处理器执行上述第一方面描述的方法。
第五方面，本申请实施例提供了一种电子设备，可包括：如上述第一方面所述的基于骨骼关键点的动作还原的装置，以及耦合于基于骨骼关键点的动作还原的装置外部的分立器件。
第六方面，本申请实施例提供一种终端，该终端包括处理器，处理器被配置为支持该终端执行第一方面提供的一种基于骨骼关键点的动作还原的方法中相应的功能。该终端还可以包括存储器，存储器用于与处理器耦合，其保存终端必要的程序指令和数据。该终端还可以包括通信接口，用于该终端与其它设备或通信网络通信。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍。
图1是本申请实施例提供的一种机器人动作生成的应用场景的示意图；
图2是本申请实施例提供的一种基于骨骼关键点的动作还原方法对应的系统架构示意图；
图3是本申请实施例提供的一种机器人动作生成的流程示意图；
图4是本申请实施例提供的一种基于骨骼关键点的动作还原方法示意图；
图5是本申请实施例提供的一种不规则的封闭曲线正面示意图；
图6是本申请实施例提供的图5所示曲线的侧面示意图；
图7是本申请实施例提供的一种第二骨骼关键点的运动轨迹图；
图8是本申请实施例提供的另一种基于骨骼关键点的动作还原方法示意图；
图9是是本申请实施例提供的一种三维点坐标的映射方法示意图；
图10是本申请实施例提供的一种二维映射平面示意图；
图11是本申请实施例提供的一种目标对象的多个骨骼关键点的示意图；
图12是本申请实施例提供的一种点与点的距离表示方法示意图；
图13是本申请实施例提供的一种机器人与人体动作之间的映射关系；
图14是本申请实施例提供的一种基于骨骼关键点的动作还原装置的结构示意图；
图15是本申请实施例提供的一种基于骨骼关键点的动作还原设备的结构示意图；
图16是本申请实施例提供的一种装置结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例进行描述。
本申请的说明书和权利要求书及所述附图中的术语“第一”、“第二”、“第三”和“第四”等是用于区别不同对象，而不是用于描述特定顺序。此外，术语“包括”和“具有”以及它们任何变形，意图在于覆盖不排他的包含。例如包含了一系列步骤或单元的过程、方法、系统、产品或设备没有限定于已列出的步骤或单元，而是可选地还包括没有列出的步骤或单元，或可选地还包括对于这些过程、方法、产品或设备固有的其它步骤或单元。
在本文中提及“实施例”意味着，结合实施例描述的特定特征、结构或特性可以包含在本申请的至少一个实施例中。在说明书中的各个位置出现该短语并不一定均是指相同的实施例，也不是与其它实施例互斥的独立的或备选的实施例。本领域技术人员显式地和隐式地理解的是，本文所描述的实施例可以与其它实施例相结合。
在本说明书中使用的术语“部件”、“模块”、“系统”等用于表示计算机相关的实体、硬件、固件、硬件和软件的组合、软件、或执行中的软件。例如，部件可以是但不限于，在处理器上运行的进程、处理器、对象、可执行文件、执行线程、程序和/或计算机。通过图示，在计算设备上运行的应用和计算设备都可以是部件。一个或多个部件可驻留在进程和/或执行线程中，部件可位于一个计算机上和/或分布在2个或更多个计算机之间。此外，这些部件可从在上面存储有各种数据结构的各种计算机可读介质执行。部件可例如根据具有一个或多个数据分组(例如来自与本地系统、分布式系统和/或网络间的另一部件交互的 二个部件的数据，例如通过信号与其它系统交互的互联网)的信号通过本地和/或远程进程来通信。
首先，对本申请中的部分用语进行解释说明，以便于本领域技术人员理解。
(1)自由度，根据机械原理，机构具有确定运动时所必须给定的独立运动参数的数目(亦即为了使机构的位置得以确定，必须给定的独立的广义坐标的数目)，称为机构自由度(degree of freedom of mechanism)，其数目常以F表示。
(2)骨骼关键点数据是一种用关键点去描述人体的动作的数据。
(3)软件开发工具包(Software Development Kit，SDK)，一般都是一些软件工程师为特定的软件包、软件框架、硬件平台、操作系统等建立应用软件时的开发工具的集合。软件开发工具包括广义上指辅助开发某一类软件的相关文档、范例和工具的集合。软件开发工具包是一些被软件工程师用于为特定的软件包、软件框架、硬件平台、操作系统等创建应用软件的开发工具的集合，一般而言SDK即开发Windows平台下的应用程序所使用的SDK。
(4)图像识别，是指利用计算机对图像进行处理、分析和理解，以识别各种不同模式的目标和对象的技术，是应用深度学习算法的一种实践应用。目前图像识别技术一般分为人脸识别与商品识别，人脸识别主要运用在安全检查、身份核验与移动支付中；商品识别主要运用在商品流通过程中，特别是无人货架、智能零售柜等无人零售领域。图像的传统识别流程分为四个步骤：图像采集→图像预处理→特征提取→图像识别。
(5)人工智能(Artificial Intelligence，AI)，是研究、开发用于模拟、延伸和扩展人的智能的理论、方法、技术及应用系统的一门新的技术科学。生产出一种新的能以人类智能相似的方式做出反应的智能机器，该领域的研究包括机器人、语言识别、图像识别、自然语言处理和专家系统等。
为了便于理解本申请实施例，以下示例性列举本申请中基于骨骼关键点的动作还原方法所应用的场景，可以包括如下机器人动作生成的应用场景。
请参见图1，图1是本申请实施例提供的一种机器人动作生成的应用场景的示意图，该应用场景中包括摄像设备(本申请实施例以摄像头为例)、目标对象、终端和机器人。目标对象可以是人、动物或者其他非生物(例如机器人或者机械结构)，本申请实施例对此不做限定。其中，摄像头，用于拍摄目标对象在一段时间内的动作变化情况；例如，当目标对象为人体时，拍摄十分钟内人体的四肢的运动情况。再例如，当目标对象为动物时，拍摄半个小时内动物的奔跑情况等。可选地，摄像头可以结合其他硬件设备，对拍摄的数据进行一定程度预先处理，比如去除目标对象没有动作的视频内容，以提高后续视频处理的效率。
终端，用于接收摄像头发送的视频数据；将视频数据进行进一步地处理，然后根据视频中的人体动作获取的对应机器人的各个运动参数传给对应的机器人。例如，将接收的每一帧视频图像映射到二维平面上；通过二维坐标系确定每一帧图像中人体的动作；再根据人体动作的坐标和预设的约束条件，在对应的机器人上还原出一系列的动作。可选地，本 申请实施例中还可以通过服务器还完成前述的数据处理、接收和发送的过程。本申请实施例对此不作限定。
机器人，用于接收终端发送的各项控制参数，在一定的时间顺序下依次根据各项参数完成每一帧图像的姿势，以最终还原目标对象的动作视频中的一系列动作。
具体地，在该应用场景中，具体可以包括人体(即目标对象)、摄像头、运行算法程序(例如摄像头驱动程序、AI图像处理程序、人体动作识别程序、机器人动作转换程序等等)的设备、智能机器人。摄像头采集人体动作，数据经过图像处理提取出人体骨骼关键点信息，一般包括肩膀、手肘、手腕、脖子、头、大腿根部、膝盖、脚等关键部位，输出为二维平面上的骨骼关键点坐标，通过人体动作识别程序的处理提取出人体的动作，结合不同机器人硬件的特点，将人体动作转化为机器人动作，最终实现人体动作转换为机器人动作。录入的人体动作，经过系统的处理后，生成动画文件应用到机器人上。
可以理解的是，图1所示应用场景的只是本申请实施例中的一种示例性的实施方式，本申请实施例中的应用场景包括但不仅限于以上应用场景。
结合上述应用场景，下面先对本申请实施例所基于的其中一种系统架构进行描述。请参见图2，图2是本申请实施例提供的一种基于骨骼关键点的动作还原方法对应的系统架构示意图，本申请提出的基于骨骼关键点的动作还原方法可以应用于该系统架构。如图2所示，该系统架构中包含了摄像头驱动模块、AI图像处理模块、人体动作识别模块、机器人动作转换模块和动作微调模块。其中，
摄像头驱动模块用于适配摄像头硬件，使用摄像头的SDK完成图像数据的处理，实现人体动作采集。AI图像处理模块用于基于机器学习的方法完成图像处理，通过对模型的训练和调优，实现人的面部表情、躯干和四肢甚至手指的识别和跟踪；本申请实施例中只使用了对人体躯干的数据，即通过14个关键点来描述。本申请实施例对动作的还原不限定于四肢等人体躯干，还可以包括面部表情等等。
人体动作识别模块用于根据平面二维坐标数据，用特定的方法描述出人体的动作，结合多张图识别人体姿态变化，应用滤波算法、关键帧提取算法，获取到相对连贯的动作序列。
机器人动作转换模块用于按机器人的硬件结构、自由度的特点，由动作数据转化为机器人的动作。
动作微调模块用于在完成人体动作录入后，可以得到机器人的动作序列，可支持对动作进行调整，在部分自由度未满足使用者期望、或者想再对动作进一步修饰时，动作微调模块提供了友好地支持。本申请实施例可以按人体的动作，快速完成机器人动作的生成，大大缩短动画制作的时间、降低了使用者的操作难度、提升动画制作的效率，由于是直接录制的人体动作，协调、流畅、自然程度上与人体动作一致。
请参见图3，图3是本申请实施例提供的一种机器人动作生成的流程示意图；如图3所示，通过使用摄像头拍摄人体动作，利用AI图像处理能力，从图像中提取出人体骨骼关键点信息，使用二维平面的坐标描述骨骼关键点；在人体姿势不断变化的过程中，部分关键点之间存在相互联系，为了更好的体现出这些特征，使用方向、幅度来描述这些关联关 系，可以实现对人体动作的识别；结合机器人的硬件结构、自由度特点，让使机器人做出相同的动作。
为了控制机器人的动作，将机器人各肢体的可运动范围，按方向、幅度两个维护进行划分，可以得到多个闭环的轨道，这些轨道在空间依次分布，识别到的人体动作与这些轨道会产生交点，空间中经过所有交点的曲线即为机器人肢体的运动轨迹。由于二维空间的信息缺失，以及用户可能对期望的动作进行调整，动作微调模块提供微调能力，最终的生成动作文件而在机器人上使用。
可以理解的是，图2中的系统架构只是本申请实施例中的一种示例性的实施方式，本申请实施例中的系统架构包括但不仅限于以上系统架构。
下面结合上述应用场景、系统架构和本申请中提供的基于骨骼关键点的动作还原装置的实施例，对本申请中提出的技术问题进行具体分析和解决。
请参见图4，图4是本申请实施例提供的一种基于骨骼关键点的动作还原方法示意图，该基于骨骼关键点的动作还原方法可以应用于基于骨骼关键点的动作还原系统(包括上述系统架构)，且适用于上述图1中所示的应用场景。下面将结合附图4，从终端的单侧进行描述。该方法可以包括以下步骤S401-步骤S404。
步骤S401：获取所述第一骨骼关键点在目标平面上的坐标位置。
具体地，在目标平面上有若干个骨骼关键点，根据目标对象在空间的姿势形态映射得到目标平面，并且根据识别算法提取该目标对象的多个骨骼关键点。选择一个骨骼关键点作为第一骨骼关键点(例如肩关节对应的骨骼关键点)，明确第一骨骼关键点在目标平面上的坐标位置(比如x坐标数据和y坐标的数据)。所述目标平面用于指示目标对象的骨骼关键点从三维空间映射到二维平面的相对位置关系。
步骤S402：确定所述目标平面上以所述第一骨骼关键点的坐标位置为中心点的一个或多个曲线。
具体地，根据确定的第一骨骼关键点为曲线的中心，预设一个或者多个曲线。其中，曲线可以是封闭或者半封闭的曲线；根据机器结构的限制和目标对象的身体结构，该曲线可以是规则或者不规则的曲线。所述曲线用于指示所述第二骨骼关键点映射到所述目标平面上的所有可能的位置；所述曲线上的每一个点的坐标位置对应所述第二骨骼关键点在三维空间中的一个或多个角度信息。
可选地，确定目标平面上以第一骨骼关键点为中心点的一个或多个不规则封闭的曲线。例如，在将三维视频图像映射到二维平面(即目标平面)后，在二维平面上确定一个或者多个不规则的封闭曲线。所述曲线上的每一个位置对应一个或多个所述机器人的舵机控制信号。所述曲线包括第二骨骼关键点所有可能的位置；其中，每一个曲线代表上的每一个点的位置都代表第二骨骼关键点(本申请实施例以手腕点为例进行说明)可能出现的位置。曲线的绘制可以根据预设的关节长度以及肩关节点的位置来确定一个或多个曲线。由于二维平面的图像转换为三维空间的需要预设一定的约束条件，可以参考以下的约束。
例如，基于二维平面提取出的有限特性，本申请实施例可以默认机器人的手臂在身体前侧活动。请参见图5，图5是本申请实施例提供的一种不规则的封闭曲线正面示意图； 从图5中机器人的正面看，比如，图中机器人的左臂以左肩为核心点，按最大幅度来运动360°；即手腕的动作轨迹是一个圆(圆心对应左手臂肩膀)，轨迹在平面上的形状是一个不规则的闭合曲线。可以理解的是，手腕的动作轨迹受到机器人结构的影响，呈现的不是一个圆形轨迹。其中，中心点为第一骨骼关键点(即左肩对应的骨骼关键点)，以中心点为核心的曲线为第二骨骼关键点(即手腕对应的骨骼关键点)所有可能出现位置的集合。具体地，由于受到机器结构的限制，在有些角度上不能达到最大的幅度。本申请实施例对曲线的具体形状不做限定。
请参见图6，图6是本申请实施例提供的图5所示曲线的侧面示意图；如图6所示的曲线与图5所示的曲线是同一个曲线不同角度的视图。可选地，通过由平面投影上的动作反推出机器人动作，将机器人手臂在身体前侧的运动轨迹按方向、幅度的比例值两个维度进行划分。一般曲线上每一个点的位置都对应该点的自由度(即该点的运动方向)。自由度可以将结合点的二维坐标将该点的位置在三维空间中还原视频中的相应动作。例如，手臂自由度一般为3-4个，下面以3个自由度为例。运动一周360°的角度，划分为N个角度分段；具体数据请参见表1，如下所示：
表1
  自由度1 自由度2 自由度3
Angle_1 Value_1_1 Value_1_2 Value_1_3
Angle_2 Value_2_1 Value_2_2 Value_2_3
Angle_3 Value_3_1 Value_3_2 Value_3_3
…… …… …… ……
Angle_N Value_N_1 Value_N_2 Value_N_3
其中，Angle_1表示角度数据；Value_1_1、Value_1_2以及Value_1_3表示在该角度下预设的三个自由度上的运动数据。
再例如，从机器人的正面看，将机器人手臂放置在最小幅度内，即手臂向身体正前方抬起，手掌位置的投影与肩膀重合，具体数据请参见表2；如下所示：
表2
  自由度1 自由度2 自由度3
Angle_1～Angle_N Value_1_1 Value_1_2 Value_1_3
可选地，从最小幅度到最大幅度，共划分为M个幅度，则其它有(M-2)张表，相应的数据请参见表3；表3所示为M张表中的其中一张表。
表3
  自由度1 自由度2 自由度3
Angle_1 Value_M_1_1 Value_M_1_2 Value_M_1_3
Angle_2 Value_M_2_1 Value_M_2_2 Value_M_2_3
Angle_3 Value_M_3_1 Value_M_3_2 Value_M_3_3
…… …… …… ……
Angle_N Value_M_N_1 Value_M_N_2 Value_M_N_3
最终会有M个表，每个表有N个角度的数据，当M、N数据增大时，可以得到几乎平滑的曲线，可以使用图形来描述出这些数据的特征；请参考图7，图7是本申请实施例提供的一种第二骨骼关键点的运动轨迹图；如图7所示，一圈圈以中心圆点为核心的不规则曲线为同一相对幅度下360°方向的运动曲线，按不同的精细程度划分，可以得到一组曲线。其中，有一条曲线与每一个曲线都存在交点，即动作数据的方向。例如，点1与y轴的夹角为1’，对应前述Angle_1；那么当机器人对应点1的位置有三个方向的舵机时，点1对应三个自由度以及相应的运动参数，使得机器人的相应结构能够移动到点1的位置上。
对于一组人体动作，迁移到机器人的运动曲线上，可在每一个方向上匹配一个相对幅度，从中心点出发的直线会与曲线相交的各个相交点，连接就是该第二骨骼关键点的在若干张图像中移动的位置变动情况。即穿过所有点的曲线，为机器人手臂的运动轨迹。通过滤波算法、动作微调等获取理想的最终位置运动数据。录制一段人体动作视频，经过本系统的处理，可得到机器人各自由度的控制数值，最终应用于机器人上。
在一种可能的实现方式中，所述第一骨骼关键点和所述第二骨骼关键点为同一帧视频图像中同一个关节上的两个关联的骨骼关键点。
在一种可能的实现方式中，根据所述第一骨骼关键点和第二骨骼关键点在所述目标平面上的二维坐标，确定所述第二骨骼关键点在目标曲线上的目标位置。
具体地，所述目标曲线为所述一个或多个不规则封闭的曲线中的一个。在二维平面上，根据设置的二维坐标来确定第一骨骼关键点和第二骨骼关键点的坐标。例如，将视频中的某一帧视频图像映射到二维目标平面上，提取目标对象的几个骨骼关键点。以肩关节点和手腕点为例，在目标平面上确定了两个点的位置，就可以得出两个点的坐标数据，即横坐标数据和纵坐标数据。
在一种可能的实现方式中，所述根据所述第一骨骼关键点和第二骨骼关键点在所述目标平面上的二维坐标，确定所述第二骨骼关键点在目标曲线上的目标位置，包括：
根据所述第一骨骼关键点和第二骨骼关键点在所述目标平面上的二维坐标，确定所述第一骨骼关键点和所述第二骨骼关键点之间的距离，以及所述第二骨骼关键点相对所述第一骨骼关键点的方向；
在所述第二骨骼关键点相对所述第一骨骼关键点的方向上，根据所述距离确定所述第 二骨骼关键点在所述目标曲线上的目标位置。
在一种可能的实现方式中，所述第一骨骼关键点和所述第二骨骼关键点之间的距离表示为所述第一骨骼关键点和所述第二骨骼关键点之间的实际距离和所述第一骨骼关键点和所述第二骨骼关键点之间的最大距离之比。
步骤S403：根据所述第二骨骼关键点在所述目标平面上的坐标位置，确定所述第二骨骼关键点所属的目标曲线。
具体地，在确定了第二骨骼关键点在目标平面上的坐标位置之后，根据该坐标数据(即坐标位置)确定该第二骨骼关键点落在预设的曲线中的某一个曲线上。可以理解的是，以第一骨骼关键点为中心的曲线一般为多组密集分布，所以第二骨骼关键点必定会落在其中的一个曲线上，该曲线就是目标曲线。所述目标曲线为所述一个或多个曲线中的一个。
在一种可能的实现方式中，根据所述目标位置对应的一个或多个所述机器人的舵机控制信号，确定所述第二骨骼关键点的空间位置。例如，确定了点所在的目标位置后，根据预设的目标位置对应的舵机控制信号，控制该位置对应舵机按照参数进行旋转或者运动，使得该点运动到指定的空间位置。例如，该目标位置对应2个自由度(往前的自由度和往右的自由度)，并且每个自由度上规定了舵机的旋转角度；那么控制往前的舵机旋转一定角度，然后控制往右的舵机旋转一定角度，最后使得该点达到了目标地。根据第二骨骼关键点的坐标数据确定了第二骨骼关键点在目标曲线的具体的目标位置后，继而可以从该点在该曲线的目标位置对应的自由度数据而还原出第二骨骼关键点的空间位置。相当于在已知了点的x轴和y轴坐标后，根据曲线上点的z轴坐标数据，确定了三维坐标数据；通过将控制参数传给机器人而将该点的位置复原。
在一种可能的实现方式中，根据所述第二骨骼关键点的空间位置，控制所述机器人对应结构的舵机运行；所述第二骨骼关键点在目标曲面上；所述目标曲面为所述一个或多个不规则封闭曲面中的一个；所述一个或多个不规则封闭曲面中每一个曲面上每一个位置对应一个或多个舵机控制信号。
在一种可能的实现方式中，将以第一骨骼关键点为中心点的一个或多个不规则封闭曲线，确定为第二骨骼关键点的一个或多个预设位置轨迹；所述第一骨骼关键点和所述第二骨骼关键点为同一帧视频图像中的两个关联的骨骼关键点；根据所述第一骨骼关键点和所述第二骨骼关键点的位置信息，计算所述第一骨骼关键点和所述第二骨骼关键点之间的距离，以及所述第二骨骼关键点相对所述第一骨骼关键点的方向；在所述第二骨骼关键点相对所述第一骨骼关键点的方向上，根据所述距离确定所述第二骨骼关键点在目标位置轨迹上的位置，所述目标位置轨迹为所述一个或多个预设位置轨迹中的一个轨迹。
在一种可能的实现方式中，所述位置信息为骨骼关键点的点坐标；所述方法还包括：在所述同一帧视频图像中确定目标对象的多个骨骼关键点；从所述多个骨骼关键点中确定所述第一骨骼关键点，以及与所述第一骨骼关键点关联的所述第二骨骼关键点；根据所述第一骨骼关键点、所述第二骨骼关键点和坐标系，确定所述第一骨骼关键点和所述第二骨骼关键点的点坐标。
在一种可能的实现方式中，所述第一骨骼关键点在所述多帧有时序的视频图像中的位置信息不变；所述方法还包括：根据所述多帧有时序的视频图像，依次确定所述第二骨骼 关键点在所述多帧有时序的视频图像中每一帧视频图像中的位置信息，生成所述第二骨骼关键点的运动序列。
可选地，根据所述第二骨骼关键点的运动序列，以及与运动序列中每一个姿态下第二骨骼关键点对应的舵机控制信号，在机器结构上还原所述第二骨骼关键点的位置变化。进一步可选地，以第一骨骼关键点为中心点的各个角度上，根据所述第一骨骼关键点和所述第二骨骼关键点之间可达到的多个距离值确定所述一个或多个不规则的封闭曲线。生成所述目标对象对应的动作序列，所述动作序列中每一个动作包括所述多个骨骼关键点的位置信息。所述动作视频为拍摄所述目标对象在一段时间内动作变化的视频；采集人体动作视频，提取其中每一帧人体骨骼关键点的位置信息，组成全身骨骼关键点动作序列。
可选地，获取目标对象的图像集合，所述图像集合包括所述目标对象的多个不同形态的图像；确定所述图像集合中每一张图像对应的所述目标对象的形态。其中，获取动作视频(图像集合)；从动作数据中提取关键点的位置以及关键点的变化情况，确定一系列的动作的平面图像；根据预设的约束条件和对应的平面图像，确定与原始动作匹配的动作。
步骤S404：根据所述第二骨骼关键点的坐标位置和所述目标曲线，确定所述第二骨骼关键点的空间位置。
具体地，在确定了第二骨骼关键点所处的目标曲线后，根据第二骨骼关键点在所述目标曲线上的坐标位置对应的点，获取预设的该点对应的一个或者多个角度信息。其中，角度信息可以是为了还原该点(即机器人结构上的某一个位置，比如手腕)在视频中的空间位置而设置的舵机控制信号(比如该位置需要舵机A，舵机B和舵机C，那么该舵机控制信号为三个舵机的控制信号，如旋转的角度)。可选地，在获取还原第二骨骼关键点空间位置的操控参数后，根据操控参数对相应位置上的舵机进行控制，使得第二骨骼关键点在目标空间位置上出现。
本申请实施例使用人体骨骼关键点的二维平面坐标数据，提取出方向、该方向上的投影当前幅度与最大幅度的比例，描述出多个互相关联的点的特性，识别人体动作；同时，将机器人的动作可按相同的方式划分，形成多个空间轨道，再由人体姿势反推回机器人的动作，最终完成人体动作到机器人动作的转换。通过提出了一种新的机器人动画制作方法，使用人工智能相关方法来处理图像，由人体的动作直接获取机器人的动作；由于二维平面坐标无法表达出人体肢体的前后动作，必要情况下可以对获得到的机器人动作进行微调，以补偿缺失的这部分信息，最终输出机器人动画。相比现在通过三维动画软件制作动画的过程，提升了制作效率、降低了成本、降低了技术门槛。实现人体动作直接转化为机器人动作，该技术应用在机器人上，机器人可模仿人做相同的动作。
请参见图8，图8是本申请实施例提供的另一种基于骨骼关键点的动作还原方法示意图，该基于骨骼关键点的动作还原方法可以应用于基于骨骼关键点的动作还原系统(包括上述系统架构)，且适用于上述图1中所示的应用场景。下面将结合附图8，从终端的单侧进行描述。该方法可以包括以下步骤S801-步骤S807；可选的步骤可以包括步骤S801、步骤S802和步骤S807。
步骤S801：获取所述目标对象的图像信息。
具体地，当图像信息为视频时，可以通过摄像设备拍摄目标对象的动作视频来获取所述目标对象的视频动作，再对视频进行逐帧地提取；当图像信息为图片时，可以直接从图片中提取骨骼关键点。本申请实施例对获取图像信息的方式不作限定。
步骤S802：基于所述图像信息确定所述目标对象的第一骨骼关键点和第二骨骼关键点。
具体地，当图像信息为视频时，首先从视频中逐帧地提取每一帧视频图片，然后在视频图片映射到二维目标平面上，再结合预设的算法，确定第一骨骼关键点和第二骨骼关键点的信息(比如二维坐标)。当图像信息为图片时，直接对将图片包含的目标对象进行降维映射，映射到目标平面上；再根据识别算法确定目标平面中包含的第一骨骼关键点和第二骨骼关键点。
例如，从视频数据中获得多帧的视频图像。其中，每一帧视频图像对应一个目标对象的不同动作。从某一帧图像中，根据预设的识别算法提取该对象的多个骨骼关键点。将图像中的多个骨骼关键点(即骨骼关键点)映射到二维平面(即目标二维平面)上。请参见图9，图9是本申请实施例提供的一种三维点坐标的映射方法示意图；如图9所示，人体的动作与机器人的动作是在三维空间中的运动，本申请实施例基于摄像头拍摄人体动作，并生成二维平面上骨骼关键点的坐标数据。将三维空间中肢体关键部位的运动，投影到二维平面上，会丢失一个维度的信息，在与投影平面垂直的方向观察投影的变化，无法区分出与平面垂直方向上的变化信息。图9中是三维空间的两个线段AB、AC，其中A点(0，0，0)，B点(5，5，5)，C点(5，5，-5)，在三维坐标系中可以区分出两条线段。请参见图10，图10是本申请实施例提供的一种二维映射平面示意图；如图10所示，人体骨骼关键点在平面上的投影，无法区分是在身体前身侧或是在身体后侧。例如，手臂向身体前侧伸出45°，与手臂向身体后侧伸出45°，在平面上的投影是相同的。
步骤S803：获取所述第一骨骼关键点在目标平面上的坐标位置。
具体地，请参见前述步骤S401；例如，所述曲线上的每一个位置对应一个或多个所述机器人的舵机控制信号，所述曲线包括第二骨骼关键点所有可能的位置。
例如，从所述多个骨骼关键点中确定所述第一骨骼关键点，以及与所述第一骨骼关键点在所述同一个关节上关联的所述第二骨骼关键点。具体地，从目标对象的多个骨骼关键点中确定一个关节上相互关联的两个骨骼关键点；例如，肩关节点和手腕点。请参见图11，图11是本申请实施例提供的一种目标对象的多个骨骼关键点的示意图，如图11所示，通过图像识别提取出的14个人体骨骼关键点。为了描述平面上点的，以平面上的一个点为原点建立二维平面坐标系，使用(x,y)来表示每一个点，各点含义如表4所示。
表4
序号 含义 序号 含义
0 头部 7 右手腕
1 颈部 8 左腿根部
2 左肩 9 左腿膝盖
3 左手肘 10 左腿
4 左手腕 11 右腿根部
5 右肩 12 右腿膝盖
6 右手肘 13 右脚
人体骨骼关键点间存在着特定的联系，如左臂的三个点2、3、4在运动过程中相互关联，其基本的特性为：点2为肩膀的位置，手臂动作变化的过程中，手肘点3、手腕点4相对点2为中心做动作，而点4还会以点3为中心做运动。用平面坐标系可以方便的表达出人体骨骼关键点的绝对位置，但对像手臂这样相互关联的点，无法很直观的描述清楚，因此，对于平面上点之间的关系，可以用方向、距离来描述。如左臂的点2、点3、点4三个点，以2为基准点，使用方向、距离来描述其它的点，则点3可以描述为：
Figure PCTCN2021076723-appb-000001
人体骨骼关键点间存在着特定的联系，如左臂的三个点2、3、4在运动过程中相互关联，其基本的特性为：点2为肩膀的位置，手臂动作变化的过程中，手肘点3、手腕点4相对点2为中心做动作，而点4还会以点3为中心做运动。
其中，根据所述第一骨骼关键点、所述第二骨骼关键点和坐标系，确定所述第一骨骼关键点和所述第二骨骼关键点的点坐标。例如，在已知第一骨骼关键点和第二骨骼关键点的位置，用平面坐标系可以方便的表达出人体骨骼关键点的绝对位置，但对像手臂这样相互关联的点，无法很直观的描述清楚，因此，对于平面上点之间的关系，可以用方向、距离来描述。如左臂的点2、点3、点4三个点，以2为基准点，使用方向、距离来描述其它的点，则点3可以描述为：
Figure PCTCN2021076723-appb-000002
点4可以描述为：
Figure PCTCN2021076723-appb-000003
以点3为基准点，则点4可以描述为：
Figure PCTCN2021076723-appb-000004
其中，角度与距离分别按如下方法进行运算：
Figure PCTCN2021076723-appb-000005
角度计算方法可以包括：选取一个坐标轴上的单位向量，通过向量运算计算角度。
其中，m为两个坐标点确定的向量，n为坐标轴上的单位向量。
距离计算方法：
Figure PCTCN2021076723-appb-000006
其中，a、b两个点坐标为(x a，y a)，(x b，y b)。
由于摄像头与被拍摄人体间的距离会变化，导致同一人体在不同时刻的投影大小会有变化，在多组投影数据间做处理时，使用点与点之前的绝对距离来描述距离Distance会导致不一致，为解决该问题，使用相对距离来描述距离Distance，具体方法如下：
在每一组数据中，选取人体躯干位置相对固定的若干个点，点与点之间的多个连线取 平均长度为本组数据的基准长度值；请参见图12，图12是本申请实施例提供的一种点与点的距离表示方法示意图；如图12所示，每个关节点有多个点，结合表5的内容对各个点进行描述；如表5所示：
表5
起始点 2 2 1 1 5 5
结束点 8 11 8 11 8 11
表5列出了躯干上的6条线段(图中的6条虚线对应6条线段)的起始点和结束点，使用平面上两点之前距离的计算方法，可以计算出单组数据的基准距离，如下：
Distance 基准＝(Distance1+..+Distance6)/6
则每组数据的14个点的相对距离的计算公式如下：
Figure PCTCN2021076723-appb-000007
综上，存在于平面坐标系中的若干个点，部分在运动过程中有相互关联的点，其描述方法可以使用角度、相对距离来描述，描述方式如下：
Figure PCTCN2021076723-appb-000008
在一种可能的实现方式中，所述点坐标为二维坐标，所述坐标系为二维坐标系；所述在所述同一帧视频图像中确定目标对象的多个骨骼关键点之前，还包括：
从包含所述目标对象的视频中提取每一帧所述目标对象的多个骨骼关键点；
将每一帧所述多个骨骼关键点按照拍摄时间顺序映射到所述二维坐标系所在的平面，获得多帧有时序的视频图像；所述多帧有时序的视频图像中每一帧视频图像对应所述目标对象的一个姿势。例如，将人体动作映射到目标平面，在该过程中人体动作识别基于以下约束：平面的坐标数据无法区分手臂的前后方向上的动作，默认运动方向是在身体前侧。约束2、线段34受线段23的影响，对空间信息表达不准确，因此动作提取上，直接使用点2、点4来描述手臂，忽略掉点3的信息，即对于手臂运动过程只关注肩膀、手腕的位置，可直接描述出以肩膀为基准点，手腕所处的位置。针对手臂的运动，在平面上可以提取出方向、幅度的比例值两个维度的特征来描述：
Figure PCTCN2021076723-appb-000009
步骤S804：确定所述目标平面上以所述第一骨骼关键点的坐标位置为中心点的一个或多个曲线。
具体地，请参见前述步骤S402；可选地，所述目标曲线为所述一个或多个不规则封闭的曲线中的一个。
步骤S805：根据所述第二骨骼关键点在所述目标平面上的坐标位置，确定所述第二骨骼关键点所属的目标曲线。
具体地，请参见前述步骤S403。
步骤S806：根据所述第二骨骼关键点的坐标位置和所述目标曲线，确定所述第二骨骼 关键点的空间位置。
具体地，请参见前述步骤S404。
在一种可能的实现方式中，无论针对实体是人体还是机器人，其平面投影都可以用这样的模型来描述；同时，对于任意一个平面投影，在满足一定约束条件的情况下，可以反推出实体的动作。由此，可以由人体的动作投影，反推出机器人的动作，其推导关系请参见图13，图13是本申请实施例提供的一种机器人与人体动作之间的映射关系。如图13所示，人体动作可以通过坐标映射在二维平面上，根据点之间的相对关系来描述节点。再者，根据点的相对关系以及机器人动作的平面投影还原人体动作。或者目标对象是机器人动作时，也可以将三维空间中的动作映射到二维平面。
例如，由投影反推回实体动作，但在部分场景下反推得到的动作有偏差，主要问题如下：
问题1、二维平面缺失深度信息，无法区分肢体在身体前侧还是在身体后侧。为了解决问题1，预设肢体的动作都是在前侧的动作，完成动作录制后，通过微调功能进行调整。
问题2、机器人的活动范围受硬件限制，其肢体自由度与人体也不同，无法做到像人体一样灵活，由人体动作投影反推回机器人动作时，部分动作无法完全展现。为了解决这个问题，在反推实体的动作时，根据实体肢体在某个方向上的实际运动范围，通过幅度的比例来做映射，使用如下公式：
Figure PCTCN2021076723-appb-000010
幅度的比例值，是当前动作的相对幅度与动作可达到的最大相对幅度的比，公式如下：
Figure PCTCN2021076723-appb-000011
按不同实体的实际情况，在某个角度上可达到的最大相对幅度为：
Distance relative_Max
对于手臂则是实体与投影平面平行，竖直站立，手臂在某个方向上伸展的最大相对幅度数据。
步骤S807：根据所述每一帧视频图片的排列顺序，依次确定所述每一帧视频图片中所述第二骨骼关键点的空间位置，以生成所述第二骨骼关键点的运动轨迹。
具体地，当图像信息为视频的时，将包含目标对象的视频进行分解成若干帧图片。其中可选地，相同姿势或者动作的图片可以进行剔除，留下的每一帧图片中目标对象的动作都是不同的。根据视频中动作的变化先后顺序，对每一帧图片进行排序。然后确定每一帧图片中第二骨骼关键点的空间位置(比如，获取为了还原空间位置而给出的舵机控制参数)。按照图片的排列顺序，逐个地还原每张图片上的第二骨骼关键点的空间位置，使得机器人做出连贯的持续动作。可选地，所述第一骨骼关键点在所述多帧有时序的视频图像中的位置信息不变。
上述详细阐述了本申请实施例的方法，下面提供了本申请实施例的相关装置。
请参见图14，图14是本申请实施例提供的一种基于骨骼关键点的动作还原装置的结构示意图，可以包括坐标获取单元1401、曲线预设单元1402、目标曲线确定单元1403、空间位置确定单元1404、图像信息获取单元1405、骨骼关键点单元1406、多骨骼关键点 单元1407、目标骨骼关键点单元1408和轨迹确定单元1409。其中，可选的单元还包括图像信息获取单元1405、骨骼关键点单元1406、多骨骼关键点单元1407、目标骨骼关键点单元1408和轨迹确定单元1409。
坐标获取单元1401，用于获取所述第一骨骼关键点在目标平面上的坐标位置；所述目标平面用于指示目标对象的骨骼关键点从三维空间映射到二维平面的相对位置关系；
曲线预设单元1402，用于确定所述目标平面上以所述第一骨骼关键点的坐标位置为中心点的一个或多个曲线，所述曲线用于指示所述第二骨骼关键点映射到所述目标平面上的所有可能的位置；所述曲线上的每一个点的坐标位置对应所述第二骨骼关键点在三维空间中的一个或多个角度信息；
目标曲线确定单元1403，用于根据所述第二骨骼关键点在所述目标平面上的坐标位置，确定所述第二骨骼关键点所属的目标曲线，所述目标曲线为所述一个或多个曲线中的一个；
空间位置确定单元1404，用于根据所述第二骨骼关键点的坐标位置和所述目标曲线，确定所述第二骨骼关键点的空间位置。
在一种可能的实现方式中，所述装置还包括图像信息获取单元1405和骨骼关键点单元1406；所述图像信息获取单元1405，用于在所述获取所述第一骨骼关键点在目标平面上的坐标位置之前，获取所述目标对象的图像信息；所述骨骼关键点单元1406，用于基于所述图像信息确定所述目标对象的第一骨骼关键点和第二骨骼关键点，所述第一骨骼关键点和所述第二骨骼关键点为所述目标对象的相邻骨骼关键点。
在一种可能的实现方式中，所述图像信息为视频；所述第一骨骼关键点和所述第二骨骼关键点为同一帧视频中同一关节上相邻的骨骼关键点；所述装置还包括多骨骼关键点单元1407和目标骨骼关键点单元1408；所述多骨骼关键点单元，用于在所述同一帧视频中确定所述目标对象的多个骨骼关键点；所述目标骨骼关键点单元，用于从所述多个骨骼关键点中确定所述第一骨骼关键点，以及与所述第一骨骼关键点在所述同一个关节上相邻的所述第二骨骼关键点。
在一种可能的实现方式中，所述多骨骼关键点单元1407，具体用于：从所述目标对象的视频中提取每一帧视频图片，并获取所述每一帧视频图片对应的目标平面；根据预设的骨骼关键点识别算法，确定所述每一帧视频图片对应的目标平面中所述目标对象的多个骨骼关键点。
在一种可能的实现方式中，所述装置还包括轨迹确定单元1409，用于：在所述确定所述第二骨骼关键点的空间位置之后，根据所述每一帧视频图片的排列顺序，依次确定所述每一帧视频图片中所述第二骨骼关键点的空间位置，以生成所述第二骨骼关键点的运动轨迹。
在一种可能的实现方式中，所述目标曲线确定单元1403，具体用于：在所述目标平面上建立二维平面坐标系，确定所述第二骨骼关键点在所述目标平面上的二维坐标；根据所述二维坐标，确定所述第二骨骼关键点所属的所述目标曲线。
在一种可能的实现方式中，所述相对距离为所述第一骨骼关键点和所述第二骨骼关键点之间在所述图像信息上的实际距离，与所述第一骨骼关键点和所述第二骨骼关键点之间在所述图像信息上能够呈现的最大距离之比。
需要说明的是，本申请实施例中所描述的基于骨骼关键点的动作还原装置可参见前述的装置实施例中的基于骨骼关键点的动作还原的相关方法描述，此处不再赘述。
本申请实施例提供了一种电子设备，如前述第一方面所述的基于骨骼关键点的动作还原的装置，以及耦合于基于骨骼关键点的动作还原的装置外部的分立器件。
本申请实施例提供一种终端，该终端包括处理器，处理器被配置为支持该终端执行第一方面提供的一种基于骨骼关键点的动作还原的方法中相应的功能。该终端还可以包括存储器，存储器用于与处理器耦合，其保存终端必要的程序指令和数据。该终端还可以包括通信接口，用于该终端与其它设备或通信网络通信。
本申请实施例提供了一种基于骨骼关键点的动作还原设备，请参见图15，图15是本申请实施例提供的一种基于骨骼关键点的动作还原设备的结构示意图，如图15所示，基于骨骼关键点的动作还原装置14能以图15的结构实现，基于骨骼关键点的动作还原设备15可以包括至少一个存储部件1501、至少一个处理部件1502、至少一个通信部件1503。此外，该设备还可以包括天线、电源等通用部件，在此不再详述。
存储部件1501可以包括一个或多个存储单元，每个单元可以包括一个或多个存储器，存储部件可用于存储程序和各种数据，并能在通用设备运行过程中高速、自动地完成程序或数据的存取。可以采用具有两种稳定状态的物理器件来存储信息，所述两种稳定状态分别表示为“0”和“1”。前述存储部件可以是只读存储器(read-only memory，ROM)或可存储静态信息和指令的其他类型的静态存储设备，随机存取存储器(random access memory，RAM)或者可存储信息和指令的其他类型的动态存储设备，也可以是电可擦可编程只读存储器(Electrically Erasable Programmable Read-Only Memory，EEPROM)、只读光盘(Compact Disc Read-Only Memory，CD-ROM)或其他光盘存储、光碟存储(可以包括压缩光碟、激光碟、光碟、数字通用光碟、蓝光光碟等)、磁盘存储介质或者其他磁存储设备、或者能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质，但不限于此。存储器可以是独立存在，通过总线与处理器相连接。存储器也可以和处理器集成在一起。
处理部件1502，也可以称为处理器，处理单元，处理单板，处理模块、处理装置等。处理部件可以是中央处理器(central processing unit，CPU)，网络处理器(network processor，NP)或者CPU和NP的组合，也可以是微处理器，特定应用集成电路(application-specific integrated circuit，ASIC)，或一个或多个用于控制以上方案程序执行的集成电路。
通信部件1503，也可以称为收发机，或收发器等，可以是用于与其他设备或通信网络通信，其中可以包括用来进行无线、有线或其他通信方式的单元。
当基于骨骼关键点的动作还原设备15为图1所述终端时，所述处理部件1502用于调用所述存储部件1501的数据执行如下操作：获取所述第一骨骼关键点在目标平面上的坐标位置；所述目标平面用于指示目标对象的骨骼关键点从三维空间映射到二维平面的相对位置关系；确定所述目标平面上以所述第一骨骼关键点的坐标位置为中心点的一个或多个曲线，所述曲线用于指示所述第二骨骼关键点映射到所述目标平面上的所有可能的位置；所述曲线上的每一个点的坐标位置对应所述第二骨骼关键点在三维空间中的一个或多个角度 信息；根据所述第二骨骼关键点在所述目标平面上的坐标位置，确定所述第二骨骼关键点所属的目标曲线，所述目标曲线为所述一个或多个曲线中的一个；根据所述第二骨骼关键点的坐标位置和所述目标曲线，确定所述第二骨骼关键点的空间位置。
本申请实施例还提供了一种装置，请参见图16，图16是本申请实施例提供的一种装置结构示意图；如图16所示，该装置16可以包括处理器1601和存储器1602；处理器1601，用于支持所述装置执行前述方法实施例中任意一项所述方法的相应功能；存储器1602，用于保存所述装置的程序指令和数据。当该装置16位一种芯片系统时，该芯片系统执行前述方法实施例中任意一项所述的方法；所述芯片系统还可以包括其他的外部分立器件。当所述装置16为一种终端设备时，可以参考前述图15所述设备的相关描述，在此不再赘述。
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。
需要说明的是，对于前述的各方法实施例，为了简单描述，故将其都表述为一系列的动作组合，但是本领域技术人员应该知悉，本申请并不受所描述的动作顺序的限制，因为依据本申请，某些步骤可能可以采用其他顺序或者同时进行。其次，本领域技术人员也应该知悉，说明书中所描述的实施例均属于优选实施例，所涉及的动作和模块并不一定是本申请所必须的。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置，可通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如上述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性或其它的形式。
上述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
上述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以为个人计算机、服务器或者网络设备等，具体可以是计算机设备中的处理器)执行本申请各个实施例上述方法的全部或部分步骤。其中，而前述的存储介质可包括：U盘、移动硬盘、磁碟、光盘、只读存储器(Read-Only Memory，缩写：ROM)或者随机存取存储器(Random Access Memory，缩写：RAM)等各种可以存储程序代码的介质。
以上所述，以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述 实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围。

Claims (17)

  1. 一种基于骨骼关键点的动作还原方法，其特征在于，包括：
    获取所述第一骨骼关键点在目标平面上的坐标位置；所述目标平面用于指示目标对象的骨骼关键点从三维空间映射到二维平面的相对位置关系；
    确定所述目标平面上以所述第一骨骼关键点的坐标位置为中心点的一个或多个曲线，所述曲线用于指示所述第二骨骼关键点映射到所述目标平面上的所有可能的位置；所述曲线上的每一个点的坐标位置对应所述第二骨骼关键点在三维空间中的一个或多个角度信息；
    根据所述第二骨骼关键点在所述目标平面上的坐标位置，确定所述第二骨骼关键点所属的目标曲线，所述目标曲线为所述一个或多个曲线中的一个；
    根据所述第二骨骼关键点的坐标位置和所述目标曲线，确定所述第二骨骼关键点的空间位置。
  2. 根据权利要求1所述的方法，其特征在于，所述获取所述第一骨骼关键点在目标平面上的坐标位置之前，还包括：
    获取所述目标对象的图像信息；
    基于所述图像信息确定所述目标对象的第一骨骼关键点和第二骨骼关键点，所述第一骨骼关键点和所述第二骨骼关键点为所述目标对象的相邻骨骼关键点。
  3. 根据权利要求2所述的方法，其特征在于，所述图像信息为视频；所述第一骨骼关键点和所述第二骨骼关键点为同一帧视频中同一关节上相邻的骨骼关键点；所述基于所述图像信息确定所述目标对象的第一骨骼关键点和第二骨骼关键点，包括：
    在所述同一帧视频中确定所述目标对象的多个骨骼关键点；
    从所述多个骨骼关键点中确定所述第一骨骼关键点，以及与所述第一骨骼关键点在所述同一个关节上相邻的所述第二骨骼关键点。
  4. 根据权利要求3所述的方法，其特征在于，所述在所述同一帧视频中确定所述目标对象的多个骨骼关键点，包括：
    从所述目标对象的视频中提取每一帧视频图片，并获取所述每一帧视频图片对应的目标平面；
    根据预设的骨骼关键点识别算法，确定所述每一帧视频图片对应的目标平面中所述目标对象的多个骨骼关键点。
  5. 根据权利要求4所述的方法，其特征在于，所述确定所述第二骨骼关键点的空间位置之后，还包括：
    根据所述每一帧视频图片的排列顺序，依次确定所述每一帧视频图片中所述第二骨骼关键点的空间位置，以生成所述第二骨骼关键点的运动轨迹。
  6. 根据权利要求1-5任一项所述的方法，其特征在于，所述根据所述第二骨骼关键点在所述目标平面上的坐标位置，确定所述第二骨骼关键点所属的目标曲线，包括：
    在所述目标平面上建立二维平面坐标系，确定所述第二骨骼关键点在所述目标平面上的二维坐标；
    根据所述二维坐标，确定所述第二骨骼关键点所属的所述目标曲线。
  7. 根据权利要求6所述的方法，其特征在于，所述方法还包括：通过相对距离表示所述第一骨骼点和第二骨骼点的距离关系；所述相对距离为所述第一骨骼关键点和所述第二骨骼关键点之间在所述图像信息上的实际距离，与所述第一骨骼关键点和所述第二骨骼关键点之间在所述图像信息上能够呈现的最大距离之比。
  8. 一种基于骨骼关键点的动作还原装置，其特征在于，包括：
    坐标获取单元，用于获取所述第一骨骼关键点在目标平面上的坐标位置；所述目标平面用于指示目标对象的骨骼关键点从三维空间映射到二维平面的相对位置关系；
    曲线预设单元，用于确定所述目标平面上以所述第一骨骼关键点的坐标位置为中心点的一个或多个曲线，所述曲线用于指示所述第二骨骼关键点映射到所述目标平面上的所有可能的位置；所述曲线上的每一个点的坐标位置对应所述第二骨骼关键点在三维空间中的一个或多个角度信息；
    目标曲线确定单元，用于根据所述第二骨骼关键点在所述目标平面上的坐标位置，确定所述第二骨骼关键点所属的目标曲线，所述目标曲线为所述一个或多个曲线中的一个；
    空间位置确定单元，用于根据所述第二骨骼关键点的坐标位置和所述目标曲线，确定所述第二骨骼关键点的空间位置。
  9. 根据权利要求8所述的装置，其特征在于，所述装置还包括图像信息获取单元和骨骼关键点单元；
    所述图像信息获取单元，用于在所述获取所述第一骨骼关键点在目标平面上的坐标位置之前，获取所述目标对象的图像信息；
    所述骨骼关键点单元，用于基于所述图像信息确定所述目标对象的第一骨骼关键点和第二骨骼关键点，所述第一骨骼关键点和所述第二骨骼关键点为所述目标对象的相邻骨骼关键点。
  10. 根据权利要求9所述的装置，其特征在于，所述图像信息为视频；所述第一骨骼关键点和所述第二骨骼关键点为同一帧视频中同一关节上相邻的骨骼关键点；所述装置还包括多骨骼关键点单元和目标骨骼关键点单元；
    所述多骨骼关键点单元，用于在所述同一帧视频中确定所述目标对象的多个骨骼关键点；
    所述目标骨骼关键点单元，用于从所述多个骨骼关键点中确定所述第一骨骼关键点，以及与所述第一骨骼关键点在所述同一个关节上相邻的所述第二骨骼关键点。
  11. 根据权利要求10所述的装置，其特征在于，所述多骨骼关键点单元，具体用于：
    从所述目标对象的视频中提取每一帧视频图片，并获取所述每一帧视频图片对应的目标平面；
    根据预设的骨骼关键点识别算法，确定所述每一帧视频图片对应的目标平面中所述目标对象的多个骨骼关键点。
  12. 根据权利要求11所述的装置，其特征在于，所述装置还包括轨迹确定单元，用于：在所述确定所述第二骨骼关键点的空间位置之后，根据所述每一帧视频图片的排列顺序，依次确定所述每一帧视频图片中所述第二骨骼关键点的空间位置，以生成所述第二骨骼关键点的运动轨迹。
  13. 根据权利要求8-12任一项所述的装置，其特征在于，所述目标曲线确定单元，具体用于：
    在所述目标平面上建立二维平面坐标系，确定所述第二骨骼关键点在所述目标平面上的二维坐标；
    根据所述二维坐标，确定所述第二骨骼关键点所属的所述目标曲线。
  14. 根据权利要求13所述的装置，其特征在于，所述装置还包括距离单元，用于通过相对距离表示所述第一骨骼点和第二骨骼点的距离关系；所述相对距离为所述第一骨骼关键点和所述第二骨骼关键点之间在所述图像信息上的实际距离，与所述第一骨骼关键点和所述第二骨骼关键点之间在所述图像信息上能够呈现的最大距离之比。
  15. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行以实现权利要求1至7任意一项所述的方法。
  16. 一种计算机程序，其特征在于，当所述计算机程序在处理器上运行时，使得处理器执行权利要求1至7任意一项描述的方法。
  17. 一种装置，其特征在于，所述装置包括处理器，用于支持所述装置执行如权利要求1-7中任意一项所述的方法的相应功能；所述装置还可以包括与所述处理器耦合的存储器，用于保存所述装置的程序指令和数据。
PCT/CN2021/076723 2020-02-29 2021-02-18 一种基于骨骼关键点的动作还原方法以及装置 WO2021169839A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010132407.5A CN111402290B (zh) 2020-02-29 2020-02-29 一种基于骨骼关键点的动作还原方法以及装置
CN202010132407.5 2020-02-29

Publications (1)

Publication Number Publication Date
WO2021169839A1 true WO2021169839A1 (zh) 2021-09-02

Family

ID=71432110

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/076723 WO2021169839A1 (zh) 2020-02-29 2021-02-18 一种基于骨骼关键点的动作还原方法以及装置

Country Status (2)

Country Link
CN (1) CN111402290B (zh)
WO (1) WO2021169839A1 (zh)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113974828A (zh) * 2021-09-30 2022-01-28 西安交通大学第二附属医院 一种手术参考方案生成方法及装置
CN114240740A (zh) * 2021-12-16 2022-03-25 数坤(北京)网络科技股份有限公司 骨骼展开图像的获取方法、装置、医疗设备以及存储介质
CN114285960A (zh) * 2022-01-29 2022-04-05 北京卡路里信息技术有限公司 视频处理方法及装置
CN114638921A (zh) * 2022-05-19 2022-06-17 深圳元象信息科技有限公司 动作捕捉方法、终端设备及存储介质
CN116246350A (zh) * 2023-05-11 2023-06-09 山东工程职业技术大学 基于动作捕捉的运动监测方法、装置、设备及存储介质
CN117315791A (zh) * 2023-11-28 2023-12-29 杭州华橙软件技术有限公司 骨骼动作识别方法、设备及存储介质

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111402290B (zh) * 2020-02-29 2023-09-12 华为技术有限公司 一种基于骨骼关键点的动作还原方法以及装置
CN112164129A (zh) * 2020-09-02 2021-01-01 北京电影学院 基于深度卷积网络的无配对动作迁移方法
CN112894828B (zh) * 2021-03-02 2022-05-20 乐聚(深圳)机器人技术有限公司 机器人动作模仿方法、装置、设备及存储介质
CN113505662B (zh) * 2021-06-23 2024-03-01 广州大学 一种健身指导方法、装置及存储介质
CN113470150A (zh) * 2021-07-12 2021-10-01 北京顺天立安科技有限公司 一种基于骨骼驱动还原口型的方法和系统
CN114225420A (zh) * 2021-11-19 2022-03-25 达闼科技（北京）有限公司 动作数据获取方法、系统、装置、设备和存储介质
CN115714888B (zh) * 2022-10-09 2023-08-29 名之梦(上海)科技有限公司 视频生成方法、装置、设备与计算机可读存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050256389A1 (en) * 2001-11-16 2005-11-17 Yoshio Koga Calculation method, calculation program and calculation system for information supporting arthroplasty
CN103543830A (zh) * 2013-10-28 2014-01-29 四川大学 一种人体骨骼点映射至立体显示中虚拟三维空间点的方法
CN104658022A (zh) * 2013-11-20 2015-05-27 中国电信股份有限公司 三维动画制作方法和装置
CN106485773A (zh) * 2016-09-14 2017-03-08 厦门幻世网络科技有限公司 一种用于生成动画数据的方法和装置
CN107225573A (zh) * 2017-07-05 2017-10-03 上海未来伙伴机器人有限公司 机器人的动作控制方法和装置
CN111402290A (zh) * 2020-02-29 2020-07-10 华为技术有限公司 一种基于骨骼关键点的动作还原方法以及装置

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5682886A (en) * 1995-12-26 1997-11-04 Musculographics Inc Computer-assisted surgical system
CN101520902A (zh) * 2009-02-24 2009-09-02 上海大学 低成本动作捕捉与演示系统及方法
EP2674913B1 (en) * 2012-06-14 2014-07-23 Softkinetic Software Three-dimensional object modelling fitting & tracking.
CN106228111A (zh) * 2016-07-08 2016-12-14 天津大学 一种基于骨骼序列提取关键帧的方法
CN106251387A (zh) * 2016-07-29 2016-12-21 武汉光之谷文化科技股份有限公司 一种基于动作捕捉的成像系统
CN106650687B (zh) * 2016-12-30 2020-05-19 山东大学 一种基于深度信息和骨骼信息的姿势矫正方法
CN108010134A (zh) * 2017-11-29 2018-05-08 湘潭大学 一种基于移动终端的实时三维虚拟试衣方法
CN108229332B (zh) * 2017-12-08 2020-02-14 华为技术有限公司 骨骼姿态确定方法、装置及计算机可读存储介质
CN110033505A (zh) * 2019-04-16 2019-07-19 西安电子科技大学 一种基于深度学习的人体动作捕捉与虚拟动画生成方法
CN110660017A (zh) * 2019-09-02 2020-01-07 北京航空航天大学 一种基于三维姿态识别的舞谱记录与演示方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050256389A1 (en) * 2001-11-16 2005-11-17 Yoshio Koga Calculation method, calculation program and calculation system for information supporting arthroplasty
CN103543830A (zh) * 2013-10-28 2014-01-29 四川大学 一种人体骨骼点映射至立体显示中虚拟三维空间点的方法
CN104658022A (zh) * 2013-11-20 2015-05-27 中国电信股份有限公司 三维动画制作方法和装置
CN106485773A (zh) * 2016-09-14 2017-03-08 厦门幻世网络科技有限公司 一种用于生成动画数据的方法和装置
CN107225573A (zh) * 2017-07-05 2017-10-03 上海未来伙伴机器人有限公司 机器人的动作控制方法和装置
CN111402290A (zh) * 2020-02-29 2020-07-10 华为技术有限公司 一种基于骨骼关键点的动作还原方法以及装置

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113974828A (zh) * 2021-09-30 2022-01-28 西安交通大学第二附属医院 一种手术参考方案生成方法及装置
CN113974828B (zh) * 2021-09-30 2024-02-09 西安交通大学第二附属医院 一种手术参考方案生成方法及装置
CN114240740A (zh) * 2021-12-16 2022-03-25 数坤(北京)网络科技股份有限公司 骨骼展开图像的获取方法、装置、医疗设备以及存储介质
CN114285960A (zh) * 2022-01-29 2022-04-05 北京卡路里信息技术有限公司 视频处理方法及装置
CN114285960B (zh) * 2022-01-29 2024-01-30 北京卡路里信息技术有限公司 视频处理方法及装置
CN114638921A (zh) * 2022-05-19 2022-06-17 深圳元象信息科技有限公司 动作捕捉方法、终端设备及存储介质
CN114638921B (zh) * 2022-05-19 2022-09-27 深圳元象信息科技有限公司 动作捕捉方法、终端设备及存储介质
CN116246350A (zh) * 2023-05-11 2023-06-09 山东工程职业技术大学 基于动作捕捉的运动监测方法、装置、设备及存储介质
CN117315791A (zh) * 2023-11-28 2023-12-29 杭州华橙软件技术有限公司 骨骼动作识别方法、设备及存储介质
CN117315791B (zh) * 2023-11-28 2024-02-20 杭州华橙软件技术有限公司 骨骼动作识别方法、设备及存储介质

Also Published As

Publication number Publication date
CN111402290A (zh) 2020-07-10
CN111402290B (zh) 2023-09-12

Similar Documents

Publication Publication Date Title
WO2021169839A1 (zh) 一种基于骨骼关键点的动作还原方法以及装置
CN109636831B (zh) 一种估计三维人体姿态及手部信息的方法
Mueller et al. Real-time hand tracking under occlusion from an egocentric rgb-d sensor
Park et al. Capturing and animating skin deformation in human motion
Stoll et al. Fast articulated motion tracking using a sums of gaussians body model
Xia et al. A survey on human performance capture and animation
Shum et al. Real-time posture reconstruction for microsoft kinect
Chai et al. Performance animation from low-dimensional control signals
CN110264509A (zh) 确定图像捕捉设备的位姿的方法、装置及其存储介质
CN113706699B (zh) 数据处理方法、装置、电子设备及计算机可读存储介质
WO2011075082A1 (en) Method and system for single view image 3 d face synthesis
Chen et al. Realtime reconstruction of an animating human body from a single depth camera
Aristidou Hand tracking with physiological constraints
Hori et al. Silhouette-based synthetic data generation for 3D human pose estimation with a single wrist-mounted 360° camera
Liu et al. Skeleton tracking based on Kinect camera and the application in virtual reality system
Perales Human motion analysis and synthesis using computer vision and graphics techniques. State of art and applications
CN115205737B (zh) 基于Transformer模型的运动实时计数方法和系统
CN115205750B (zh) 基于深度学习模型的运动实时计数方法和系统
CN116863044A (zh) 人脸模型的生成方法、装置、电子设备及可读存储介质
KR20200057572A (ko) 딥러닝 기반 손 인식 증강현실 상호 작용 장치 및 방법
Wang et al. A Generative Human-Robot Motion Retargeting Approach Using a Single RGBD Sensor.
CN113496168B (zh) 手语数据采集方法、设备、存储介质
Jian et al. Realistic face animation generation from videos
Yang et al. Synthetic hands generator for RGB hand tracking
Hori et al. Silhouette-Based 3D Human Pose Estimation Using a Single Wrist-Mounted 360° Camera

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21760273

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21760273

Country of ref document: EP

Kind code of ref document: A1