CN112102451B - Wearable virtual live broadcast method and equipment based on common camera - Google Patents

Wearable virtual live broadcast method and equipment based on common camera Download PDF

Info

Publication number
CN112102451B
CN112102451B CN202010739818.0A CN202010739818A CN112102451B CN 112102451 B CN112102451 B CN 112102451B CN 202010739818 A CN202010739818 A CN 202010739818A CN 112102451 B CN112102451 B CN 112102451B
Authority
CN
China
Prior art keywords
dimensional coordinate
points
bone
dimensional
virtual character
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010739818.0A
Other languages
Chinese (zh)
Other versions
CN112102451A (en
Inventor
冯志强
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yungoal Tech Co ltd
Original Assignee
Yungoal Tech Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yungoal Tech Co ltd filed Critical Yungoal Tech Co ltd
Priority to CN202010739818.0A priority Critical patent/CN112102451B/en
Publication of CN112102451A publication Critical patent/CN112102451A/en
Application granted granted Critical
Publication of CN112102451B publication Critical patent/CN112102451B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—3D [Three Dimensional] animation
    • G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Abstract

The application discloses a wearable virtual live broadcast method and equipment based on a common camera, which are used for solving the technical problems that the existing virtual live broadcast method is inaccurate in virtual character action or expression display and cannot be live broadcast for a long time. The method comprises the following steps: determining three-dimensional coordinates of a plurality of feature points of the anchor face through a bone point three-dimensional coordinate recognition network model; determining three-dimensional coordinates of a plurality of feature points of the face of the virtual character; calculating three-dimensional coordinate offset and skeleton included angles corresponding to the feature points; based on the three-dimensional coordinate offset and the bone included angles, determining target three-dimensional coordinates of a plurality of feature points of the face of the virtual character through an expression bone redirection network model; and controlling the expression of the virtual character based on the target three-dimensional coordinates. According to the method, the virtual live broadcast can be realized by the anchor without wearing hardware equipment, the requirement of long-time live broadcast is met, and the accuracy of the virtual character display action or expression is ensured.

Description

Wearable virtual live broadcast method and equipment based on common camera
Technical Field
The application relates to the technical field of live broadcasting, in particular to a wearable virtual live broadcasting method and equipment based on a common camera.
Background
With the continuous development of live broadcasting technology, many people who are not willing to face and want to engage in live broadcasting industry can select corresponding motion capture equipment to perform virtual live broadcasting to achieve live broadcasting wish.
However, in the existing virtual live broadcast method, the host needs to wear hardware devices for a long time, and the hardware devices usually need to be connected with various cables, so that the host can not comfortably complete various actions, and the actions or expressions displayed by the virtual roles are inaccurate; in addition, these hardware devices need to be charged frequently, and it is also difficult to meet the long-term live demand.
Disclosure of Invention
The embodiment of the application provides a wearable virtual live broadcast method and equipment based on a common camera, which are used for solving the technical problems that the existing virtual live broadcast technology is easy to cause inaccurate virtual character actions or expression display and cannot meet the long-time live broadcast requirement.
In a first aspect, an embodiment of the present application provides a wearable virtual live broadcast method based on a common camera, including: identifying a network model according to the three-dimensional coordinates of the bone points, and determining the three-dimensional coordinates of a plurality of characteristic points of the face of the anchor under a first preset three-dimensional coordinate system based on two-dimensional plane image data related to the anchor; calculating three-dimensional coordinate offset of a plurality of feature points of the anchor face relative to the root skeleton points under a first preset three-dimensional coordinate system; and calculating the bone included angle between the bone between any two adjacent feature points and the corresponding bone of the previous stage; in a second preset three-dimensional coordinate system, determining three-dimensional coordinates of a plurality of characteristic points of the face of the virtual character; calculating three-dimensional coordinate offset of a plurality of feature points of the face of the virtual character relative to the root bone points and bone included angles between bones between any two adjacent feature points and corresponding upper-level bones; determining target three-dimensional coordinates of a plurality of feature points of the face of the virtual character under a second preset three-dimensional coordinate system through an expression skeleton redirection network model based on three-dimensional coordinate offsets and skeleton included angles corresponding to the plurality of feature points of the face of the main cast and three-dimensional coordinate offsets and skeleton included angles corresponding to the plurality of feature points of the face of the virtual character; and adjusting a plurality of characteristic points of the face of the virtual character to the target three-dimensional coordinate positions so as to control the expression of the virtual character.
According to the wearable virtual live broadcast method based on the common camera, three-dimensional coordinates of a plurality of characteristic points of a main broadcasting face are identified through a skeleton point three-dimensional coordinate identification network model; and then, based on the three-dimensional coordinates, the expression skeleton is used for redirecting the network model to control the expression of the virtual character, so that the virtual character can accurately display the expression of the host, the display is more natural, and the accuracy of the true host expression of the virtual character is ensured. Meanwhile, the expression of the virtual character is adjusted through the neural network model, so that a host can easily complete the virtual live broadcast process without hardware equipment, the requirement of long-time live broadcast is met, and the host can comfortably and naturally complete the live broadcast expression or action in the live broadcast process.
In one implementation of the present application, the bone included angle between the bone between any two adjacent feature points of the anchor face and its corresponding superior bone is determined by the following formula:
α＝(α 1 ，α 2 ，α 3 )＝(x 1 -x 2 ，y 1 -y 2 ，z 1 -z 2 )
β＝(β 1 ，β 2 ，β 3 )＝(x 3 -x 2 ，y 3 -y 2 ，z 3 -z 2 )
wherein ,(x1 ，y 1 ，z 1 ) Three-dimensional coordinates of the first feature point; (x) 2 ，y 2 ，z 2 ) Three-dimensional coordinates of the second feature points; (x) 3 ，y 3 ，z 3 ) Three-dimensional coordinates of the third feature point; α represents three-dimensional coordinates of the first bone between the first feature point and the second feature point; beta represents the three-dimensional coordinates of the second bone between the second feature point and the third feature point; r Representing a unit vector corresponding to the first bone; θ represents the bone angle between the second bone and the first bone; the first feature point is adjacent to the second feature point, and the second feature point is adjacent to the third feature point.
In one implementation of the present application, the method further comprises: converting the bone included angle theta into a quaternion Q through the following formula;
wherein θ= (θ) 0 ,θ 1 ,θ 2 )；Q＝(Q 0 ,Q 1 ,Q 2 ,Q 3 )。
In the embodiment of the application, the bone included angle theta is converted into the quaternion Q for representing the angle in the three-dimensional space, and the quaternion Q is equivalent to the Euler angle, but the problem of universal lock in the Euler angle representation is avoided, and the included angle in the three-dimensional space is represented more accurately.
In one implementation of the present application, before the network model is identified according to the three-dimensional coordinates of the bone points, the method further includes, before determining the three-dimensional coordinates of the plurality of feature points of the anchor face in the first preset three-dimensional coordinate system: collecting a plurality of two-dimensional plane image data related to a human body, and constructing a first training data set; screening the content of the first training data set, and removing image data which does not contain all the nodes and the characteristic points of the human body to obtain a second training data set; inputting the second training data into the neural network model to train the neural network model; training until the output converges, and obtaining the three-dimensional coordinate recognition network model of the skeleton points.
In one implementation of the application, the bone point three-dimensional coordinate recognition network model comprises a Gaussian heat map layer and a Gaussian heat map normalization layer; the Gaussian heat map layer is used for receiving the characteristic data output by the convolution layer and obtaining a Gaussian heat map with the size of (N, W, H, D) based on the characteristic data; wherein N is the total number of a plurality of characteristic points of the anchor face, W is the width of the Gaussian heat map, H is the height of the Gaussian heat map, and D is the depth of the Gaussian heat map; the Gaussian heat map normalization layer is used for normalizing the Gaussian heat map with the size of (N, W, H, D) by the following formula:
wherein G is a Gaussian heat map;is normalized Gaussian heat map.
In one implementation of the application, the three-dimensional coordinate recognition network model of the skeleton point also comprises a three-dimensional coordinate output layer; the three-dimensional coordinate output layer is used for receiving the normalized Gaussian heat map and outputting three-dimensional coordinates of a plurality of characteristic points of the anchor face under a first preset three-dimensional coordinate system through the following formula:
wherein ,coordinates of an x-axis representing the nth feature point; />Coordinates of a y-axis representing the nth feature point; />Coordinates of a z-axis representing the nth feature point; />Normalized Gaussian heat maps corresponding to the nth feature points; w is- >Width of (2)The method comprises the steps of carrying out a first treatment on the surface of the H is->Is of a height of (2); d is->Is a depth of (c).
In one implementation of the application, the expression skeleton redirection network model comprises a data input layer, a convolution layer and a target three-dimensional coordinate output layer; the data input layer is used for inputting three-dimensional coordinate offset and skeleton included angles corresponding to a plurality of feature points of the anchor face and three-dimensional coordinate offset and skeleton included angles corresponding to a plurality of feature points of the virtual character face into the expression skeleton redirection network model; the convolution layer is used for receiving the output data of the data input layer and carrying out convolution, filling and sampling operations on the output data so as to obtain characteristic data corresponding to the output data; the target three-dimensional coordinate output layer is used for receiving the characteristic data, carrying out nonlinear transformation on the characteristic data, and outputting target three-dimensional coordinates of a plurality of characteristic points of the face of the virtual character under a second preset three-dimensional coordinate system.
According to the embodiment of the application, the expression of the virtual character is controlled through the expression skeleton redirection network model according to the offset of the three-dimensional coordinate table and the skeleton included angles corresponding to the plurality of feature points of the anchor face, so that the expression displayed by the virtual character is more accurate and natural. The accuracy of displaying the expression of the main broadcasting in the live broadcasting process of the virtual character is ensured.
In one implementation of the present application, the method further comprises: identifying a network model according to the three-dimensional coordinates of the bone points, and determining the three-dimensional coordinates of a plurality of joint points of the body and the hand of the anchor under a first preset three-dimensional coordinate system; calculating three-dimensional coordinate offset of a plurality of joint points of the body and the hand of the anchor relative to root skeleton points under a first preset three-dimensional coordinate system; and calculating a bone included angle between bones between any two adjacent joint points and corresponding bones at the upper level; wherein the root skeletal points are pelvic bone points of the anchor body; in a second preset three-dimensional coordinate system, determining three-dimensional coordinates of a plurality of joints of the body and the hand of the virtual character, and calculating three-dimensional coordinate offset of the joints of the body and the hand of the virtual character relative to a root bone point and bone included angles between bones between any two adjacent joints and corresponding upper bones under the second preset three-dimensional coordinate system; the first preset three-dimensional coordinate system is a right-hand coordinate system established by taking a pelvic bone point of a main body as an origin; the second preset three-dimensional coordinate system is a right-hand coordinate system established by taking a pelvic bone point of the virtual character body as an origin; determining target three-dimensional coordinates of the plurality of joints of the body and the hand of the virtual character under a second preset three-dimensional coordinate system through an action bone redirection network model based on the three-dimensional coordinate offset and the bone included angle corresponding to the plurality of joints of the body and the hand of the anchor and the three-dimensional coordinate offset and the bone included angle corresponding to the plurality of joints of the body and the hand of the virtual character; and adjusting a plurality of joint points of the body and the hand of the virtual character to the target three-dimensional coordinate positions so as to control the actions of the virtual character.
In the embodiment of the application, the three-dimensional coordinates of a plurality of joints of the body and the hand of the anchor are identified through the bone point three-dimensional coordinate identification network model, and the three-dimensional coordinate offset and the bone included angle corresponding to the joints of the body and the hand of the anchor are determined based on the three-dimensional coordinates. The actions of the virtual character are then controlled by the action skeleton redirection network model. The virtual character can display the action of the anchor more accurately, and the accuracy of the action of the anchor displayed by the virtual character is further ensured. Meanwhile, the three-dimensional coordinates of a plurality of joints of the body and the hand of the anchor can be identified without wearing hardware equipment by the anchor, and a depth image related to the anchor is not required to be acquired by a specific camera, but the joints of the body and the hand of the anchor in a common two-dimensional plane image are identified directly by a deep learning neural network model, so that the anchor can comfortably complete live action in the live broadcast process, and the comfort and convenience of the live broadcast process by the anchor are ensured; but also reduces the cost of virtual live broadcast.
In one implementation of the application, the action skeleton redirection network model comprises a data input layer, a convolution layer and a target three-dimensional coordinate output layer; the data input layer is used for inputting three-dimensional coordinate offset and skeleton included angles corresponding to a plurality of joints of the body and the hand of the anchor, and three-dimensional coordinate offset and skeleton included angles corresponding to a plurality of joints of the body and the hand of the virtual character into the action skeleton redirection network model; the convolution layer is used for receiving the output data of the data input layer and carrying out convolution, filling and sampling operations on the output data so as to obtain characteristic data corresponding to the output data; the target three-dimensional coordinate output layer is used for receiving the characteristic data, carrying out nonlinear transformation on the characteristic data, and outputting target three-dimensional coordinates of a plurality of joints of the body and the hand of the virtual character under a second preset three-dimensional coordinate system. The actions corresponding to the joints of the body and the hand of the host are redirected to the virtual character body through the action skeleton redirection network model, so that the accuracy of the virtual character to display the host action is ensured.
In a second aspect, the embodiment of the application also provides wearable virtual live broadcast equipment based on the common camera. The device comprises: a processor; and a memory having executable code stored thereon, which when executed causes the processor to perform a wearable virtual live broadcast method based on a common camera as described above.
In a third aspect, the embodiment of the application also provides a wearable virtual live broadcast device based on a common camera, which comprises an acquisition module, a control module and a control module, wherein the acquisition module is used for acquiring two-dimensional plane image data related to a host; the identification module is used for inputting the two-dimensional plane image data into a skeleton point three-dimensional coordinate identification network model so as to identify a plurality of joint points of the body and the hand of the anchor and a plurality of characteristic points of the face of the anchor, and the three-dimensional coordinates in a first preset three-dimensional coordinate system; the control module is used for controlling the actions of the virtual roles through the action skeleton redirection network model based on three-dimensional coordinates corresponding to a plurality of joint points of the body and the hand of the anchor respectively; and the three-dimensional coordinates are used for controlling the expression of the virtual character through redirecting the network model through the expression skeleton based on the three-dimensional coordinates corresponding to the feature points of the anchor face.
In a fourth aspect, embodiments of the present application further provide a storage medium, where the storage medium is a non-volatile computer readable storage medium; the non-transitory computer readable storage medium stores at least one program, each program comprising instructions that, when executed by a device having a processor, cause the device to perform a wearable virtual live method based on a generic camera as described above.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
fig. 1 is a flowchart of a wearable virtual live broadcast method based on a common camera according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a distribution of joints of a human body according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a distribution of human hand joints according to an embodiment of the present application;
FIG. 4 is a schematic diagram of distribution of facial feature points of a person according to an embodiment of the present application;
FIG. 5 is a schematic diagram of a three-dimensional coordinate recognition network model of bone points according to an embodiment of the present application;
FIG. 6 is a schematic diagram of an action skeleton redirection network model according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of an expressive bone redirecting network model according to an embodiment of the present application;
fig. 8 is a schematic diagram of an internal structure of a wearable virtual live broadcast device based on a common camera according to an embodiment of the present application;
fig. 9 is a schematic diagram of an internal structure of a wearable virtual live broadcast device based on a common camera according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
With the development of live broadcasting technology, many people who are not willing to show the face and get out of the mirror and want to engage in the anchor industry can achieve the wish by selecting to use the motion capture device to perform virtual live broadcasting.
In the prior art, after a host selects a virtual character, the host needs to wear a hardware motion capture device for a long time to perform live broadcast in the live broadcast process. Such hardware motion capture devices capture the three-dimensional coordinates of a number of skeletal points of a host through various sensors. For any one of a plurality of bone points, the anchor terminal adjusts the coordinates of the corresponding bone point of the virtual character according to the three-dimensional coordinates of the bone point, so that the action of the virtual character displayed on the live interface is synchronous with the action of the anchor.
However, because the hardware motion capture device is inconvenient to wear and is usually connected with various cables, the anchor is difficult to comfortably complete various motions; and the coordinates of the corresponding skeleton points of the virtual character are adjusted according to the coordinates of any skeleton point of the anchor, so that the action displayed by the virtual character is easy to be inaccurate. Moreover, some wireless hardware motion capture devices require frequent charging and cannot meet long live campaigns.
In order to solve the technical problems, the embodiment of the application provides the wearable virtual live broadcast method and the wearable virtual live broadcast device based on the common camera, and a host can complete the virtual live broadcast process without wearing hardware equipment, so that the accuracy of the action display of the virtual character is ensured, and the requirement of long-time live broadcast can be met.
The following describes the technical scheme provided by the embodiment of the application in detail through the attached drawings.
Fig. 1 is a flowchart of a wearable virtual live broadcast method based on a common camera according to an embodiment of the present application. As shown in fig. 1, the virtual live broadcast method provided by the embodiment of the application includes the following steps:
step 101, acquiring two-dimensional plane image data related to a anchor.
The wearable virtual live broadcasting method based on the common camera is applied to a scene of live broadcasting through a virtual character. When the host starts virtual live, any one of several virtual roles provided by the live device can be selected as the virtual role of the live.
Further, when the anchor selects the virtual character, the image acquisition device acquires two-dimensional plane image data related to the anchor and uploads the image data to the server.
In one embodiment of the application, the image acquisition device that acquires image data related to the anchor may be an external stand-alone image acquisition device, such as a video camera; the camera can also be carried on live broadcast equipment such as a computer.
In another embodiment of the application, the anchor-related two-dimensional planar image data is acquired by a common camera.
It will be apparent to those skilled in the art that in the virtual live broadcast technique, image data may be acquired by a particular camera and then three-dimensional coordinates of skeletal points of the human body may be determined based on the image data. In the embodiment of the application, in order to reduce live broadcast cost, a common camera can be used for collecting two-dimensional plane image data of a host, the image data collected by the common camera is common RGB two-dimensional image data, and a server cannot directly identify three-dimensional coordinates of a plurality of bone points of a host body according to the two-dimensional plane image.
Step 102, inputting the two-dimensional plane image into a bone point three-dimensional coordinate recognition network model, and recognizing a plurality of joints of the body and the hand of the anchor and three-dimensional coordinates of a plurality of characteristic points of the face of the anchor.
When the action or expression of the anchor changes in the virtual live broadcast process, a plurality of nodes of the body and the hand of the anchor and a plurality of characteristic points of the face of the anchor correspondingly change. Therefore, in order to ensure that the virtual character is consistent with the action and/or expression of the anchor, after the two-dimensional plane image is acquired, several nodes of the anchor's body and hand, and three-dimensional coordinates of several feature points of the anchor's face need to be identified from the image.
The articulation points and/or feature points refer to key points whose positions change when the human body moves. Wherein, the distribution diagram of the joints of the human body is shown in fig. 2. A schematic diagram of the distribution of the joints of the human hand is shown in fig. 3. A schematic diagram of the distribution of feature points of the face of a person is shown in fig. 4.
Fig. 2 is a schematic diagram of distribution of joints of a human body according to an embodiment of the present application. As shown in fig. 2, the human body mainly includes 22 joint points. For example: skull points, neck points, left shoulder points, right shoulder points, left elbow points, right elbow points, left wrist points, right wrist points, spinal points, pelvic bone points, left hip points, right hip points, left knee points, right knee points, left ankle points, right ankle points, left foot points, right foot points, and the like.
Fig. 3 is a schematic diagram of distribution of human hand joints according to an embodiment of the present application. As shown in fig. 3, a single hand includes 15 joints, and the total of both left and right hands includes 30 joints. For example: left thumb (proximal phalangeal point, middle phalangeal point, distal phalangeal point); left index finger (proximal phalangeal point, middle phalangeal point, distal phalangeal point); left middle finger (proximal phalangeal point, middle phalangeal point, distal phalangeal point); left ring finger (proximal phalangeal point, middle phalangeal point, distal phalangeal point); left little finger (proximal phalangeal point, middle phalangeal point, distal phalangeal point), etc.
Fig. 4 is a schematic distribution diagram of facial feature points of a person according to an embodiment of the present application. As shown in fig. 4, the face of the person has 68 feature points. For example: left eyebrow point, right eyebrow point, left eye point, right eye point, nose bridge point, left nasal alar point, right nasal alar point, upper lip point, lower lip point, etc. Inputting a two-dimensional plane image related to the anchor into a bone point three-dimensional coordinate recognition network model, and recognizing three-dimensional coordinates of 120 joint points and/or characteristic points of the human body under a first preset three-dimensional coordinate system.
It will be apparent to those skilled in the art that the bone point three-dimensional coordinate recognition network model may also recognize three-dimensional coordinates of human body part joints and/or feature points. In practical application, the application can be properly adjusted according to practical requirements, and the embodiment of the application is not limited to the practical requirements.
It should be noted that, the schematic distribution of the joint points of the human body and the hand and the schematic distribution of the feature points of the human face provided in the embodiment of the present application are only an exemplary schematic distribution. Those skilled in the art may have other distribution diagrams in specific applications, and the embodiments of the present application are not limited thereto.
In one embodiment of the present application, before inputting the two-dimensional plane image into the bone point three-dimensional coordinate recognition network model, the bone point three-dimensional coordinate recognition network model is trained first, specifically as follows:
Collecting a plurality of two-dimensional plane image data related to a human body, and constructing a first training data set; and then screening the content of the first training data set, and removing the image data which does not contain all the nodes and the characteristic points of the human body to obtain a second training data set. Inputting the second training data into the neural network model to train the neural network model; training until the output converges, and obtaining the three-dimensional coordinate recognition network model of the skeleton points.
The bone point three-dimensional coordinate recognition network model recognizes three-dimensional coordinates of a plurality of joint points of the body and the hand of the anchor and a plurality of characteristic points of the face of the anchor by analyzing the two-dimensional plane image. The specific implementation mode is as follows:
the three-dimensional coordinate recognition network model of bone points internally comprises a plurality of layers, as shown in fig. 5.
Fig. 5 is a schematic structural diagram of a three-dimensional coordinate recognition network model of a bone point according to an embodiment of the present application. As shown in fig. 5, the bone point three-dimensional coordinate recognition network model includes: the data input layer is used for inputting the acquired two-dimensional plane image into the bone point three-dimensional coordinate recognition network model. And the convolution layer receives the output from the data input layer and performs convolution, padding, sampling and nonlinear variation operations.
It can be clear to those skilled in the art that operations such as convolution, padding, sampling, and nonlinear transformation performed by the convolution layer can be implemented by the convolution layer of the existing neural network model, and the embodiments of the present application are not described herein.
Further, the three-dimensional coordinate recognition network model of the bone points in the embodiment of the application further comprises a Gaussian heat map layer after the convolution layer, wherein the Gaussian heat map layer is used for receiving the output of the last convolution layer and obtaining the Gaussian heat map with the dimensions of (N, W, H, D). Wherein N is the number of the node points and/or the characteristic points; w is the width of the Gaussian heat map; h is the height of the gaussian heat map; d is the depth of the gaussian heat map.
Further, a gaussian heat map normalization layer is further included after the gaussian heat map layer, and is configured to receive an output of the gaussian heat map layer, and normalize the gaussian heat map by the following formula:
wherein G is a Gaussian heat map;is normalized Gaussian heat map.
Further, after the normalized Gaussian heat map is obtained, the skeleton point three-dimensional coordinate recognition network model outputs a plurality of joint points of the body and the hand of the anchor and three-dimensional coordinates of a plurality of characteristic points of the face of the anchor under a first preset three-dimensional coordinate system through the coordinate output layer.
The coordinate output layer receives the output from the Gaussian heat map normalization layer, and calculates a plurality of joint points of the body and the hand of the anchor and three-dimensional coordinates corresponding to a plurality of characteristic points of the face of the anchor respectively through the following formulas:
wherein ,coordinates of the x-axis representing the nth node/feature point; />Coordinates representing the y-axis of the nth node/feature point; />Coordinates representing the z-axis of the nth node/feature point; />Normalized Gaussian heat maps corresponding to the nth node/feature point are obtained; w is->Is a width of (2); h is->Is of a height of (2); d is->Is a depth of (c).
In one embodiment of the present application, the gaussian heat map output by the gaussian heat map layer includes a plurality of light spots, and each light spot corresponds to one of a plurality of joints of the body and the hands of the anchor, and a plurality of feature points of the face of the anchor. After normalizing the gaussian heat maps, each light spot corresponds to one normalized gaussian heat map, and each normalized gaussian heat map corresponds to one joint point of a plurality of joint points of the anchor body and the hand, or to one feature point of a plurality of feature points of the anchor face.
In one embodiment of the present application, the three-dimensional coordinate recognition network model of bone points may be a whole network, or may be divided into three separate networks of body, hand, face, or some combination of the three (body hand network, face network, etc.).
Further, in the skeleton point three-dimensional coordinate recognition network model, the captured two-dimensional RGB image is normalized, 4 feature maps (1 x, 2x, 4x and 8 x) with downsampling resolution are obtained, a low-resolution feature map branch network is gradually and parallelly added into a high-resolution feature map main network, the low-resolution feature map branch network comprises a main network and three parallel branch networks, the resolution of the parallel branch networks is gradually reduced to half, and the corresponding width (the number of channels) is increased to twice. Wherein the trunk part obtains a feature map of 1x downsampling resolution, the first branch network part obtains a feature map of 2x downsampling resolution, the second branch network part obtains a feature map of 4x downsampling resolution, and the third branch network part obtains a feature map of 8x downsampling resolution. And multi-scale fusion and feature extraction are realized among different trunks and branch networks, the branch networks are connected in parallel, and reliable high-resolution representation is generated by repeatedly fusing representations generated by high-to-low sub-networks, so that spatially accurate Gaussian heat map estimation is obtained.
Thus, a plurality of joint points of the body and the hand of the anchor and three-dimensional coordinates of a plurality of characteristic points of the face of the anchor in a first preset three-dimensional coordinate system are obtained.
Step 103, based on the three-dimensional coordinates of a plurality of joints of the body and the hand of the anchor, the action of the virtual character is controlled through the action skeleton redirection network model.
After three-dimensional coordinates corresponding to a plurality of joints of a body and a hand of a host and three-dimensional coordinates corresponding to a plurality of feature points of a face of the host are obtained, redirecting the motion of the host to the virtual character through a motion skeleton redirection network model, and redirecting the expression of the host to the virtual character through an expression skeleton redirection network model, so that the motion and expression of the virtual character change along with the motion and expression of the host.
Based on the three-dimensional coordinates of a plurality of joints of the body and the hand of the anchor, the network model is redirected through the action skeleton, and the action of the virtual character is controlled. The specific implementation mode is as follows:
firstly, determining three-dimensional coordinate offset of a plurality of joint points of a body and a hand of a host player relative to a root bone point and bone included angles between bones between any two joint points and corresponding upper-level bones in a first preset three-dimensional coordinate system.
In one embodiment of the application, the root skeletal points are pelvic points of the anchor body. Taking the pelvic bone point as a root bone point, calculating three-dimensional coordinate offset of each joint point of the body and the hand of the anchor relative to the root bone point in a first preset three-dimensional coordinate system by the following method:
S n (x n -x 0 ，y n -y 0 ，z n -z 0 )，n∈(1，N)
Wherein the coordinates of the root skeleton point are V 0 (x 0 ，y 0 ，z 0 ) The coordinate of any joint point of the body and the hand of the anchor is V n (x n ，y n ，z n ). N is the total number of nodes of the body and hands of the anchor.
It should be noted that, according to actual needs, those skilled in the art may select other skeletal points of the anchor body as root skeletal points of a plurality of joints of the anchor body and the hand, which is not limited in the embodiment of the present application.
In one embodiment of the application, the bone included angle between any two adjacent joints in a plurality of joints of the body and the hand of the anchor and the corresponding bone of the upper stage is determined by the following formula:
α＝(α 1 ，α 2 ，α 3 )＝(x 1 -x 2 ，y 1 -y 2 ，z 1 -z 2 )
β＝(β 1 ，β 2 ，β 3 )＝(x 3 -x 2 ，y 3 -y 2 ，z 3 -z 2 )
wherein ,(x1 ，y 1 ，z 1 ) Three-dimensional coordinates of the first articulation point; (x) 2 ，y 2 ，z 2 ) Three-dimensional coordinates of the second articulation point; (x) 3 ，y 3 ，z 3 ) Is the three-dimensional coordinates of the third articulation point.
Further, α represents three-dimensional coordinates of the first bone between the first articulation point and the second articulation point; beta represents the three-dimensional coordinates of the second bone between the second articulation point and the third articulation point; r represents a unit vector corresponding to the first skeleton; θ represents the bone angle between the second bone and the first bone;
further, the first articulation point is adjacent to the second articulation point, and the second articulation point is adjacent to the third articulation point.
In another embodiment of the present application, after determining bone angles corresponding to a plurality of joints of the body and the hand of the anchor, converting the bone angle θ into a quaternion Q by the following formula;
wherein θ= (θ) 0 ,θ 1 ,θ 2 )；Q＝(Q 0 ,Q 1 ,Q 2 ,Q 3 ). Quaternion Q is a rotation that may be used to represent rotation in three dimensions. It is equivalent to euler angles, but avoids the problem of universal locking in euler angle notation.
Then, determining three-dimensional coordinates of a plurality of joints of the body and the hand of the virtual character in a second preset three-dimensional coordinate system; and calculating three-dimensional coordinate offset of each joint point of the body and the hand of the virtual character relative to the root bone point and bone between any two adjacent joint points and bone included angles between the corresponding upper bone level.
In one embodiment of the application, the root skeletal points of the body and of the hand of the avatar are pelvic points of the body of the avatar.
In another embodiment of the present application, the first preset three-dimensional coordinate system is a right-hand coordinate system established with the pelvic point of the anchor body as the origin. The second preset three-dimensional coordinate system is a right-hand coordinate system established by taking the pelvic bone point of the virtual character body as an origin.
Further, determining three-dimensional coordinate offset of a plurality of joint points of the body and the hand of the anchor relative to a root bone point in a first preset three-dimensional coordinate system and bone included angles between any two adjacent joint points and corresponding upper-level bones; and after the three-dimensional coordinate offset of the joints of the body and the hand of the virtual character relative to the root bone points in the second preset three-dimensional coordinate system and the bone included angles between any two adjacent joints and the corresponding bone included angles between the upper bones, all the three-dimensional coordinate offset, the bone included angles and the three-dimensional coordinates of the virtual character are input into the action bone redirection network model, so that the target three-dimensional coordinates of the joints of the body and the hand of the virtual character in the second preset three-dimensional coordinate system are obtained. The internal structure of the action skeleton redirection network model is shown in fig. 6.
Fig. 6 is a schematic structural diagram of an action skeleton redirection network model according to an embodiment of the present application. As shown in fig. 6, the action bone redirection network model mainly comprises a 3-layer structure. The method comprises the following steps: the system comprises a data input layer, a convolution layer and a target three-dimensional coordinate output layer.
The data input layer is used for shifting three-dimensional coordinates of a plurality of joint points of the body and the hand of the anchor relative to a root bone point under a first preset three-dimensional coordinate system, and a bone included angle between a bone between any two adjacent joint points and a corresponding bone of the upper stage; and the three-dimensional coordinates of a plurality of joint points of the body and the hand of the virtual character under a second preset three-dimensional coordinate system, the three-dimensional coordinate offset relative to the root bone point and the bone included angle between the bone between any two adjacent joint points and the corresponding upper bone are input into the action bone redirection network model. The convolution layer is used for carrying out convolution, filling, sampling and nonlinear operation on the input data. And the target three-dimensional coordinate output layer integrates the characteristic data output by the convolution layer, and outputs target three-dimensional coordinates of a plurality of joint points of the body and the hand of the virtual character under a second preset three-dimensional coordinate system after nonlinear transformation operation.
In one embodiment of the application, in the action skeleton redirection network model, three-dimensional coordinate offsets and skeleton included angle quaternions respectively corresponding to a plurality of joint points of the body and the hand of the anchor and the virtual character are spliced into tensors to be used as input. The input of the method comprises two input branches, wherein the two branches perform characteristic extraction on input data through Convolution Convolition, batch normalization Batchnormal and activation function Rule for a plurality of times; and then fusing the feature graphs obtained by the two branches, and performing Convolution Convolvulation, batch normalization Batchnormal and activation function Rule for a plurality of times to finally obtain target three-dimensional coordinates of a plurality of joints of the body and the hand of the virtual character under a second preset three-dimensional coordinate system.
Further, the positions of the three-dimensional coordinates of the body and the joints of the hand of the virtual character are adjusted to control the actions of the virtual character.
Step 104, redirecting a network model through an expression skeleton based on three-dimensional coordinates of a plurality of feature points of the face of the host, and controlling the expression of the virtual character.
After the motion adjustment of the virtual character is completed, the expression of the virtual character is controlled based on the three-dimensional coordinates of a plurality of feature points of the face of the host.
And controlling the expression of the virtual character through the expression skeleton redirection network model. The specific implementation mode is as follows:
firstly, determining three-dimensional coordinate offset of a plurality of characteristic points of a main broadcasting face relative to root bone points and bone included angles between bones between any two characteristic points and corresponding upper-level bones in a first preset three-dimensional coordinate system.
In one embodiment of the application, the root skeletal point is the skull point of the anchor body. Taking the skull points as root bone points, calculating three-dimensional coordinate offset of each characteristic point of the anchor face relative to the root bone points in a first preset three-dimensional coordinate system by the following method:
S n (x n -x 0 ，y n -y 0 ，z n -z 0 )，n∈(1，N)
wherein the coordinates of the root skeleton point are V 0 (x 0 ，y 0 ，z 0 ) The coordinates of any feature point of the anchor face are V n (x n ，y n ，z n ). N is the total number of feature points of the anchor face.
It should be noted that, according to actual needs, those skilled in the art may select other skeletal points of the anchor body as root skeletal points of several feature points of the anchor face, which is not limited in the embodiment of the present application.
In one embodiment of the application, the bone included angle between any two adjacent feature points in a plurality of feature points of the anchor face and the corresponding bone of the upper level is determined by the following formula:
α＝(α 1 ，α 2 ，α 3 )＝(x 1 -x 2 ，y 1 -y 2 ，z 1 -z 2 )
β＝(β 1 ，β 2 ，β 3 )＝(x 3 -x 2 ，y 3 -y 2 ，z 3 -z 2 )
wherein ,(x1 ，y 1 ，z 1 ) Three-dimensional coordinates of the first feature point; (x) 2 ，y 2 ，z 2 ) Three-dimensional coordinates of the second feature points; (x) 3 ，y 3 ，z 3 ) Is the three-dimensional coordinates of the third feature point.
Further, α represents three-dimensional coordinates of the first bone between the first feature point and the second feature point; beta represents the three-dimensional coordinates of the second bone between the second feature point and the third feature point; r represents a unit vector corresponding to the first skeleton; θ represents the bone angle between the second bone and the first bone;
further, the first feature point is adjacent to the second feature point, and the second feature point is adjacent to the third feature point.
In another embodiment of the present application, after determining bone angles corresponding to a plurality of feature points of a anchor face, converting the bone angle θ into a quaternion Q by the following formula;
Wherein θ= (θ) 0 ，θ 1 ，θ 2 )；Q＝(Q 0 ，Q 1 ，Q 2 ，Q 3 )。
Then, determining three-dimensional coordinates of a plurality of feature points of the face of the virtual character in a second preset three-dimensional coordinate system; and calculating three-dimensional coordinate offset of each characteristic point of the face of the virtual character relative to the root skeleton point and skeleton angles between any two adjacent characteristic points and corresponding upper-level skeletons in a preset mode.
In one embodiment of the application, the root skeletal points of the feature points of the face of the avatar are the skull points of the body of the avatar.
Further, determining three-dimensional coordinate offset of a plurality of feature points of the anchor face relative to a root bone point in a first preset three-dimensional coordinate system, and bone included angles between any two adjacent feature points and corresponding upper-level bones; and the three-dimensional coordinate offset of the feature points of the face of the virtual character in the second preset three-dimensional coordinate system relative to the root skeleton points, and the skeleton included angles between any two adjacent feature points and the corresponding upper skeleton, and then inputting all the three-dimensional coordinate offset, the skeleton included angles and the three-dimensional coordinates corresponding to the feature points of the face of the virtual character into the expression skeleton redirection network model to obtain target three-dimensional coordinates of the feature points of the face of the virtual character in the second preset three-dimensional coordinate system. The internal structure of the expressive bone redirecting network model is shown in fig. 7.
Fig. 7 is a schematic diagram of an expression skeleton redirection network model according to an embodiment of the present application. As shown in fig. 7, the expressive bone redirection network model mainly includes a 3-layer structure. The method comprises the following steps: the system comprises a data input layer, a convolution layer and a target three-dimensional coordinate output layer.
The data input layer is used for shifting three-dimensional coordinates of a plurality of feature points of the anchor face relative to root bone points in a first preset three-dimensional coordinate system and bone included angles between bones between any two adjacent feature points and corresponding upper bones; and the three-dimensional coordinate offset of the three-dimensional coordinates of the plurality of feature points of the face of the virtual character under the second preset three-dimensional coordinate system relative to the root bone points and the bone included angles between bones between any two adjacent feature points and corresponding upper bones are input into the expression bone redirection network model. The convolution layer is used for carrying out convolution, filling, sampling and nonlinear operation on the input data. The target three-dimensional coordinate output layer integrates the characteristic data output by the convolution layer, and outputs target three-dimensional coordinates of a plurality of characteristic points of the face of the virtual character under a second preset three-dimensional coordinate system after nonlinear transformation operation.
In one embodiment of the application, in the expression skeleton redirection network model, three-dimensional coordinate offsets and skeleton angle quaternions respectively corresponding to a plurality of feature points of the faces of the anchor and the virtual characters are spliced into tensors to be used as input. The input of the method comprises two input branches, wherein the two branches perform characteristic extraction on input data through Convolution Convolition, batch normalization Batchnormal and activation function Rule for a plurality of times; and then fusing the feature graphs obtained by the two branches, and performing Convolution Convolvulation, batch normalization Batchnormal and activation function Rule for a plurality of times to finally obtain target three-dimensional coordinates of a plurality of feature points of the face of the virtual character under a second preset three-dimensional coordinate system.
Further, a plurality of feature points of the face of the virtual character are adjusted to the positions of the target three-dimensional coordinates so as to control the expression of the virtual character.
Thus, the actions and expressions of the virtual roles are controlled through the neural network model. And displaying the actions and the expressions of the adjusted virtual roles in a live broadcast interface, thereby realizing the virtual live broadcast process.
It should be noted that the above steps for controlling the actions and expressions of the virtual characters are merely a description manner of the embodiment of the present application, and are not limited to the execution sequence. It can be clear to those skilled in the art that in practical applications, the actions and expressions of the virtual character can be controlled simultaneously; or the expression and the action can be controlled first and then displayed simultaneously, and the embodiment of the application is not limited to this.
It should be further noted that, in the embodiment of the present application, the first preset three-dimensional coordinate system and the second preset three-dimensional coordinate system are only three-dimensional coordinate systems proposed for the anchor and the virtual character, so that when the anchor and the virtual character perform motion transformation or expression change under the corresponding coordinate systems, three-dimensional coordinates of a plurality of bone points of the anchor can be obtained in real time. It can be clear to those skilled in the art that in the actual application process, the origin of the three-dimensional coordinate system can be properly adjusted according to the requirement, and only the three-dimensional coordinates of a plurality of bone points under the coordinate system can be obtained when the anchor performs actions or expression changes. Accordingly, the embodiment of the present application is not limited thereto.
Based on the same inventive concept, the embodiment of the application also provides a wearable virtual live broadcast device based on a common camera, and the internal structure schematic diagram of the wearable virtual live broadcast device is shown in fig. 8.
Fig. 8 is a schematic diagram of an internal structure of a wearable virtual live broadcast device based on a common camera according to an embodiment of the present application. As shown in fig. 8, the apparatus includes a processor 801; and a memory 802 having executable code stored thereon that, when executed, causes the processor 801 to perform a generic camera-based wearable avatar method as described above.
In one embodiment of the application, the processor 801 is configured to acquire two-dimensional planar image data associated with a host in real time; the method comprises the steps of inputting two-dimensional plane image data into a skeleton point three-dimensional coordinate recognition network model to recognize a plurality of joint points of a body and a hand of a host, and a plurality of characteristic points of a face of the host, and obtaining three-dimensional coordinates in a first preset three-dimensional coordinate system; the virtual character control system comprises a virtual character model, a virtual skeleton model, a virtual character model and a virtual character model, wherein the virtual character model is used for representing the virtual character model, and the virtual character model is used for representing the virtual character model according to the virtual character model; and the three-dimensional coordinates are used for controlling the expression of the virtual character through redirecting the network model through the expression skeleton based on the three-dimensional coordinates corresponding to the feature points of the anchor face.
In another embodiment of the present application, the processor 801 is configured to identify a network model according to three-dimensional coordinates of skeletal points, and determine three-dimensional coordinates of a plurality of feature points of a face of a host on the basis of two-dimensional planar image data related to the host in a first preset three-dimensional coordinate system; the three-dimensional coordinate offset of a plurality of characteristic points of the anchor face relative to the root skeleton points is calculated under a first preset three-dimensional coordinate system; the method is also used for calculating the bone included angle between the bone between any two adjacent feature points and the corresponding bone of the upper stage; the method is also used for determining three-dimensional coordinates of a plurality of feature points of the face of the virtual character in a second preset three-dimensional coordinate system, and calculating three-dimensional coordinate offset of the feature points of the face of the virtual character relative to root bone points and bone included angles between bones between any two adjacent feature points and corresponding upper bones; the three-dimensional coordinate system is also used for determining target three-dimensional coordinates of the plurality of feature points of the face of the virtual character under a second preset three-dimensional coordinate system through the expression skeleton redirection network model based on the three-dimensional coordinate offset and the skeleton included angle corresponding to the plurality of feature points of the face of the anchor character and the three-dimensional coordinate offset and the skeleton included angle corresponding to the plurality of feature points of the face of the virtual character; and the method is also used for adjusting a plurality of characteristic points of the face of the virtual character to the target three-dimensional coordinate position so as to control the expression of the virtual character.
Based on the same inventive concept, the embodiment of the application also provides a wearable virtual live broadcast device based on a common camera, and the internal structure schematic diagram of the wearable virtual live broadcast device is shown in fig. 9.
Fig. 9 is a schematic diagram of an internal structure of a wearable virtual live broadcast device based on a common camera according to an embodiment of the present application. As shown in fig. 9, the apparatus includes: an acquisition module 901, configured to acquire two-dimensional plane image data related to a anchor in real time; the recognition module 902 is configured to input two-dimensional plane image data into a bone point three-dimensional coordinate recognition network model, so as to recognize a plurality of joints of a body and a hand of a host, and three-dimensional coordinates of a plurality of feature points of a face of the host in a first preset three-dimensional coordinate system; the control module 903 is configured to control an action of the virtual character by redirecting the network model through an action skeleton based on three-dimensional coordinates corresponding to a plurality of joint points of the body and the hand of the anchor; the control module 903 is further configured to control an expression of the virtual character based on three-dimensional coordinates corresponding to the feature points of the anchor face respectively through an expression skeleton redirection network model.
Based on the same inventive concept, the embodiment of the application also provides a storage medium, which is a nonvolatile computer readable storage medium; the non-transitory computer readable storage medium stores at least one program, each program comprising instructions that, when executed by a device having a processor, cause the device to perform a wearable virtual live method based on a generic camera as described above.
According to the wearable virtual live broadcast method and device based on the common camera, in the live broadcast process of a host, one or more image acquisition devices acquire two-dimensional plane images related to the host, and the actions and the expressions of virtual characters displayed in a live broadcast interface are adjusted according to the acquired plane images; according to the embodiment of the application, the three-dimensional coordinates of a plurality of bone points of the anchor are obtained through the neural network model, and then the anchor actions and expressions are redirected to the virtual roles through the deep learning neural network model according to a plurality of joint points of the body and the hand of the anchor and a plurality of characteristic points of the face of the anchor, so that the actions and expressions displayed by the virtual roles are more similar to those of the anchor, and the accuracy of the virtual roles in displaying the actions and expressions is ensured. Meanwhile, when the virtual live broadcast method or the virtual live broadcast equipment provided by the embodiment of the application is used for carrying out virtual live broadcast, a host broadcast does not need to wear hardware equipment, the burden of the host broadcast during live broadcast is reduced, the comfort and convenience of the host broadcast are ensured, the problem that the live broadcast duration is limited due to the fact that the hardware equipment needs to be charged is avoided, and the requirement of the host broadcast for long-time live broadcast is met.
The embodiments of the present application are described in a progressive manner, and the same and similar parts of the embodiments are all referred to each other, and each embodiment is mainly described in the differences from the other embodiments. In particular, for the apparatus, device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, as relevant to see the section description of the method embodiments.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.

Claims (6)

1. The wearable virtual live broadcasting method based on the common camera is characterized by comprising the following steps of:
identifying a network model according to the three-dimensional coordinates of the bone points, and determining the three-dimensional coordinates of a plurality of characteristic points of the face of the anchor under a first preset three-dimensional coordinate system based on two-dimensional plane image data related to the anchor; wherein the bone point three-dimensional coordinate recognition network model comprises a three-dimensional coordinate output layer; the three-dimensional coordinate output layer is used for receiving the normalized Gaussian heat map and outputting three-dimensional coordinates of a plurality of characteristic points of the anchor face under a first preset three-dimensional coordinate system through the following formula:
wherein ,coordinates of an x-axis representing the nth feature point; />Coordinates of a y-axis representing the nth feature point; />Coordinates of a z-axis representing the nth feature point; />Normalized Gaussian heat maps corresponding to the nth feature points; w is->Is a width of (2); h isIs of a height of (2); d is->Is a depth of (2);
calculating three-dimensional coordinate offset of a plurality of feature points of the anchor face relative to root skeleton points under a first preset three-dimensional coordinate system; and calculating the bone included angle between the bone between any two adjacent feature points and the corresponding bone of the previous stage;
In a second preset three-dimensional coordinate system, determining three-dimensional coordinates of a plurality of characteristic points of the face of the virtual character; calculating three-dimensional coordinate offset of a plurality of feature points of the face of the virtual character relative to the root skeleton points and skeleton included angles between skeletons between any two adjacent feature points and corresponding upper-level skeletons;
determining target three-dimensional coordinates of the plurality of feature points of the face of the virtual character under a second preset three-dimensional coordinate system through an expression bone redirection network model based on the three-dimensional coordinate offset and the bone included angle corresponding to the plurality of feature points of the face of the anchor character and the three-dimensional coordinate offset and the bone included angle corresponding to the plurality of feature points of the face of the virtual character; the expression skeleton redirection network model comprises a data input layer, a convolution layer and a target three-dimensional coordinate output layer; the data input layer is used for inputting three-dimensional coordinate offsets and skeleton angles corresponding to a plurality of feature points of the anchor face and three-dimensional coordinate offsets and skeleton angles corresponding to a plurality of feature points of the virtual character face into the expression skeleton redirection network model; the convolution layer is used for receiving output data of the data input layer and carrying out convolution, filling and sampling operations on the output data so as to obtain characteristic data corresponding to the output data; the target three-dimensional coordinate output layer is used for receiving the characteristic data, carrying out nonlinear transformation on the characteristic data, and outputting target three-dimensional coordinates of a plurality of characteristic points of the face of the virtual character under a second preset three-dimensional coordinate system;
Adjusting a plurality of feature points of the face of the virtual character to the target three-dimensional coordinate position so as to control the expression of the virtual character;
the method further comprises the steps of:
identifying a network model according to the three-dimensional coordinates of the skeleton points, and determining the three-dimensional coordinates of a plurality of joint points of the body and the hand of the anchor under a first preset three-dimensional coordinate system; calculating three-dimensional coordinate offset of a plurality of joint points of the body and the hand of the anchor relative to root skeleton points under a first preset three-dimensional coordinate system; and calculating a bone included angle between bones between any two adjacent joint points and corresponding bones at the upper level; wherein the root skeletal points are pelvic bone points of the anchor body; in a second preset three-dimensional coordinate system, determining three-dimensional coordinates of a plurality of joints of the body and the hand of the virtual character, and calculating three-dimensional coordinate offset of the joints of the body and the hand of the virtual character relative to a root bone point and bone included angles between bones between any two adjacent joints and corresponding upper bones under the second preset three-dimensional coordinate system; the first preset three-dimensional coordinate system is a right-hand coordinate system established by taking a pelvic bone point of the anchor body as an origin; the second preset three-dimensional coordinate system is a right-hand coordinate system established by taking a pelvic bone point of the virtual character body as an origin; determining target three-dimensional coordinates of a plurality of joints of the body and the hand of the virtual character under a second preset three-dimensional coordinate system through an action skeleton redirection network model based on three-dimensional coordinate offsets and skeleton angles corresponding to the plurality of joints of the body and the hand of the anchor and three-dimensional coordinate offsets and skeleton angles corresponding to the plurality of joints of the body and the hand of the virtual character; adjusting a plurality of joint points of the body and the hand of the virtual character to the target three-dimensional coordinate position so as to control the action of the virtual character;
The action skeleton redirection network model comprises a data input layer, a convolution layer and a target three-dimensional coordinate output layer; the data input layer is used for inputting three-dimensional coordinate offset and skeleton included angles corresponding to a plurality of joints of the body and the hand of the anchor, and three-dimensional coordinate offset and skeleton included angles corresponding to a plurality of joints of the body and the hand of the virtual character into the action skeleton redirection network model; the convolution layer is used for receiving output data of the data input layer and carrying out convolution, filling and sampling operations on the output data so as to obtain characteristic data corresponding to the output data; the target three-dimensional coordinate output layer is used for receiving the characteristic data, carrying out nonlinear transformation on the characteristic data, and outputting target three-dimensional coordinates of a plurality of joints of the body and the hand of the virtual character under a second preset three-dimensional coordinate system.
2. The wearable virtual live broadcasting method based on the common camera according to claim 1, wherein the bone included angle between the bone between any two adjacent feature points of the main broadcasting face and the corresponding upper bone is determined by the following formula:
α＝(α 1 ,α 2 ,α 3 )＝(x 1 -x 2 ,y 1 -y 2 ,z 1 -z 2 )
β＝(β 1 ,β 2 ,β 3 )＝(x 3 -x 2 ,y 3 -y 2 ,z 3 -z 2 )
wherein ,(x1 ,y 1 ,z 1 ) Three-dimensional coordinates of the first feature point; (x) 2 ,y 2 ,z 2 ) Three-dimensional coordinates of the second feature points; (x) 3 ,y 3 ,z 3 ) Three-dimensional coordinates of the third feature point;
α represents three-dimensional coordinates of the first bone between the first feature point and the second feature point; beta represents the three-dimensional coordinates of the second bone between the second feature point and the third feature point; r represents a unit vector corresponding to the first skeleton; θ represents the bone angle between the second bone and the first bone;
the first feature point is adjacent to a second feature point, and the second feature point is adjacent to a third feature point.
3. The wearable virtual live broadcast method based on the common camera according to claim 2, wherein the method further comprises:
converting the bone included angle theta into a quaternion Q through the following formula;
wherein θ= (θ) 0 ,θ 1 ,θ 2 )；Q＝(Q 0 ,Q 1 ,Q 2 ,Q 3 )。
4. The method of wearable virtual live broadcast based on a common camera according to claim 1, wherein before identifying a network model according to three-dimensional coordinates of skeletal points, determining three-dimensional coordinates of a plurality of feature points of a main cast face in a first preset three-dimensional coordinate system, the method further comprises:
collecting a plurality of two-dimensional plane image data related to a human body, and constructing a first training data set;
Screening the content of the first training data set, and removing image data which does not contain all the nodes and the characteristic points of the human body to obtain a second training data set;
inputting the second training data into a neural network model to train the neural network model;
training until the output converges, and obtaining the three-dimensional coordinate recognition network model of the skeleton points.
5. The wearable virtual live broadcast method based on the common camera according to claim 4, wherein the bone point three-dimensional coordinate recognition network model comprises a Gaussian heat map layer and a Gaussian heat map normalization layer;
the Gaussian heat map layer is used for receiving the characteristic data output by the convolution layer and obtaining a Gaussian heat map with the dimensions of (N, W, H, D) based on the characteristic data; wherein N is the total number of a plurality of characteristic points of the anchor face, W is the width of the Gaussian heat map, H is the height of the Gaussian heat map, and D is the depth of the Gaussian heat map;
the Gaussian heat map normalization layer is used for normalizing the Gaussian heat map with the size of (N, W, H, D) according to the following formula:
wherein G is a Gaussian heat map;is normalized Gaussian heat map.
6. A wearable virtual live broadcast device based on a common camera, the device comprising:
A processor;
and a memory having executable code stored thereon that, when executed, causes the processor to perform a wearable virtual live method based on a common camera as claimed in any one of claims 1-5.
CN202010739818.0A 2020-07-28 2020-07-28 Wearable virtual live broadcast method and equipment based on common camera Active CN112102451B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010739818.0A CN112102451B (en) 2020-07-28 2020-07-28 Wearable virtual live broadcast method and equipment based on common camera

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010739818.0A CN112102451B (en) 2020-07-28 2020-07-28 Wearable virtual live broadcast method and equipment based on common camera

Publications (2)

Publication Number Publication Date
CN112102451A CN112102451A (en) 2020-12-18
CN112102451B true CN112102451B (en) 2023-08-22

Family

ID=73749813

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010739818.0A Active CN112102451B (en) 2020-07-28 2020-07-28 Wearable virtual live broadcast method and equipment based on common camera

Country Status (1)

Country Link
CN (1) CN112102451B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114602177A (en) * 2022-03-28 2022-06-10 百果园技术(新加坡)有限公司 Action control method, device, equipment and storage medium of virtual role

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105654537A (en) * 2015-12-30 2016-06-08 中国科学院自动化研究所 Expression cloning method and device capable of realizing real-time interaction with virtual character
WO2017133009A1 (en) * 2016-02-04 2017-08-10 广州新节奏智能科技有限公司 Method for positioning human joint using depth image of convolutional neural network
CN109922355A (en) * 2019-03-29 2019-06-21 广州虎牙信息科技有限公司 Virtual image live broadcasting method, virtual image live broadcast device and electronic equipment
CN110570499A (en) * 2019-09-09 2019-12-13 珠海金山网络游戏科技有限公司 Expression generation method and device, computing equipment and storage medium
CN111209861A (en) * 2020-01-06 2020-05-29 浙江工业大学 Dynamic gesture action recognition method based on deep learning

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105654537A (en) * 2015-12-30 2016-06-08 中国科学院自动化研究所 Expression cloning method and device capable of realizing real-time interaction with virtual character
WO2017133009A1 (en) * 2016-02-04 2017-08-10 广州新节奏智能科技有限公司 Method for positioning human joint using depth image of convolutional neural network
CN109922355A (en) * 2019-03-29 2019-06-21 广州虎牙信息科技有限公司 Virtual image live broadcasting method, virtual image live broadcast device and electronic equipment
CN110570499A (en) * 2019-09-09 2019-12-13 珠海金山网络游戏科技有限公司 Expression generation method and device, computing equipment and storage medium
CN111209861A (en) * 2020-01-06 2020-05-29 浙江工业大学 Dynamic gesture action recognition method based on deep learning

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
李红波 ; 孙舶源 ; 李双生 ; .基于骨骼信息的虚拟角色控制方法.重庆邮电大学学报(自然科学版).2016,(01),全文. *

Also Published As

Publication number Publication date
CN112102451A (en) 2020-12-18

Similar Documents

Publication Publication Date Title
US20200380594A1 (en) Virtual try-on system, virtual try-on method, computer program product, and information processing device
KR101711736B1 (en) Feature extraction method for motion recognition in image and motion recognition method using skeleton information
KR101323966B1 (en) A system and method for 3D space-dimension based image processing
CN109815776B (en) Action prompting method and device, storage medium and electronic device
CN103578135A (en) Virtual image and real scene combined stage interaction integrating system and realizing method thereof
KR101639161B1 (en) Personal authentication method using skeleton information
CN108153421B (en) Somatosensory interaction method and device and computer-readable storage medium
JP2019096113A (en) Processing device, method and program relating to keypoint data
KR20170030633A (en) Gait analysis system and gait analysis program
CN112734632A (en) Image processing method, image processing device, electronic equipment and readable storage medium
CN203630822U (en) Virtual image and real scene combined stage interaction integrating system
CN111401340B (en) Method and device for detecting motion of target object
JP2017037424A (en) Learning device, recognition device, learning program and recognition program
CN112102451B (en) Wearable virtual live broadcast method and equipment based on common camera
WO2022174574A1 (en) Sensor-based bare-hand data annotation method and system
CN113033526A (en) Computer-implemented method, electronic device and computer program product
KR102483387B1 (en) Augmented reality content provision method and finger rehabilitation training system for finger rehabilitation training
CN110321008B (en) Interaction method, device, equipment and storage medium based on AR model
CN113569775A (en) Monocular RGB input-based mobile terminal real-time 3D human body motion capture method and system, electronic equipment and storage medium
JP2022095332A (en) Learning model generation method, computer program and information processing device
JP2021026265A (en) Image processing device, image processing program, and image processing method
Panaite et al. Motion Sensors Based Human Arm Pose Estimation
CN114973396B (en) Image processing method, image processing device, terminal equipment and computer readable storage medium
EP4303824A1 (en) System and method for monitoring a body pose of a user
CN116152931B (en) Gesture recognition method and VR system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB03 Change of inventor or designer information

Inventor after: Feng Zhiqiang

Inventor before: Chen Min

CB03 Change of inventor or designer information
GR01 Patent grant
GR01 Patent grant