WO2023232103A1 - Film-watching interaction method and apparatus, and computer-readable storage medium - Google Patents

Film-watching interaction method and apparatus, and computer-readable storage medium Download PDF

Info

Publication number
WO2023232103A1
WO2023232103A1 PCT/CN2023/097743 CN2023097743W WO2023232103A1 WO 2023232103 A1 WO2023232103 A1 WO 2023232103A1 CN 2023097743 W CN2023097743 W CN 2023097743W WO 2023232103 A1 WO2023232103 A1 WO 2023232103A1
Authority
WO
WIPO (PCT)
Prior art keywords
space angle
target
line segment
line
joint point
Prior art date
Application number
PCT/CN2023/097743
Other languages
French (fr)
Chinese (zh)
Inventor
邱玉华
周效军
Original Assignee
咪咕文化科技有限公司
中国移动通信集团有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 咪咕文化科技有限公司, 中国移动通信集团有限公司 filed Critical 咪咕文化科技有限公司
Publication of WO2023232103A1 publication Critical patent/WO2023232103A1/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/34—Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L17/00—Speaker identification or verification techniques
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/435—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/439—Processing of audio elementary streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting

Definitions

  • the present disclosure relates to the field of information technology, and in particular, to a movie viewing interaction method, device and computer-readable storage medium.
  • users can usually only watch the currently playing video resource or post barrages for interaction during the movie viewing process.
  • the current movie viewing design is mainly aimed at the entire audience, and the interaction method is relatively single and less interesting.
  • Embodiments of the present disclosure provide a movie-viewing interaction method, device, and computer-readable storage medium to solve the problem that related movie-viewing interaction methods are relatively single and less interesting.
  • embodiments of the present disclosure provide a movie viewing interaction method, including:
  • the response posture of the target character is displayed in the video frame of the target video.
  • embodiments of the present disclosure also provide an interactive device for watching movies, including:
  • the first acquisition module is used to acquire the user's image while the user is watching the target video
  • a recognition module used to recognize the posture of the target object in the user image
  • a first determination module configured to determine the response posture of the target character in the target video based on the posture of the target object
  • a first adjustment module configured to adjust the response posture of the target part of the target character based on the posture of the target part of the target object
  • the first display module is configured to display the response posture of the target character in the video frame of the target video based on the adjusted response posture of the target part of the target character.
  • embodiments of the present disclosure also provide a computer-readable storage medium.
  • a computer program is stored on the computer-readable storage medium.
  • the computer program is executed by a processor, the above-mentioned interactive method for watching movies is implemented. step.
  • the user image in the process of the user watching the target video, the user image is obtained; the posture of the target object in the user image is identified; and based on the posture of the target object, the target character in the target video is determined.
  • a response gesture based on the gesture of the target part of the target object, adjusting the response gesture of the target part of the target character; based on the adjusted response gesture of the target part of the target character, in the video of the target video
  • the target character's response gesture is displayed on the screen.
  • Figure 1 is a flow chart of a movie viewing interaction method provided by an embodiment of the present disclosure
  • Figure 2 is a schematic diagram of user communication in a movie viewing booth provided by an embodiment of the present disclosure
  • Figure 3 is a schematic diagram of salient target detection provided by an embodiment of the present disclosure.
  • Figure 4 is a schematic diagram of bone detection provided by an embodiment of the present disclosure.
  • Figure 5 is a schematic diagram of continuous action detection provided by an embodiment of the present disclosure.
  • Figure 6 is a schematic diagram of bone key point matching provided by an embodiment of the present disclosure.
  • Figure 7 is a schematic diagram of three-dimensional modeling of gestures provided by an embodiment of the present disclosure.
  • Figure 8 is a schematic diagram of the minimum circumferential sphere projection provided by an embodiment of the present disclosure.
  • Figure 9 is a schematic diagram of hand spatial coordinates provided by an embodiment of the present disclosure.
  • Figure 10 is a schematic diagram of the spatial angle provided by an embodiment of the present disclosure.
  • Figure 11 is a schematic diagram of the gesture interaction effect provided by an embodiment of the present disclosure.
  • Figure 12 is a schematic diagram of an auxiliary spectrum diagram provided by an embodiment of the present disclosure.
  • Figure 13 is a structural diagram of a movie viewing interactive device provided by an embodiment of the present disclosure.
  • Figure 1 is a flow chart of a movie viewing interaction method provided by an embodiment of the present disclosure. As shown in Figure 1, it includes the following steps:
  • Step 101 Obtain the user's image while the user is watching the target video.
  • the above target video can be any video currently watched by the user.
  • the user can be supported to interact with the body or expression of the character in the video, that is, the user can perform certain actions or expressions, and the characters in the video can respond accordingly. For example, when a movie-watching user makes a hug gesture, the character in the video responds with a hug, the movie-watching user raises his right hand to shake hands, and the character in the video raises his right hand in response, etc.
  • the user's image can be obtained while the user is watching the target video.
  • the user's image can be collected and uploaded in real time through the camera.
  • the method further includes:
  • the step 101 includes:
  • the user image is acquired.
  • users can be supported to create a viewing room while watching videos, and members in the viewing room can interact in the viewing room.
  • the user can set whether to enable the viewing room function after logging into the video application. If turned on, it means that the user has the permission to create a viewing room. Otherwise, like ordinary viewers, they cannot create a viewing room. Secondly, you can set whether to accept other viewing room invitations. If it is turned on, the viewing room invitations of other users pushed by the system will be allowed, otherwise the viewing room invitations of other users will be blocked.
  • the barrage When user A clicks on any video to watch, the barrage floats on the screen. At this time, user A can click on an interesting barrage to initiate an invitation to create a viewing room.
  • the invitation message is sent directly to the other party's account through the video application, that is, the barrage is posted. User Account. If invited user B is not online during this period, the system can directly send a prompt to user A; if user B is online, user B can choose to accept or reject user A's invitation. If user B accepts the invitation, the viewing room is successfully constructed. At this time, there are user A and user B in the viewing room, and other users who have not received the invitation cannot enter the room.
  • the user who creates the viewing room is the administrator.
  • Invitations sent by other users must be approved by the administrator.
  • the invitation information sent is directly pushed by the video platform to the invited users.
  • Users come to the viewing room to watch videos, realizing the need for simultaneous viewing in the room.
  • Indoor members can initiate interactions, and all messages are only visible to indoor members. When the movie viewing room is disbanded, all messages will be cleared, improving the privacy of movie viewing topics.
  • the schematic diagram of user communication in the movie viewing booth is shown in Figure 2. That is, the client user submits the message to the server, and the server completes the message interaction and displays it on the video playback interface.
  • Members in the viewing booth can interact physically while watching the video, and the scene can be brought into interaction and video rendering based on barrage.
  • users can create a viewing cabin for group viewing, and can interact privately in the viewing cabin, which improves the fun of movie-watching interaction and the privacy of group viewing.
  • Step 102 Identify the posture of the target object in the user image.
  • the target object in the user image can be identified based on the acquired user image, usually a movie-watching user, and the posture of the target object can be determined through image recognition and posture detection, that is, the target object can be identified body movements or expressions.
  • the step 102 includes:
  • a pose of the target object is determined.
  • the target object in the user image can be first identified through a target detection algorithm.
  • the target outline can be drawn through a saliency detection algorithm and returned.
  • Target position result ⁇ x, y, w, h ⁇ , where x and y are the abscissa and ordinate coordinates of the center point of the target rectangular frame, w is the width of the target rectangular frame, and h is the height of the target rectangular frame.
  • the image background contains multiple people and buildings.
  • the recognized salient targets are shown in (b) of Figure 3.
  • the salient targets are within the rectangular frame.
  • a skeletal model can be constructed for the target character outline in the rectangular frame.
  • a skeletal model of the target object can be constructed based on key points of key parts of the target object within the rectangular frame.
  • the detection results on the original image are shown in (a) of Figure 4, and the skeleton model abstracted from the original image is shown in (b) of Figure 4.
  • the model contains a total of 15 key points. They correspond to 15 key parts of the human body, such as point p1 for the head, point p2 for the right hand, and point p15 for the left foot.
  • the posture of the target object can be determined based on the skeletal model of the target object. For example, according to the skeletal model shown in (b) of FIG. 4 , it can be determined that the character posture is a standing posture with limbs extended.
  • the specific action of the target object can be determined by combining lens transitions and continuity behaviors.
  • the skeleton model of the continuity detection image result of a certain user's video clip is shown in Figure 5.
  • the posture result returned by the algorithm is dancing, and the result is encapsulated ⁇ result:'dance' ⁇ .
  • the posture of the target object in the user's image can be accurately recognized.
  • Step 103 Based on the posture of the target object, determine the response posture of the target character in the target video.
  • the response posture of the target character in the target video can be determined according to the posture of the target object. Specifically, a similar method such as a saliency detection algorithm can be used to determine the response posture of the target character in the target video.
  • the target character usually accounts for the largest area in the video screen. character, and then determine the response gesture of the target character.
  • the response gesture of the target character can be a gesture that is basically the same as the gesture of the target object.
  • the character in the video should also To make a handshake gesture in response to the user's physical interaction, and if it is detected that the user makes a hug gesture, the character in the video also needs to make a hug gesture in response to the user's hug.
  • the method further includes:
  • Skeleton key point matching is performed on the skeletal model of the target object and the skeletal model of the target character to adjust the response posture of the target character.
  • the response posture of the target character can be further adjusted to make the interaction between the movie-watching user and the character in the video more detailed.
  • the skeletal model of the target character in the target video can be obtained.
  • the skeletal model of the target character in the target video can be called to make it perform appropriate body movements, and then the interaction can be ensured through skeletal key point matching.
  • the rationality of the posture is to match the skeletal key points of the target object's skeletal model and the target character's skeletal model so that the corresponding skeletal joint points are located in similar spatial positions, thereby ensuring that the target character poses Basically the same pose as the target object.
  • the user extends his right hand to shake hands
  • the character in the video also needs to extend his right hand, not his left hand.
  • p1 in the bone model of the user watching the video on the left corresponds to B1 in the bone model of the character in the video on the right
  • p2 corresponds to B2, etc.
  • Step 104 Adjust the response posture of the target part of the target character based on the posture of the target part of the target object.
  • the postures of the viewing user and the characters in the video can also be adjusted to determine the corresponding interactive postures.
  • the response posture of the target part of the target character can be adjusted based on the posture of the target part of the target object, so that the target character makes a gesture that echoes the target object, so that the response posture of the target part of the target character can be adjusted based on the posture of the target part of the target object.
  • the adjusted posture of the target part of the target object displays the response posture of the target character in the video screen, and can highlight the target of the target character. part's response.
  • Step 105 Based on the adjusted response posture of the target part of the target character, display the response posture of the target character in the video frame of the target video.
  • the response gesture of the target character can be displayed in the video screen of the target video, that is, a display effect of the target character's response gesture is generated in the video screen, so as to realize the interaction between the viewing user and the character in the video. of interaction.
  • the response gesture of the target part of the target character can be highlighted to generate a vivid interactive effect for watching movies.
  • the step 105 includes:
  • the interactive effect of the target part of the target object and the target part of the target character is displayed in the video screen .
  • corresponding interactive postures can also be determined for detailed parts of the movie-watching user and the characters in the video, such as gestures and facial expressions.
  • the response posture of the target part of the target character is adjusted, so that the target character makes a gesture that echoes the target object and displays it in the video screen.
  • the interaction effect between the target part of the target object and the target part of the target character is displayed in Respond to the gesture and form an interactive gesture of the target parts of the two.
  • the user when the user raises his right hand, he can also raise his right hand to the character in the video, and the user's raised hand and the raised hand touch of the character in the video are displayed on the video screen, forming an interactive effect of the user and the character in the video touching each other across the screen;
  • the user when the user makes a face expression, he or she can also make a face expression on the character in the video, creating the effect of the user interacting with the character in the video making a face expression.
  • movie-watching users can interact with specific parts of the characters in the video in more detail, thereby improving the interactive experience.
  • the step 104 includes:
  • the response posture of the target part of the target character is adjusted.
  • the minimum circumscribed sphere of the three-dimensional model of the target part of the target object and the minimum circumscribed sphere of the three-dimensional model of the target part of the target character can be calculated first, that is, the minimum circumscribed sphere that can completely surround the target part.
  • the target part such as a gesture can be three-dimensionally modeled, and then the minimum circumscribed sphere of the three-dimensional model surrounding the target part is calculated.
  • the minimum circumscribed sphere of the character's target part model in the video is V1
  • the minimum circumscribed sphere of the user's target part model is V2
  • the centers of the two spheres are M1 and M2 respectively.
  • the minimum circumscribed ball V2 can be projected into the video picture of the target video according to the center coordinate M2 of the minimum circumscribed ball V2, that is, the three-dimensional model of the target part of the target object can be projected into the video picture. displayed in the video screen, and then the coordinate position of the minimum external ball V1 can be adjusted according to the position of the minimum external ball V2 projected in the video screen, so that the position of the minimum external ball V1 and the position of the minimum external ball V2 are spatially aligned. Furthermore, the response posture of the target part of the target character can be adaptively adjusted according to the adjusted position of the minimum external ball V1.
  • the posture of the target part of the target object can be mapped to the video screen display, and by adjusting the response posture of the target part of the target character, the two can interact in the video screen attitude.
  • adjusting the position of the second minimum circumscribed ball according to the projected position of the first minimal circumscribed ball in the video screen includes:
  • the response posture of the target part of the target character is adjusted according to the adjusted center position of the second minimum external ball and the positions of each point in the target part of the target character.
  • the center coordinate position of the minimum circumscribed ball V1 can be adjusted according to the center coordinate position M2' of the minimum circumscribed ball V2 projected in the video screen, such as aligning the coordinate positions of the two sphere centers.
  • the response posture of the target part of the target character can be adjusted accurately and carefully.
  • the target part is the hand
  • the step 104 includes:
  • a three-dimensional coordinate system is established with the first palm heel node of the target object's hand as the origin, wherein the target object's hand includes a thumb joint point, an index finger joint point, a middle finger joint point, a ring finger joint point and a little finger joint point. point;
  • the first space angle is based on the first palm heel node as the vertex and the first line segment and the second line segment is the space angle of the side
  • the second space angle is the space angle with the first palm heel node as the vertex and the second line segment and the third line segment as the side
  • the third space angle is The space angle between the first palm heel node as a vertex and the third line segment and the fourth line segment as sides
  • the fourth space angle is a space angle with the first palm heel node as a vertex and the fourth line segment as an edge.
  • the line segment and the fifth line segment are the spatial angles of the sides; the first line segment is the line segment between the first palm heel node and the adjacent thumb joint point, and the second line segment is the first palm heel node. and the adjacent index finger joint point, the third line segment is the line segment between the first palm heel node and the adjacent middle finger joint point, and the fourth line segment is the line segment between the first palm heel node and the adjacent middle finger joint point.
  • the line segment of the ring finger joint, the fifth line segment is the first palm heel joint A line segment between a point and the adjacent little finger joint point;
  • first line-plane space angle is based on the target object The space angle between the line segment where the thumb joint point of the hand is located and the first surface, and the second line-plane space angle is the space angle between the line segment where the index finger joint point of the target object's hand is located and the first surface.
  • the space angle, the third line-plane space angle is the space angle between the line segment where the middle finger joint point of the target object's hand is and the first surface
  • the fourth line-plane space angle is the space angle between the The angle between the line segment where the joint point of the ring finger of the target object's hand is located and the first plane
  • the fifth line-plane space angle is the line segment where the joint point of the little finger of the target object's hand is located and the space angle between the line segment and the first plane.
  • the spatial angle of the first surface; the first surface is the plane formed by the z-axis and the y-axis in the three-dimensional coordinate system;
  • angle and the fifth line-plane space angle adjust the positions of the thumb joint point, index finger joint point, middle finger joint point, ring finger joint point and little finger joint point of the target character's hand.
  • the target part may be a hand
  • the hand movements of the target character may be carefully adjusted based on the hand movements of the target object.
  • a three-dimensional coordinate system can be established with the palm root node q1 of the target object's hand as the origin, and the set points ⁇ q1, q2, q3...q20 ⁇ simulate 20 main joint points of the user's hand.
  • q1 corresponds to the palm root node
  • q6, q7 and q20 correspond to the thumb joint point
  • q5, q8, q18 and q19 correspond to the index finger joint point
  • q4 q9, q16 and q17 correspond to the middle finger joint point
  • q3, q10, q14 and q15 correspond to the joint point of the ring finger
  • q2, q11, q12 and p13 correspond to the joint point of the little finger.
  • ⁇ 1 is the angle in space with q1 as the vertex and line segment q1q6 and line segment q1q5 as sides
  • ⁇ 2 is the space angle with q1 as the vertex and line segment q1q5 and line segment q1q4 as sides
  • ⁇ 3 is the angle in space with q1 as the vertex and line segment q1q5 and line segment q1q4 as sides.
  • q1 is the vertex, the line segment q1q4 and the line segment q1q3 are the sides
  • ⁇ 4 is the space angle with q1 as the vertex, the line segment q1q3 and the line segment q1q2 are the sides.
  • the thumb joint point Q6, index finger joint point Q5, and middle finger joint point of the target character's hand that are closest to the root node of the hand can be determined Q4.
  • the positions of the ring finger joint point Q3 and the little finger joint point Q2 are adjusted so that the positions of the joint points of the hand of the target character correspond to the positions of the joint points of the hand of the target object.
  • the positions of the remaining joint points of the target character's hand can also be adjusted. Specifically, the angles between the line segments q6q7, q5q8, q4q9, q3q10, q2q11 and the surface zq1y can be calculated respectively, which are recorded as ⁇ 1 and ⁇ 2 respectively. , ⁇ 3 , ⁇ 4 , ⁇ 5 .
  • the remaining thumb joint points, index finger joint points, middle finger joint points and ring finger joint points of the target character’s hand can be determined and the positions of the little finger joints, that is, ⁇ Q7, Q8, Q9...Q20 ⁇ , so that the positions of the joint points of the hand of the target character correspond to the positions of the joint points of the hand of the target object, so that the The target character's gestures remain substantially symmetrical to the target object's gestures.
  • the adjusted hand interaction effect is shown in Figure 11.
  • User gestures can form an interactive experience of touching across the screen with the character gestures in the video.
  • the fourth line-plane space angle and the fifth line-plane space angle adjust the positions of the thumb joint point, index finger joint point, middle finger joint point, ring finger joint point and little finger joint point of the target character's hand, including:
  • the fifth space angle be equal to the first space angle
  • the sixth space angle be equal to the second space angle
  • the seventh space angle be equal to the third space angle
  • the eighth space angle be equal to the fourth space angle
  • the sixth line-plane space angle be equal to the first line-plane space angle
  • the seventh line-plane space angle be equal to the second line-plane space angle
  • the eighth line-plane space angle be equal to the third line-plane space angle
  • the ninth line-plane space angle be equal to the third line-plane space angle.
  • the plane space angle is equal to the fourth line plane space angle
  • the tenth line plane space angle is equal to the fifth line plane space angle. Adjust other points of the target character's hand except the first thumb joint point.
  • Thumb joint point, other index finger joint points except the first index finger joint point, other index finger joint points except the first middle finger joint point The positions of the joint points of his middle finger, other ring finger joint points except the first ring finger joint point, and other little finger joint points except the first little finger joint point;
  • the fifth space angle is the space angle with the second palm root node of the target character's hand as the vertex and the sixth line segment and the seventh line segment as the sides
  • the sixth space angle is based on the The space angle between the second palm heel node as a vertex and the seventh line segment and the eighth line segment as sides.
  • the seventh space angle is the second palm heel node as a vertex and the eighth line segment and the eighth line segment as sides.
  • the nine line segments are the space angles of the sides, and the eighth space angle is the space angle with the second palm heel node as the vertex and the ninth line segment and the tenth line segment as the sides; the sixth line segment is the space angle.
  • the eighth line segment is a line segment between the second palm heel node and the adjacent first middle finger joint point, and the ninth line segment is a line segment between the second palm heel node and the adjacent first ring finger joint point.
  • the tenth line segment is a line segment between the second palm heel node and the adjacent first little finger joint point;
  • the sixth line-plane space angle is the space angle between the line segment where the thumb joint point of the target character's hand is located and the second surface
  • the seventh line-plane space angle is the space angle between the target character's hand and the second surface.
  • the eighth line-plane space angle is the space between the line segment where the joint point of the middle finger of the hand of the target character is located and the second surface.
  • the included angle, the ninth line-plane space angle is the space angle between the line segment where the ring finger joint point of the target character's hand is located and the second surface
  • the tenth line-plane space angle is the space angle between the The angle between the line segment where the little finger joint point of the target character's hand is located and the second surface
  • the second surface is the z-axis and y-axis in the three-dimensional coordinate system established with the second palm root node as the origin.
  • the corresponding spatial angle is and Q1 corresponds to the corresponding point q1 in the target object's hand model, Q2 corresponds to the corresponding point q2 in the target object's hand model, Q3 corresponds to the corresponding point q3 in the target object's hand model, and so on.
  • ⁇ 1 corresponds to the character hand model collection points ⁇ Q1, Q2, Q3...Q20 ⁇ in the video
  • Q2 corresponds to the corresponding point q2 in the target object's hand model
  • Q3 corresponds to the corresponding point q3 in the target object's hand model
  • the spatial distance between the current point Q6 and all points in the set G can be calculated sequentially, recorded as L 1 .
  • the original Q6 (x q , y q , z q )
  • the calculation of the spatial distance between point Q6 and point K is as follows:
  • the angle between Q6′Q7 and surface zQ1y can be equal to ⁇ 1 , and a set of vector solutions for Q6′Q7 can be obtained. Subtract the coordinates of Q6′ from the coordinates in the vector to obtain the coordinate values of all possible Q7 points. This set is marked R. Calculate the spatial distance between the coordinates of the original Q7 point and all values in R, and obtain the coordinate point corresponding to the smallest spatial distance value, which is the adjusted coordinate point of Q7, recorded as Q7′.
  • the previous node of Q7 is Q6
  • the previous node of Q8 is Q5
  • the line-plane space angle determines the final adjustment coordinate value by finding the minimum spatial distance. Adjust the values of all remaining coordinate points in turn.
  • the adjusted hand interaction effect is shown in Figure 11.
  • User gestures can form a screen-touch interactive experience with the character gestures in the video.
  • the positions of the joints of the character's hand in the video can be adjusted sequentially, ultimately forming an interactive experience in which the user's gestures and the character's gestures in the video are touched across the screen, increasing the fun of interaction.
  • the method also includes:
  • the user When the user chooses to enter the interpretation mode, display the video dubbing interface, and display an auxiliary spectrogram in the video dubbing interface, wherein the auxiliary spectrogram is a spectrogram generated based on the original sound in the target video;
  • the dubbing audio is integrated into the target video for playback.
  • a user when a user watches a video in a created viewing booth, members of the booth can initiate character dubbing, and the members of the booth complete the character lines in the video, bringing themselves into the role of the plot.
  • voiceprint matching rules can be introduced, which can be used to assist the dubbing of character lines, so that users can have voiceprint reference when performing voice interpretation.
  • the audio in the video can be automatically extracted and a voiceprint schematic diagram, also known as an auxiliary spectrogram, is generated, as shown by the solid line in Figure 12.
  • the video screen is switched out of the movie viewing and the video dubbing interface is entered.
  • the native auxiliary voiceprint that is, the auxiliary spectrogram pops up in the interface.
  • users can refer to the auxiliary dubbing voiceprint to adjust their intonation and speaking speed to better complete the voice interpretation.
  • the voices of multiple people are obtained and integrated into a complete video.
  • the single-player interpretation mode the original voices of a single person and other characters selected by the user or the voices of other users are synthesized to obtain a video dubbed by the user. In this way, direct interaction between users in the viewing booth and between users and videos is achieved, making watching movies more interesting.
  • the method further includes:
  • the step of integrating the dubbing audio into the target video for playback includes:
  • the adjusted dubbing audio is integrated into the target video for playback.
  • voiceprint matching rules can also be introduced to modify the user's voice.
  • the system's user voice improves the voiceprint through automatic voiceprint matching, making the user's voice more suitable for the character.
  • the system can match the recorded voiceprint with the original voiceprint.
  • the voiceprint entered by the user that is, the spectrum
  • the original voiceprint is shown as a solid line.
  • the voice domain value of the original video's voiceprint can be matched by calculating the variance within the voice domain to improve the fit of the user's voice.
  • the method further includes:
  • the step of integrating the dubbing audio into the target video for playback includes:
  • the dubbing audio is integrated into the target video for playback, and the score is displayed.
  • voiceprint matching rules can also be introduced for the user scoring mechanism to score all users' dubbing of the same character, and those with the top scores will be included in the list collection.
  • the difference between each voiceprint point in the user's dubbing audio and the original voiceprint can be calculated.
  • the original voiceprint has n voiceprint points.
  • the difference between the user's voiceprint and the original voiceprint accounts for 0% of the original voiceprint value, one unit point can be obtained.
  • the difference accounts for more than 50% of the original voiceprint value, we get 0 unit points, when the difference accounts for 0 to 50% of the original voiceprint value, it will decrease by 0.2 unit points for every 10 percentage points increase.
  • the unit of each voiceprint point is divided into a set ⁇ , the voiceprint set entered by the user is X, and the original voiceprint set is N. If a user's voiceprint value in the user voiceprint set is x i and the corresponding original voiceprint value is n i , then the calculation formula of the user's total score R is as follows:
  • the score can be displayed in the user's dubbing video.
  • the dubbing score of each user can be displayed in the synthesized dubbing video and sorted by score.
  • the method also includes:
  • the video scene can also be rendered based on the barrages in the viewing booth, the corresponding emotional words are extracted through semantic analysis of the barrages, and special effect pendants, sound effects, etc. are equipped according to different emotional words.
  • the specific implementation method is that users in the viewing booth can post barrages while watching videos; count the number M of barrages within a certain period of time, such as counting the number of barrages within 10 minutes; count each barrage within this period Extract existing emotional words, such as horror, joy, sadness, etc.
  • the extraction method can be keyword matching and semantic understanding in natural language processing algorithms; count the number of emotional words N, and trigger scene rendering when N/M is greater than a certain threshold.
  • Emotional words can be used to automatically add relevant special effects widgets to the video played in the viewing room.
  • Emotional words can also be used as search matching words to obtain and play corresponding sound effects. For example, for happy emotional words, you can add the special effect of falling stars to the video screen, and you can also call up cheerful sound effects for playback.
  • the plot can be made more flexible and the interactive interest further increased.
  • This disclosure proposes a barrage-based interactive method for viewing rooms. Users can choose user groups with the same interests to build movie-watching cabins. For example, if a group of celebrity followers watches movies simultaneously, the viewing privacy in the cabin will be better. Various interactions can also be carried out in the viewing booth, such as physical interaction, scene introduction and barrage rendering, etc., which greatly increases the fun of video watching and transforms the audience from a viewer into a participant in the video interpretation.
  • the movie-viewing interaction method of the embodiment of the present disclosure acquires the user image while the user watches the target video; identifies the posture of the target object in the user image; based on the posture of the target object, Determine the response posture of the target character in the target video; adjust the response posture of the target part of the target character based on the posture of the target part of the target object; based on the adjusted target part of the target character
  • the response gesture displays the response gesture of the target character in the video screen of the target video.
  • FIG. 13 is a structural diagram of an interactive device for watching movies provided by an embodiment of the present disclosure. Since the problem-solving principle of the interactive viewing device is similar to the interactive viewing method in the embodiment of the present disclosure, the implementation of the interactive viewing device can be referred to the implementation of the method, and repeated details will not be repeated.
  • the movie viewing interactive device 1300 includes:
  • the first acquisition module 1301 is used to acquire user images while the user is watching the target video;
  • Recognition module 1302 used to recognize the posture of the target object in the user image
  • the first determination module 1303 is configured to determine the response posture of the target character in the target video based on the posture of the target object;
  • the first adjustment module 1304 is configured to adjust the response posture of the target part of the target character based on the posture of the target part of the target object;
  • the first display module 1305 is configured to display the response gesture of the target character in the video screen of the target video based on the adjusted response gesture of the target part of the target character.
  • the identification module 1302 includes:
  • An identification unit configured to identify the target object in the user image through a target detection algorithm
  • a first determination unit configured to determine the posture of the target object based on the skeletal model of the target object.
  • the interactive viewing device 1300 also includes:
  • the second acquisition module is used to acquire the skeleton model of the target character in the target video
  • a matching module configured to match skeletal key points between the skeletal model of the target object and the skeletal model of the target character to adjust the response posture of the target character.
  • the first display module 1305 is further configured to display the target part in the video screen based on the posture of the target part of the target object and the adjusted response posture of the target part of the target character.
  • the first adjustment module 1304 includes:
  • a second determination unit configured to determine the first minimum circumscribed sphere of the three-dimensional model surrounding the target part of the target object
  • a third determination unit configured to determine the second minimum circumscribed sphere of the three-dimensional model surrounding the target part of the target character
  • a projection unit configured to project the first minimum circumscribed ball into the video picture of the target video according to the center position of the first minimum circumscribed ball
  • a first adjustment unit configured to adjust the position of the second minimum circumscribed ball according to the position of the first minimum circumscribed ball projected in the video screen
  • the second adjustment unit is configured to adjust the response posture of the target part of the target character according to the adjusted position of the second minimum external ball.
  • the first adjustment unit includes:
  • a first adjustment subunit configured to adjust the center position of the second minimum circumscribed ball according to the center position of the first minimum circumscribed ball projected in the video screen
  • Determining subunit used to determine the position of each point in the target part of the target character based on the angle relationship between any two points in the three-dimensional model of the target part of the target character;
  • the second adjustment unit is configured to adjust the response posture of the target part of the target character according to the adjusted center position of the second minimum external ball and the positions of each point in the target part of the target character.
  • the target part is the hand;
  • the first adjustment module 1304 includes:
  • An establishment unit configured to establish a three-dimensional coordinate system with the first palm heel node of the target object's hand as the origin, wherein the target object's hand includes a thumb joint point, an index finger joint point, a middle finger joint point, and a ring finger joint point. joints and little finger joints;
  • the first calculation unit is used to calculate the first spatial angle, the second spatial angle, the third spatial angle and the fourth spatial angle; wherein the first spatial angle is based on the first palm heel node as the vertex and the third spatial angle.
  • the first line segment and the second line segment are the angles in space between the sides, and the second space angle is the angle in space with the first palm heel node as the vertex and the second line segment and the third line segment as the sides, so
  • the third space angle is the space angle with the first palm heel node as the vertex and the third line segment and the fourth line segment as the sides, so
  • the fourth space angle is the space angle with the first palm heel node as the vertex and the fourth line segment and the fifth line segment as the sides; the first line segment is the first palm heel node and the corresponding space angle.
  • the second line segment is the line segment between the first palm heel node and the adjacent index finger joint point
  • the third line segment is the line segment between the first palm heel node and the adjacent middle finger joint.
  • the line segment of the joint point, the fourth line segment is the line segment between the first palm heel node and the adjacent ring finger joint point, the fifth line segment is the line segment between the first palm heel node and the adjacent little finger joint point ;
  • the second calculation unit is used to calculate the first line-plane space angle, the second line-plane space angle, the third line-plane space angle, the fourth line-plane space angle and the fifth line-plane space angle; wherein, the first line-plane space angle The angle is the space angle between the line segment where the thumb joint point of the target object's hand is located and the first plane, and the second line-plane space angle is the line segment where the index finger joint point of the target object's hand is located.
  • the spatial angle between the third line and the first surface is the spatial angle between the line segment where the middle finger joint point of the target object's hand is and the first surface.
  • the fourth line is The surface-space angle is the space angle between the line segment where the ring finger joint point of the target object's hand is located and the first surface
  • the fifth line-surface space angle is the space angle between the little finger joint of the target object's hand.
  • the spatial angle between the line segment where the point is located and the first surface; the first surface is the plane formed by the z-axis and the y-axis in the three-dimensional coordinate system;
  • the third adjustment unit is used to adjust the first space angle, the second space angle, the third space angle and the fourth space angle according to the first line-plane space angle, the second line-plane space angle and the third line-plane space angle.
  • angle, the fourth line-plane space angle and the fifth line-plane space angle and adjust the positions of the thumb joint point, index finger joint point, middle finger joint point, ring finger joint point and little finger joint point of the target character's hand.
  • the third adjustment unit includes:
  • the second adjustment subunit is used to make the fifth space angle equal to the first space angle, the sixth space angle equal to the second space angle, the seventh space angle equal to the third space angle, and the eighth space angle equal to
  • the fourth space angle adjusts the positions of the first thumb joint point, first index finger joint point, first middle finger joint point, first ring finger joint point and first little finger joint point of the target character's hand;
  • the third adjustment subunit is used to make the sixth line-plane space angle equal to the first line-plane space angle, the seventh line-plane space angle equal to the second line-plane space angle, and the eighth line-plane space angle equal to the third line-plane space angle.
  • thumb joint points except the first thumb joint point other index finger joint points except the first index finger joint point, except the above-mentioned first index finger joint point
  • the fifth space angle is the space angle with the second palm root node of the target character's hand as the vertex and the sixth line segment and the seventh line segment as the sides
  • the sixth space angle is based on the The space angle between the second palm heel node as a vertex and the seventh line segment and the eighth line segment as sides
  • the seventh space angle is the second palm heel node as a vertex and the eighth line segment and the eighth line segment as sides
  • the nine line segments are the space angles of the sides
  • the eighth space angle is the space angle with the second palm heel node as the vertex and the ninth line segment and the tenth line segment as the sides
  • the sixth line segment is the space angle.
  • the eighth line segment is a line segment between the second palm heel node and the adjacent first middle finger joint point, and the ninth line segment is a line segment between the second palm heel node and the adjacent first ring finger joint point.
  • the tenth line segment is a line segment between the second palm heel node and the adjacent first little finger joint point;
  • the sixth line-plane space angle is the space angle between the line segment where the thumb joint point of the target character's hand is located and the second surface
  • the seventh line-plane space angle is the space angle between the target character's hand and the second surface.
  • the eighth line-plane space angle is the space between the line segment where the joint point of the middle finger of the hand of the target character is located and the second surface.
  • the included angle, the ninth line-plane space angle is the space angle between the line segment where the ring finger joint point of the target character's hand is located and the second surface
  • the tenth line-plane space angle is the space angle between the The angle between the line segment where the little finger joint point of the target character's hand is located and the second surface
  • the second surface is the z-axis and y-axis in the three-dimensional coordinate system established with the second palm root node as the origin.
  • the movie viewing interactive device 1300 also includes:
  • a receiving module configured to receive input from the first user regarding the first barrage while watching the target video, wherein the input is used to instruct the second user to initiate an invitation to create a viewing room, and the second user is a publisher.
  • a creation module configured to create a viewing cabin for the first user and the second user when the second user accepts the invitation to create the viewing cabin, and play all the viewing cabins in the viewing cabin. Describe the target video;
  • the first acquisition module 1301 is used to obtain the target video when the user watches the target video in the viewing room. During the process, the user image is obtained.
  • the interactive viewing device 1300 also includes:
  • the second display module is used to display the video dubbing interface when the user chooses to enter the interpretation mode, and to display an auxiliary spectrogram in the video dubbing interface, wherein the auxiliary spectrogram is based on the target video. Spectrogram generated from the original sound;
  • the third acquisition module is used to acquire the dubbing audio input by the user according to the auxiliary spectrogram
  • a playback module is used to integrate the dubbing audio into the target video for playback.
  • the interactive viewing device 1300 also includes:
  • the second adjustment module is used to adjust the dubbing audio so that the adjusted dubbing audio matches the auxiliary spectrogram
  • the playback module is used to integrate the adjusted dubbing audio into the target video for playback.
  • the interactive viewing device 1300 also includes:
  • a calculation module configured to calculate the score of the dubbing audio based on the difference between the dubbing audio and each frequency point in the auxiliary spectrogram
  • the playback module is used to integrate the dubbing audio into the target video for playback, and display the score.
  • the interactive viewing device 1300 also includes:
  • the fourth acquisition module is used to obtain the barrages and the number of barrages posted by the user in the first period when watching the target video in the viewing room;
  • the second determination module is used to determine the emotional words corresponding to each barrage released within the first period, and count the number of each emotional word;
  • a processing module configured to add a special effect pendant related to the first emotional word in the viewing room when the ratio of the number of first emotional words to the number of barrages is greater than a preset threshold, or, Play sound effects related to the first emotional word, where the first emotional word is any emotional word.
  • the movie viewing interactive device 1300 provided by the embodiment of the present disclosure can execute the above method embodiments, and its implementation principles and technical effects are similar, and will not be described again in this embodiment.
  • the movie-viewing interactive device 1300 in the embodiment of the present disclosure obtains Obtain the user image; identify the posture of the target object in the user image; determine the response posture of the target character in the target video based on the posture of the target object; based on the posture of the target part of the target object, determine the response posture of the target character in the target video.
  • the response posture of the target part of the target character is adjusted; based on the adjusted response posture of the target part of the target character, the response posture of the target character is displayed in the video screen of the target video.
  • the computer-readable storage medium of the embodiment of the present disclosure is used to store a computer program, and the computer program can be executed by a processor to implement each step in the method embodiment shown in Figure 1.
  • the disclosed methods and devices can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division. In actual implementation, there may be other division methods.
  • multiple units or components may be combined or can be integrated into another system, or some features can be ignored, or not implemented.
  • the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.
  • each functional unit in various embodiments of the present disclosure may be integrated into one processing unit, each unit may be physically included separately, or two or more units may be integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units.
  • the above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium.
  • the above-mentioned software functional unit is stored in a storage medium and includes a number of instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute some steps of the sending and receiving methods described in various embodiments of the present disclosure.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program code. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Signal Processing (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Acoustics & Sound (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Databases & Information Systems (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

The present disclosure relates to the field of information technology. Provided are a film-watching interaction method and apparatus, and a computer-readable storage medium. The method comprises: during the process of a user watching a target video, acquiring a user image (101); identifying the posture of a target object in the user image (102); on the basis of the posture of the target object, determining a response posture of a target character in the target video (103); on the basis of the posture of a target part of the target object, adjusting a response posture of a target part of the target character (104); and on the basis of the adjusted response posture of the target part of the target character, displaying the response posture of the target character in a video picture of the target video (105).

Description

一种观影互动方法、装置及计算机可读存储介质An interactive method, device and computer-readable storage medium for watching movies
相关申请的交叉引用Cross-references to related applications
本申请主张在2022年6月2日在中国提交的中国专利申请号No.202210624394.2的优先权，其全部内容通过引用包含于此。This application claims priority from Chinese Patent Application No. 202210624394.2 filed in China on June 2, 2022, the entire content of which is incorporated herein by reference.
技术领域Technical field
本公开涉及信息技术领域，尤其涉及一种观影互动方法、装置及计算机可读存储介质。The present disclosure relates to the field of information technology, and in particular, to a movie viewing interaction method, device and computer-readable storage medium.
背景技术Background technique
相关技术中，用户在观影过程中通常只能观看当前播放的视频资源，或者发布弹幕进行互动，且目前的观影设计主要是针对全体观众，互动方式较为单一，趣味性较差。In related technologies, users can usually only watch the currently playing video resource or post barrages for interaction during the movie viewing process. Moreover, the current movie viewing design is mainly aimed at the entire audience, and the interaction method is relatively single and less interesting.
发明内容Contents of the invention
本公开实施例提供一种观影互动方法、装置及计算机可读存储介质，以解决相关观影互动方式较为单一，趣味性较差的问题。Embodiments of the present disclosure provide a movie-viewing interaction method, device, and computer-readable storage medium to solve the problem that related movie-viewing interaction methods are relatively single and less interesting.
第一方面，本公开实施例提供了一种观影互动方法，包括：In a first aspect, embodiments of the present disclosure provide a movie viewing interaction method, including:
在用户观看目标视频的过程中，获取用户图像；While the user is watching the target video, the user image is obtained;
识别所述用户图像中的目标对象的姿态；identifying the pose of the target object in the user image;
基于所述目标对象的姿态，确定所述目标视频中的目标角色的回应姿态；Based on the posture of the target object, determine the response posture of the target character in the target video;
基于所述目标对象的目标部位的姿态，对所述目标角色的目标部位的回应姿态进行调整；Based on the posture of the target part of the target object, adjust the response posture of the target part of the target character;
基于调整后的所述目标角色的目标部位的回应姿态，在所述目标视频的视频画面中显示所述目标角色的回应姿态。Based on the adjusted response posture of the target part of the target character, the response posture of the target character is displayed in the video frame of the target video.
第二方面，本公开实施例还提供一种观影互动装置，包括：In a second aspect, embodiments of the present disclosure also provide an interactive device for watching movies, including:
第一获取模块，用于在用户观看目标视频的过程中，获取用户图像；The first acquisition module is used to acquire the user's image while the user is watching the target video;
识别模块，用于识别所述用户图像中的目标对象的姿态； A recognition module, used to recognize the posture of the target object in the user image;
第一确定模块，用于基于所述目标对象的姿态，确定所述目标视频中的目标角色的回应姿态；A first determination module, configured to determine the response posture of the target character in the target video based on the posture of the target object;
第一调整模块，用于基于所述目标对象的目标部位的姿态，对所述目标角色的目标部位的回应姿态进行调整；A first adjustment module, configured to adjust the response posture of the target part of the target character based on the posture of the target part of the target object;
第一显示模块，用于基于调整后的所述目标角色的目标部位的回应姿态，在所述目标视频的视频画面中显示所述目标角色的回应姿态。The first display module is configured to display the response posture of the target character in the video frame of the target video based on the adjusted response posture of the target part of the target character.
第三方面，本公开实施例还提供一种计算机可读存储介质，所述计算机可读存储介质上存储计算机程序，所述计算机程序被处理器执行时实现如上所述的观影互动方法中的步骤。In a third aspect, embodiments of the present disclosure also provide a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the above-mentioned interactive method for watching movies is implemented. step.
在本公开实施例中，在用户观看目标视频的过程中，获取用户图像；识别所述用户图像中的目标对象的姿态；基于所述目标对象的姿态，确定所述目标视频中的目标角色的回应姿态；基于所述目标对象的目标部位的姿态，对所述目标角色的目标部位的回应姿态进行调整；基于调整后的所述目标角色的目标部位的回应姿态，在所述目标视频的视频画面中显示所述目标角色的回应姿态。这样，用户可以与视频中角色进行互动，使得互动方式不限于弹幕，增加了互动的趣味性。In an embodiment of the present disclosure, in the process of the user watching the target video, the user image is obtained; the posture of the target object in the user image is identified; and based on the posture of the target object, the target character in the target video is determined. A response gesture; based on the gesture of the target part of the target object, adjusting the response gesture of the target part of the target character; based on the adjusted response gesture of the target part of the target character, in the video of the target video The target character's response gesture is displayed on the screen. In this way, users can interact with the characters in the video, so that the interaction method is not limited to barrages, making the interaction more interesting.
附图说明Description of the drawings
为了更清楚地说明本公开实施例的技术方案，下面将对本公开实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本公开的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed to be used in the description of the embodiments of the present disclosure will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting any creative effort.
图1是本公开实施例提供的观影互动方法的流程图；Figure 1 is a flow chart of a movie viewing interaction method provided by an embodiment of the present disclosure;
图2是本公开实施例提供的观影小室内用户交流的示意图；Figure 2 is a schematic diagram of user communication in a movie viewing booth provided by an embodiment of the present disclosure;
图3是本公开实施例提供的显著性目标检测示意图；Figure 3 is a schematic diagram of salient target detection provided by an embodiment of the present disclosure;
图4是本公开实施例提供的骨骼检测示意图；Figure 4 is a schematic diagram of bone detection provided by an embodiment of the present disclosure;
图5是本公开实施例提供的连续性动作检测示意图；Figure 5 is a schematic diagram of continuous action detection provided by an embodiment of the present disclosure;
图6是本公开实施例提供的骨骼关键点匹配的示意图；Figure 6 is a schematic diagram of bone key point matching provided by an embodiment of the present disclosure;
图7是本公开实施例提供的手势三维建模示意图； Figure 7 is a schematic diagram of three-dimensional modeling of gestures provided by an embodiment of the present disclosure;
图8是本公开实施例提供的最小外接球投影示意图；Figure 8 is a schematic diagram of the minimum circumferential sphere projection provided by an embodiment of the present disclosure;
图9是本公开实施例提供的手部空间坐标示意图；Figure 9 is a schematic diagram of hand spatial coordinates provided by an embodiment of the present disclosure;
图10是本公开实施例提供的空间夹角示意图；Figure 10 is a schematic diagram of the spatial angle provided by an embodiment of the present disclosure;
图11是本公开实施例提供的手势互动效果示意图；Figure 11 is a schematic diagram of the gesture interaction effect provided by an embodiment of the present disclosure;
图12是本公开实施例提供的辅助频谱图的示意图；Figure 12 is a schematic diagram of an auxiliary spectrum diagram provided by an embodiment of the present disclosure;
图13是本公开实施例提供的观影互动装置的结构图。Figure 13 is a structural diagram of a movie viewing interactive device provided by an embodiment of the present disclosure.
具体实施方式Detailed ways
下面将结合本公开实施例中的附图，对本公开实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本公开一部分实施例，而不是全部的实施例。基于本公开中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本公开保护的范围。The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of this disclosure.
参见图1，图1是本公开实施例提供的观影互动方法的流程图，如图1所示，包括以下步骤：Referring to Figure 1, Figure 1 is a flow chart of a movie viewing interaction method provided by an embodiment of the present disclosure. As shown in Figure 1, it includes the following steps:
步骤101、在用户观看目标视频的过程中，获取用户图像。Step 101: Obtain the user's image while the user is watching the target video.
上述目标视频可以是用户当前观看的任意视频。The above target video can be any video currently watched by the user.
本公开实施例中，可以支持观影用户与视频中角色的肢体或表情互动，即观影用户可以摆出某种动作或作出某种表情，视频中角色则可以相应地作出回应动作或表情。例如，观影用户作出拥抱动作时，视频中角色给予拥抱呼应，观影用户举起右手握手，视频中角色举起右手回应等。In the embodiments of the present disclosure, the user can be supported to interact with the body or expression of the character in the video, that is, the user can perform certain actions or expressions, and the characters in the video can respond accordingly. For example, when a movie-watching user makes a hug gesture, the character in the video responds with a hug, the movie-watching user raises his right hand to shake hands, and the character in the video raises his right hand in response, etc.
该步骤中，可以在用户观看目标视频的过程中，获取用户图像，例如，当用户进入肢体互动时，可通过摄像头实时采集和上传用户图像。In this step, the user's image can be obtained while the user is watching the target video. For example, when the user enters physical interaction, the user's image can be collected and uploaded in real time through the camera.
可选地，所述步骤101之前，所述方法还包括：Optionally, before step 101, the method further includes:
接收第一用户在观看所述目标视频中针对第一弹幕的输入，其中，所述输入用于指示对第二用户发起观影小室创建邀请，所述第二用户为发布所述第一弹幕的用户；Receive input from the first user regarding the first bullet while watching the target video, wherein the input is used to instruct the second user to initiate an invitation to create a viewing room, and the second user is responsible for posting the first bullet. screen user;
在所述第二用户接受所述观影小室创建邀请的情况下，创建所述第一用户与所述第二用户的观影小室，并在所述观影小室中播放所述目标视频；When the second user accepts the invitation to create a viewing room, create a viewing room for the first user and the second user, and play the target video in the viewing room;
所述步骤101包括： The step 101 includes:
在用户在所述观影小室中观看所述目标视频的过程中，获取用户图像。During the process of the user watching the target video in the viewing booth, the user image is acquired.
即一种实施方式中，可以支持用户在观看视频的过程中创建观影小室，观影小室内成员可以在观影小室内进行互动。That is, in one implementation, users can be supported to create a viewing room while watching videos, and members in the viewing room can interact in the viewing room.
具体地，用户可在登录视频应用后设置是否开启观影小室功能。如果开启，代表用户有创建观影小室的权限，否则和普通观众一样，不能创建观影小室。其次可以设置是否接受其他观影小室邀请，如果开启，则允许系统推送的其他用户的观影小室邀请，否则屏蔽其他用户的观影小室邀请。Specifically, the user can set whether to enable the viewing room function after logging into the video application. If turned on, it means that the user has the permission to create a viewing room. Otherwise, like ordinary viewers, they cannot create a viewing room. Secondly, you can set whether to accept other viewing room invitations. If it is turned on, the viewing room invitations of other users pushed by the system will be allowed, otherwise the viewing room invitations of other users will be blocked.
当用户A点击任意视频进入观看，弹幕飘屏，这时用户A可以点击某个有趣弹幕，发起创建观影小室邀请，邀请消息直接通过视频应用发送到对方账户，即发布该弹幕的用户账户。如果受邀用户B该时段不在线，系统可直接发送提示给用户A；如果用户B在线，用户B可以选择接受和拒绝用户A的邀请。如果用户B接受邀请则观影小室构建成功，此时观影小室中存在用户A和用户B，其他未收到邀请的用户不能进入该小室。When user A clicks on any video to watch, the barrage floats on the screen. At this time, user A can click on an interesting barrage to initiate an invitation to create a viewing room. The invitation message is sent directly to the other party's account through the video application, that is, the barrage is posted. User Account. If invited user B is not online during this period, the system can directly send a prompt to user A; if user B is online, user B can choose to accept or reject user A's invitation. If user B accepts the invitation, the viewing room is successfully constructed. At this time, there are user A and user B in the viewing room, and other users who have not received the invitation cannot enter the room.
其中，创建观影小室的用户为管理员，其他用户发送的邀请需得到管理员同意，发送的邀请信息直接由视频平台推送给被邀请用户。用户来到观影小室中观看视频，实现小室内同步观影需求。室内成员可发起互动，所有消息仅室内成员可见，当观影小室解散后，所有消息被清空，提高了观影话题的私密性。观影小室内用户交流示意图如图2所示，即客户端用户将消息提交到服务器端，服务器端完成消息交互，从而展示在视频播放界面上。Among them, the user who creates the viewing room is the administrator. Invitations sent by other users must be approved by the administrator. The invitation information sent is directly pushed by the video platform to the invited users. Users come to the viewing room to watch videos, realizing the need for simultaneous viewing in the room. Indoor members can initiate interactions, and all messages are only visible to indoor members. When the movie viewing room is disbanded, all messages will be cleared, improving the privacy of movie viewing topics. The schematic diagram of user communication in the movie viewing booth is shown in Figure 2. That is, the client user submits the message to the server, and the server completes the message interaction and displays it on the video playback interface.
观影小室内成员在观看视频的过程中可以进行肢体互动，情景带入互动和基于弹幕的视频渲染。Members in the viewing booth can interact physically while watching the video, and the scene can be brought into interaction and video rendering based on barrage.
这样，通过该实施方式，用户可以通过创建观影小室的方式来进行群体观影，并可在观影小室内进行私密互动，提高了观影互动的趣味性和群体观影的私密性。In this way, through this implementation, users can create a viewing cabin for group viewing, and can interact privately in the viewing cabin, which improves the fun of movie-watching interaction and the privacy of group viewing.
步骤102、识别所述用户图像中的目标对象的姿态。Step 102: Identify the posture of the target object in the user image.
该步骤中，可以基于获取的用户图像，识别所述用户图像中的目标对象，通常为观影用户，并可通过图像识别和姿态检测，确定所述目标对象的姿态，即识别所述目标对象的肢体动作或表情。In this step, the target object in the user image can be identified based on the acquired user image, usually a movie-watching user, and the posture of the target object can be determined through image recognition and posture detection, that is, the target object can be identified body movements or expressions.
可选地，所述步骤102包括： Optionally, the step 102 includes:
通过目标检测算法识别所述用户图像中的目标对象；Identify the target object in the user image through a target detection algorithm;
构建所述目标对象的骨骼模型；Construct a skeletal model of the target object;
基于所述目标对象的骨骼模型，确定所述目标对象的姿态。Based on the skeletal model of the target object, a pose of the target object is determined.
一种实施方式中，为识别所述用户图像中的目标对象的姿态，可以先通过目标检测算法识别所述用户图像中的目标对象，例如，可通过显著性检测算法等描绘目标轮廓，并返回目标位置结果{x,y,w,h}，其中，x和y是目标矩形框中心点的横坐标和纵坐标，w为目标矩形框的宽，h为目标矩形框的高。In one embodiment, in order to identify the posture of the target object in the user image, the target object in the user image can be first identified through a target detection algorithm. For example, the target outline can be drawn through a saliency detection algorithm and returned. Target position result {x, y, w, h}, where x and y are the abscissa and ordinate coordinates of the center point of the target rectangular frame, w is the width of the target rectangular frame, and h is the height of the target rectangular frame.
如图3中的(a)图所示，图像背景中包含了多个人物和建筑，识别的显著目标如图3中的(b)图所示，矩形框内即为显著性目标。As shown in (a) of Figure 3, the image background contains multiple people and buildings. The recognized salient targets are shown in (b) of Figure 3. The salient targets are within the rectangular frame.
接下来可根据显著性目标检测结果，对矩形框中的目标人物轮廓来构建骨骼模型。具体地，可根据矩形框内目标对象的关键部位的关键点，构建所述目标对象的骨骼模型。Next, based on the salient target detection results, a skeletal model can be constructed for the target character outline in the rectangular frame. Specifically, a skeletal model of the target object can be constructed based on key points of key parts of the target object within the rectangular frame.
例如，在原图像上的检测结果如图4中的(a)图所示，从原图像中抽象出来的骨骼模型如图4中的(b)图所示，模型中一共包含15个关键点，分别对应人体的15个关键部位，如p1点对应头部，p2点对应右手，p15点是左脚。For example, the detection results on the original image are shown in (a) of Figure 4, and the skeleton model abstracted from the original image is shown in (b) of Figure 4. The model contains a total of 15 key points. They correspond to 15 key parts of the human body, such as point p1 for the head, point p2 for the right hand, and point p15 for the left foot.
这样，可基于所述目标对象的骨骼模型，确定所述目标对象的姿态，如根据图4中的(b)图所示的骨骼模型，可以确定人物姿态是四肢伸展的站立姿态。In this way, the posture of the target object can be determined based on the skeletal model of the target object. For example, according to the skeletal model shown in (b) of FIG. 4 , it can be determined that the character posture is a standing posture with limbs extended.
进一步地，由于单张图像很难检测出目标对象的主要动作，因此可结合镜头转换和连续性行为来确定所述目标对象的具体动作。例如，某个用户视频片段的连续性检测图像结果的骨骼模型如图5所示，则算法返回的姿态结果为跳舞，结果封装{result:‘dance’}。Furthermore, since it is difficult to detect the main action of the target object in a single image, the specific action of the target object can be determined by combining lens transitions and continuity behaviors. For example, the skeleton model of the continuity detection image result of a certain user's video clip is shown in Figure 5. The posture result returned by the algorithm is dancing, and the result is encapsulated {result:'dance'}.
这样，通过该实施方式，可以准确识别用户图像中目标对象的姿态。In this way, through this embodiment, the posture of the target object in the user's image can be accurately recognized.
步骤103、基于所述目标对象的姿态，确定所述目标视频中的目标角色的回应姿态。Step 103: Based on the posture of the target object, determine the response posture of the target character in the target video.
在确定所述目标对象的姿态后，可以根据所述目标对象的姿态，确定所述目标视频中的目标角色的回应姿态，具体地，可以采用类似方法如显著性检测算法确定所述目标视频中的目标角色，通常为视频画面中面积占比最大 的角色，然后确定所述目标角色的回应姿态，所述目标角色的回应姿态可以是与所述目标对象的姿态基本相同的姿态，例如，检测到的用户姿态是握手，则视频中角色也应该要做出握手姿态，以回应用户与用户肢体互动，又如检测到用户作出拥抱姿势，则视频中角色也需要作出拥抱姿势，以回应用户的拥抱。After determining the posture of the target object, the response posture of the target character in the target video can be determined according to the posture of the target object. Specifically, a similar method such as a saliency detection algorithm can be used to determine the response posture of the target character in the target video. The target character usually accounts for the largest area in the video screen. character, and then determine the response gesture of the target character. The response gesture of the target character can be a gesture that is basically the same as the gesture of the target object. For example, if the detected user gesture is a handshake, the character in the video should also To make a handshake gesture in response to the user's physical interaction, and if it is detected that the user makes a hug gesture, the character in the video also needs to make a hug gesture in response to the user's hug.
可选地，所述步骤103之后，所述方法还包括：Optionally, after step 103, the method further includes:
获取所述目标视频中的目标角色的骨骼模型；Obtain the skeleton model of the target character in the target video;
对所述目标对象的骨骼模型与所述目标角色的骨骼模型进行骨骼关键点匹配，以调整所述目标角色的回应姿态。Skeleton key point matching is performed on the skeletal model of the target object and the skeletal model of the target character to adjust the response posture of the target character.
即一种实施方式中，可以进一步对所述目标角色的回应姿态进行调整，以使观影用户和视频中角色的互动更加细致化。That is, in one implementation, the response posture of the target character can be further adjusted to make the interaction between the movie-watching user and the character in the video more detailed.
具体地，可以获取所述目标视频中的目标角色的骨骼模型，如可以调用所述目标视频中的目标角色的骨骼模型使其摆出合适的肢体动作，然后可通过骨骼关键点匹配来保证互动姿态的合理性，即对所述目标对象的骨骼模型与所述目标角色的骨骼模型进行骨骼关键点匹配，使各对应的骨骼关节点位于相类似的空间位置，从而保证所述目标角色摆出与所述目标对象基本相同的姿态。例如，用户伸出右手握手，视频中角色也需要伸出右手，而非左手。如图6所示，在骨骼关键点匹配中，左边观影用户的骨骼模型中的p1对应是右边视频中角色的骨骼模型中B1，以此类推，p2对应B2等。Specifically, the skeletal model of the target character in the target video can be obtained. For example, the skeletal model of the target character in the target video can be called to make it perform appropriate body movements, and then the interaction can be ensured through skeletal key point matching. The rationality of the posture is to match the skeletal key points of the target object's skeletal model and the target character's skeletal model so that the corresponding skeletal joint points are located in similar spatial positions, thereby ensuring that the target character poses Basically the same pose as the target object. For example, if the user extends his right hand to shake hands, the character in the video also needs to extend his right hand, not his left hand. As shown in Figure 6, in the bone key point matching, p1 in the bone model of the user watching the video on the left corresponds to B1 in the bone model of the character in the video on the right, and so on, p2 corresponds to B2, etc.
这样，通过骨骼关键点匹配能够让用户和视频中的人物互动更加一步细致化。In this way, the interaction between the user and the characters in the video can be made more detailed through bone key point matching.
步骤104、基于所述目标对象的目标部位的姿态，对所述目标角色的目标部位的回应姿态进行调整。Step 104: Adjust the response posture of the target part of the target character based on the posture of the target part of the target object.
本公开实施例中，还可对于观影用户与视频中角色的细致部位，比如手势、脸部表情，进行姿态调整，以确定相应的互动姿态。In the embodiments of the present disclosure, the postures of the viewing user and the characters in the video, such as gestures and facial expressions, can also be adjusted to determine the corresponding interactive postures.
该步骤中，可基于所述目标对象的目标部位的姿态，对所述目标角色的目标部位的回应姿态进行调整，以使所述目标角色作出与所述目标对象相呼应的手势，从而可基于调整后的所述目标对象的目标部位的姿态，在所述视频画面中显示所述目标角色的回应姿态，并可突出显示所述目标角色的目标 部位的回应姿态。In this step, the response posture of the target part of the target character can be adjusted based on the posture of the target part of the target object, so that the target character makes a gesture that echoes the target object, so that the response posture of the target part of the target character can be adjusted based on the posture of the target part of the target object. The adjusted posture of the target part of the target object displays the response posture of the target character in the video screen, and can highlight the target of the target character. part's response.
步骤105、基于调整后的所述目标角色的目标部位的回应姿态，在所述目标视频的视频画面中显示所述目标角色的回应姿态。Step 105: Based on the adjusted response posture of the target part of the target character, display the response posture of the target character in the video frame of the target video.
该步骤中，可以在所述目标视频的视频画面中显示所述目标角色的回应姿态，即在所述视频画面中生成所述目标角色作出回应姿态的显示效果，实现观影用户与视频中角色的互动。在显示所述目标角色的回应姿态时，可以突出显示所述目标角色的目标部位的回应姿态，以生成生动的观影互动效果。In this step, the response gesture of the target character can be displayed in the video screen of the target video, that is, a display effect of the target character's response gesture is generated in the video screen, so as to realize the interaction between the viewing user and the character in the video. of interaction. When displaying the response gesture of the target character, the response gesture of the target part of the target character can be highlighted to generate a vivid interactive effect for watching movies.
可选地，所述步骤105包括：Optionally, the step 105 includes:
基于所述目标对象的目标部位的姿态和调整后的所述目标角色的目标部位的回应姿态，在所述视频画面中显示所述目标对象的目标部位与所述目标角色的目标部位的互动效果。Based on the posture of the target part of the target object and the adjusted response posture of the target part of the target character, the interactive effect of the target part of the target object and the target part of the target character is displayed in the video screen .
一种实施方式中，还可对于观影用户与视频中角色的细致部位，比如手势、脸部表情，确定相应的互动姿态。In one implementation, corresponding interactive postures can also be determined for detailed parts of the movie-watching user and the characters in the video, such as gestures and facial expressions.
即可基于所述目标对象的目标部位的姿态，对所述目标角色的目标部位的回应姿态进行调整，以使所述目标角色作出与所述目标对象相呼应的手势，并在所述视频画面中显示所述目标对象的目标部位与所述目标角色的目标部位的互动效果，即可将所述目标对象的目标部位的姿态投射到视频画面中显示，以及显示所述目标角色的目标部位的回应姿态，形成二者的目标部位的互动姿态。例如，当用户抬起右手时，也可对视频中角色抬起右手，并在视频画面中显示用户抬手与视频中角色抬手触摸，形成用户与视频中角色隔屏触摸的互动效果；当用户作出鬼脸表情时，也可对视频中角色作出鬼脸表情，形成用户与视频中角色作出鬼脸互动的效果。That is, based on the posture of the target part of the target object, the response posture of the target part of the target character is adjusted, so that the target character makes a gesture that echoes the target object and displays it in the video screen. The interaction effect between the target part of the target object and the target part of the target character is displayed in Respond to the gesture and form an interactive gesture of the target parts of the two. For example, when the user raises his right hand, he can also raise his right hand to the character in the video, and the user's raised hand and the raised hand touch of the character in the video are displayed on the video screen, forming an interactive effect of the user and the character in the video touching each other across the screen; when When the user makes a face expression, he or she can also make a face expression on the character in the video, creating the effect of the user interacting with the character in the video making a face expression.
这样，通过该实施方式，观影用户可与视频中角色进行更为细致的特定部位的互动，提高互动体验。In this way, through this implementation, movie-watching users can interact with specific parts of the characters in the video in more detail, thereby improving the interactive experience.
可选地，所述步骤104包括：Optionally, the step 104 includes:
确定包围所述目标对象的目标部位的三维模型的第一最小外接球；Determine the first minimum circumscribed sphere of the three-dimensional model surrounding the target part of the target object;
确定包围所述目标角色的目标部位的三维模型的第二最小外接球；Determine the second minimum circumscribed sphere of the three-dimensional model surrounding the target part of the target character;
根据所述第一最小外接球的球心位置，将所述第一最小外接球投影到所述目标视频的视频画面中； Project the first minimum circumscribed ball into the video frame of the target video according to the center position of the first minimum circumscribed ball;
根据所述第一最小外接球投影在所述视频画面中的位置，对所述第二最小外接球的位置进行调整；Adjust the position of the second minimum circumscribed ball according to the projected position of the first minimal circumscribed ball in the video screen;
根据调整后的所述第二最小外接球的位置，调整所述目标角色的目标部位的回应姿态。According to the adjusted position of the second minimum external ball, the response posture of the target part of the target character is adjusted.
一种具体地实施方式中，可以先计算所述目标对象的目标部位的三维模型的最小外接球和所述目标角色的目标部位的三维模型的最小外接球，即能完全包围所述目标部位的三维模型的最小球体。如图7所示，可对目标部位如手势进行三维建模，然后计算包围目标部位的三维模型的最小外接球，如图8所示，假设视频中角色目标部位模型的最小外接球为V1，用户目标部位模型的最小外接球为V2，两个球体的球心分别为M1和M2。In a specific implementation, the minimum circumscribed sphere of the three-dimensional model of the target part of the target object and the minimum circumscribed sphere of the three-dimensional model of the target part of the target character can be calculated first, that is, the minimum circumscribed sphere that can completely surround the target part. The smallest sphere of the 3D model. As shown in Figure 7, the target part such as a gesture can be three-dimensionally modeled, and then the minimum circumscribed sphere of the three-dimensional model surrounding the target part is calculated. As shown in Figure 8, assuming that the minimum circumscribed sphere of the character's target part model in the video is V1, The minimum circumscribed sphere of the user's target part model is V2, and the centers of the two spheres are M1 and M2 respectively.
接下来可根据最小外接球V2的球心坐标M2，将最小外接球V2投影到所述目标视频的视频画面中，也即可将所述目标对象的目标部位的三维模型投影到所述视频画面中显示，然后可根据最小外接球V2投影在所述视频画面中的位置，对最小外接球V1的坐标位置进行调整，以使最小外接球V1的位置与最小外接球V2的位置达到空间对齐，进而可根据调整后的最小外接球V1的位置，适应调整所述目标角色的目标部位的回应姿态。Next, the minimum circumscribed ball V2 can be projected into the video picture of the target video according to the center coordinate M2 of the minimum circumscribed ball V2, that is, the three-dimensional model of the target part of the target object can be projected into the video picture. displayed in the video screen, and then the coordinate position of the minimum external ball V1 can be adjusted according to the position of the minimum external ball V2 projected in the video screen, so that the position of the minimum external ball V1 and the position of the minimum external ball V2 are spatially aligned. Furthermore, the response posture of the target part of the target character can be adaptively adjusted according to the adjusted position of the minimum external ball V1.
这样，通过该实施方式中的调整，可使所述目标对象的目标部位的姿态映射至视频画面显示，并通过调整所述目标角色的目标部位的回应姿态，使二者在视频画面中形成互动姿态。In this way, through the adjustment in this embodiment, the posture of the target part of the target object can be mapped to the video screen display, and by adjusting the response posture of the target part of the target character, the two can interact in the video screen attitude.
进一步地，所述根据所述第一最小外接球投影在所述视频画面中的位置，对所述第二最小外接球的位置进行调整，包括：Further, adjusting the position of the second minimum circumscribed ball according to the projected position of the first minimal circumscribed ball in the video screen includes:
根据所述第一最小外接球投影在所述视频画面中的球心位置，调整所述第二最小外接球的球心位置；Adjust the center position of the second minimum circumscribed ball according to the center position of the first minimum circumscribed ball projected in the video screen;
根据所述目标角色的目标部位的三维模型中任两点间的夹角关系，确定所述目标角色的目标部位中各点的位置；Determine the position of each point in the target part of the target character based on the angle relationship between any two points in the three-dimensional model of the target part of the target character;
所述根据调整后的所述第二最小外接球的位置，调整所述目标角色的目标部位的回应姿态，包括：Adjusting the response posture of the target part of the target character according to the adjusted position of the second minimum external ball includes:
根据调整后的所述第二最小外接球的球心位置，以及所述目标角色的目标部位中各点的位置，调整所述目标角色的目标部位的回应姿态。 The response posture of the target part of the target character is adjusted according to the adjusted center position of the second minimum external ball and the positions of each point in the target part of the target character.
更具体地，可以根据最小外接球V2投影在所述视频画面中的球心坐标位置M2′，调整最小外接球V1的球心坐标位置，如使二者的球心坐标位置对齐。More specifically, the center coordinate position of the minimum circumscribed ball V1 can be adjusted according to the center coordinate position M2' of the minimum circumscribed ball V2 projected in the video screen, such as aligning the coordinate positions of the two sphere centers.
例如，W1是＝{(x1,y1,z1),(x2,y2,z2)…(xn,yn,zn)}是所述目标角色的目标部位的三维模型中的所有点的集合。对于任意点A到三维坐标原点形成的向量与任一点Q到三维坐标原点形成的向量之间的夹角关系为θ，则模型中任意两点的夹角关系集合S＝{θ1,θ2,θ3…θm}。假设集合W1是视频中角色的目标部位模型点的集合，集合W2是用户目标部位模型点的集合，则可根据所述目标角色的目标部位的三维模型中任两点间的夹角关系S＝{θ1,θ2,θ3…θm}，计算视频中角色的目标部位中其他点的坐标，计算的结果集合为W3，这样，可根据计算得出的新集合点坐标W3，对视频中角色的目标部位的回应姿态进行调整。For example, W 1 is={(x 1 , y 1 , z 1 ), (x 2 , y 2 , z 2 )...(x n , y n , z n )} which is the three-dimensional image of the target part of the target character. The set of all points in the model. For the vector formed from any point A to the origin of the three-dimensional coordinates The vector formed from any point Q to the origin of the three-dimensional coordinates The angle relationship between them is θ, then the set of angle relationships between any two points in the model S = {θ 1 , θ 2 , θ 3 ...θ m }. Assuming that set W 1 is a set of model points of the target part of the character in the video, and set W 2 is a set of model points of the user's target part, then the angle relationship between any two points in the three-dimensional model of the target part of the target character can be S={θ 1 , θ 2 , θ 3 ...θ m }, calculate the coordinates of other points in the target part of the character in the video, and the calculated result set is W 3 . In this way, the calculated new set point coordinates W can be 3. Adjust the response posture of the target part of the character in the video.
这样，通过该实施方式，可实现对所述目标角色的目标部位的回应姿态进行准确细致地调整。In this way, through this embodiment, the response posture of the target part of the target character can be adjusted accurately and carefully.
可选地，所述目标部位为手部；Optionally, the target part is the hand;
所述步骤104包括：The step 104 includes:
以所述目标对象的手部的第一手掌根节点为原点建立三维坐标系，其中，所述目标对象的手部包括大拇指关节点、食指关节点、中指关节点、无名指关节点和小指关节点；A three-dimensional coordinate system is established with the first palm heel node of the target object's hand as the origin, wherein the target object's hand includes a thumb joint point, an index finger joint point, a middle finger joint point, a ring finger joint point and a little finger joint point. point;
计算第一空间角、第二空间角、第三空间角和第四空间角；其中，所述第一空间角是以所述第一手掌根节点为顶点且以第一线段和第二线段为边的空间夹角，所述第二空间角是以所述第一手掌根节点为顶点且以所述第二线段和第三线段为边的空间夹角，所述第三空间角是以所述第一手掌根节点为顶点且以所述第三线段和第四线段为边的空间夹角，所述第四空间角是以所述第一手掌根节点为顶点且以所述第四线段和第五线段为边的空间夹角；所述第一线段为所述第一手掌根节点与相邻的大拇指关节点的线段，所述第二线段为所述第一手掌根节点与相邻的食指关节点的线段，所述第三线段为所述第一手掌根节点与相邻的中指关节点的线段，所述第四线段为所述第一手掌根节点与相邻的无名指关节点的线段，所述第五线段为所述第一手掌根节 点与相邻的小指关节点的线段；Calculate the first space angle, the second space angle, the third space angle and the fourth space angle; wherein, the first space angle is based on the first palm heel node as the vertex and the first line segment and the second line segment is the space angle of the side, the second space angle is the space angle with the first palm heel node as the vertex and the second line segment and the third line segment as the side, and the third space angle is The space angle between the first palm heel node as a vertex and the third line segment and the fourth line segment as sides, and the fourth space angle is a space angle with the first palm heel node as a vertex and the fourth line segment as an edge. The line segment and the fifth line segment are the spatial angles of the sides; the first line segment is the line segment between the first palm heel node and the adjacent thumb joint point, and the second line segment is the first palm heel node. and the adjacent index finger joint point, the third line segment is the line segment between the first palm heel node and the adjacent middle finger joint point, and the fourth line segment is the line segment between the first palm heel node and the adjacent middle finger joint point. The line segment of the ring finger joint, the fifth line segment is the first palm heel joint A line segment between a point and the adjacent little finger joint point;
计算第一线面空间角、第二线面空间角、第三线面空间角、第四线面空间角和第五线面空间角；其中，所述第一线面空间角是以所述目标对象的手部的大拇指关节点所在线段与第一面的空间夹角，所述第二线面空间角是以所述目标对象的手部的食指关节点所在线段与所述第一面的空间夹角，所述第三线面空间角是以所述目标对象的手部的中指关节点所在线段与所述第一面的空间夹角，所述第四线面空间角是以所述目标对象的手部的无名指关节点所在线段与所述第一面的空间夹角，所述第五线面空间角是以所述目标对象的手部的小指关节点所在线段与所述第一面的空间夹角；所述第一面为所述三维坐标系中z轴和y轴所构成的平面；Calculate the first line-plane space angle, the second line-plane space angle, the third line-plane space angle, the fourth line-plane space angle and the fifth line-plane space angle; wherein the first line-plane space angle is based on the target object The space angle between the line segment where the thumb joint point of the hand is located and the first surface, and the second line-plane space angle is the space angle between the line segment where the index finger joint point of the target object's hand is located and the first surface. The space angle, the third line-plane space angle is the space angle between the line segment where the middle finger joint point of the target object's hand is and the first surface, the fourth line-plane space angle is the space angle between the The angle between the line segment where the joint point of the ring finger of the target object's hand is located and the first plane, and the fifth line-plane space angle is the line segment where the joint point of the little finger of the target object's hand is located and the space angle between the line segment and the first plane. The spatial angle of the first surface; the first surface is the plane formed by the z-axis and the y-axis in the three-dimensional coordinate system;
根据所述第一空间角、第二空间角、第三空间角和第四空间角，以及所述第一线面空间角、第二线面空间角、第三线面空间角、第四线面空间角和第五线面空间角，调整所述目标角色的手部的大拇指关节点、食指关节点、中指关节点、无名指关节点和小指关节点的位置。According to the first space angle, the second space angle, the third space angle and the fourth space angle, as well as the first line-plane space angle, the second line-plane space angle, the third line-plane space angle and the fourth line-plane space angle. angle and the fifth line-plane space angle, adjust the positions of the thumb joint point, index finger joint point, middle finger joint point, ring finger joint point and little finger joint point of the target character's hand.
一种实施方式中，所述目标部位可以手部，且可以对基于所述目标对象的手部动作，对所述目标角色的手部动作进行细致调整。In one implementation, the target part may be a hand, and the hand movements of the target character may be carefully adjusted based on the hand movements of the target object.
如图9所示，可以以所述目标对象的手部的手掌根节点q1为原点建立三维坐标系，集合点{q1，q2,q3…q20}模拟了用户手部的20个主要关节点。其中，q1对应为手掌根节点，q6、q7和q20对应为大拇指关节点，q5、q8、q18和q19对应为食指关节点，q4、q9、q16和q17对应的是中指关节点，q3、q10、q14和q15对应的是无名指关节点，q2、q11、q12和p13对应的是小指关节点。As shown in Figure 9, a three-dimensional coordinate system can be established with the palm root node q1 of the target object's hand as the origin, and the set points {q1, q2, q3...q20} simulate 20 main joint points of the user's hand. Among them, q1 corresponds to the palm root node, q6, q7 and q20 correspond to the thumb joint point, q5, q8, q18 and q19 correspond to the index finger joint point, q4, q9, q16 and q17 correspond to the middle finger joint point, q3, q10, q14 and q15 correspond to the joint point of the ring finger, and q2, q11, q12 and p13 correspond to the joint point of the little finger.
如图10所示，ω1是以q1为顶点，线段q1q6和线段q1q5为边的空间夹角；ω2是以q1为顶点，线段q1q5和线段q1q4为边的空间夹角；ω3是以q1为顶点，线段q1q4和线段q1q3为边的空间夹角；ω4是以q1为顶点，线段q1q3和线段q1q2为边的空间夹角。As shown in Figure 10, ω 1 is the angle in space with q1 as the vertex and line segment q1q6 and line segment q1q5 as sides; ω 2 is the space angle with q1 as the vertex and line segment q1q5 and line segment q1q4 as sides; ω 3 is the angle in space with q1 as the vertex and line segment q1q5 and line segment q1q4 as sides. q1 is the vertex, the line segment q1q4 and the line segment q1q3 are the sides; ω 4 is the space angle with q1 as the vertex, the line segment q1q3 and the line segment q1q2 are the sides.
则可计算空间角ω1，ω2，ω3和ω4。以计算ω1为例，将线段q1q6记为向量线段q1q5记为向量则ω1计算公示如下：
Then the spatial angles ω 1 , ω 2 , ω 3 and ω 4 can be calculated. Taking the calculation of ω 1 as an example, record the line segment q1q6 as a vector Line segments q1q5 are recorded as vectors Then the calculation of ω 1 is as follows:
其他ω2，ω3和ω4的计算同理可得。The calculations of other ω 2 , ω 3 and ω 4 can be obtained in the same way.
这样，可根据计算得到的空间角ω1，ω2，ω3和ω4，对所述目标角色的手部距离其手掌根节点最近的大拇指关节点Q6、食指关节点Q5、中指关节点Q4、无名指关节点Q3和小指关节点Q2的位置进行调整，使所述目标角色的手部各关节点位置与所述目标对象的手部各关节点位置对应。In this way, based on the calculated space angles ω 1 , ω 2 , ω 3 and ω 4 , the thumb joint point Q6, index finger joint point Q5, and middle finger joint point of the target character's hand that are closest to the root node of the hand can be determined Q4. The positions of the ring finger joint point Q3 and the little finger joint point Q2 are adjusted so that the positions of the joint points of the hand of the target character correspond to the positions of the joint points of the hand of the target object.
接下来，还可调整所述目标角色的手部剩余关节点的位置，具体地，可分别计算线段q6q7，q5q8，q4q9，q3q10，q2q11与面zq1y的夹角，分别记为γ1，γ2，γ3，γ4，γ5。然后可根据计算得到的线面空间角γ1，γ2，γ3，γ4和γ5，对所述目标角色的手部剩余大拇指关节点、食指关节点、中指关节点、无名指关节点和小指关节点，即{Q7，Q8，Q9…Q20}的位置进行调整，使所述目标角色的手部各关节点位置与所述目标对象的手部各关节点位置对应，从而使所述目标角色的手势与所述目标对象的手势保持基本对称。调整之后的手部互动效果可如图11所示，用户手势可以和视频中角色手势形成隔屏幕触碰的互动体验。Next, the positions of the remaining joint points of the target character's hand can also be adjusted. Specifically, the angles between the line segments q6q7, q5q8, q4q9, q3q10, q2q11 and the surface zq1y can be calculated respectively, which are recorded as γ 1 and γ 2 respectively. , γ 3 , γ 4 , γ 5 . Then, based on the calculated line-plane space angles γ 1 , γ 2 , γ 3 , γ 4 and γ 5 , the remaining thumb joint points, index finger joint points, middle finger joint points and ring finger joint points of the target character’s hand can be determined and the positions of the little finger joints, that is, {Q7, Q8, Q9...Q20}, so that the positions of the joint points of the hand of the target character correspond to the positions of the joint points of the hand of the target object, so that the The target character's gestures remain substantially symmetrical to the target object's gestures. The adjusted hand interaction effect is shown in Figure 11. User gestures can form an interactive experience of touching across the screen with the character gestures in the video.
进一步地，所述根据所述第一空间角、第二空间角、第三空间角和第四空间角，以及所述第一线面空间角、第二线面空间角、第三线面空间角、第四线面空间角和第五线面空间角，调整所述目标角色的手部的大拇指关节点、食指关节点、中指关节点、无名指关节点和小指关节点的位置，包括：Further, according to the first space angle, the second space angle, the third space angle and the fourth space angle, and the first line-plane space angle, the second line-plane space angle, the third line-plane space angle, The fourth line-plane space angle and the fifth line-plane space angle adjust the positions of the thumb joint point, index finger joint point, middle finger joint point, ring finger joint point and little finger joint point of the target character's hand, including:
令第五空间角等于所述第一空间角，第六空间角等于所述第二空间角，第七空间角等于所述第三空间角，第八空间角等于所述第四空间角，调整所述目标角色的手部的第一大拇指关节点、第一食指关节点、第一中指关节点、第一无名指关节点和第一小指关节点的位置；Let the fifth space angle be equal to the first space angle, the sixth space angle be equal to the second space angle, the seventh space angle be equal to the third space angle, and the eighth space angle be equal to the fourth space angle, adjust The positions of the first thumb joint point, first index finger joint point, first middle finger joint point, first ring finger joint point and first little finger joint point of the target character's hand;
令第六线面空间角等于所述第一线面空间角，第七线面空间角等于所述第二线面空间角，第八线面空间角等于所述第三线面空间角，第九线面空间角等于所述第四线面空间角，第十线面空间角等于所述第五线面空间角，调整所述目标角色的手部的除所述第一大拇指关节点外的其他大拇指关节点、除所述第一食指关节点外的其他食指关节点、除所述第一中指关节点外的其 他中指关节点、除所述第一无名指关节点外的其他无名指关节点和除所述第一小指关节点外的其他小指关节点的位置；Let the sixth line-plane space angle be equal to the first line-plane space angle, the seventh line-plane space angle be equal to the second line-plane space angle, the eighth line-plane space angle be equal to the third line-plane space angle, and the ninth line-plane space angle be equal to the third line-plane space angle. The plane space angle is equal to the fourth line plane space angle, and the tenth line plane space angle is equal to the fifth line plane space angle. Adjust other points of the target character's hand except the first thumb joint point. Thumb joint point, other index finger joint points except the first index finger joint point, other index finger joint points except the first middle finger joint point The positions of the joint points of his middle finger, other ring finger joint points except the first ring finger joint point, and other little finger joint points except the first little finger joint point;
其中，所述第五空间角是以所述目标角色的手部的第二手掌根节点为顶点且以第六线段和第七线段为边的空间夹角，所述第六空间角是以所述第二手掌根节点为顶点且以所述第七线段和第八线段为边的空间夹角，所述第七空间角是以所述第二手掌根节点为顶点且以第八线段和第九线段为边的空间夹角，所述第八空间角是以所述第二手掌根节点为顶点且以第九线段和第十线段为边的空间夹角；所述第六线段为所述第二手掌根节点与相邻的所述第一大拇指关节点的线段，所述第七线段为所述第二手掌根节点与相邻的所述第一食指关节点的线段，所述第八线段为所述第二手掌根节点与相邻的所述第一中指关节点的线段，所述第九线段为所述第二手掌根节点与相邻的所述第一无名指关节点的线段，所述第十线段为所述第二手掌根节点与相邻的所述第一小指关节点的线段；Wherein, the fifth space angle is the space angle with the second palm root node of the target character's hand as the vertex and the sixth line segment and the seventh line segment as the sides, and the sixth space angle is based on the The space angle between the second palm heel node as a vertex and the seventh line segment and the eighth line segment as sides. The seventh space angle is the second palm heel node as a vertex and the eighth line segment and the eighth line segment as sides. The nine line segments are the space angles of the sides, and the eighth space angle is the space angle with the second palm heel node as the vertex and the ninth line segment and the tenth line segment as the sides; the sixth line segment is the space angle. The line segment between the second palm heel node and the adjacent first thumb joint point, the seventh line segment is the line segment between the second palm heel node and the adjacent first index finger joint point, and the seventh line segment is the line segment between the second palm heel node and the adjacent first index finger joint point. The eighth line segment is a line segment between the second palm heel node and the adjacent first middle finger joint point, and the ninth line segment is a line segment between the second palm heel node and the adjacent first ring finger joint point. , the tenth line segment is a line segment between the second palm heel node and the adjacent first little finger joint point;
所述第六线面空间角以所述目标角色的手部的大拇指关节点所在线段与第二面的空间夹角，所述第七线面空间角是以所述目标角色的手部的食指关节点所在线段与所述第二面的空间夹角，所述第八线面空间角是以所述目标角色的手部的中指关节点所在线段与所述第二面的空间夹角，所述第九线面空间角是以所述目标角色的手部的无名指关节点所在线段与所述第二面的空间夹角，所述第十线面空间角是以所述目标角色的手部的小指关节点所在线段与所述第二面的空间夹角；所述第二面为以所述第二手掌根节点为原点建立的三维坐标系中z轴和y轴所构成的平面。The sixth line-plane space angle is the space angle between the line segment where the thumb joint point of the target character's hand is located and the second surface, and the seventh line-plane space angle is the space angle between the target character's hand and the second surface. The angle between the line segment where the joint point of the index finger is located and the second surface. The eighth line-plane space angle is the space between the line segment where the joint point of the middle finger of the hand of the target character is located and the second surface. The included angle, the ninth line-plane space angle is the space angle between the line segment where the ring finger joint point of the target character's hand is located and the second surface, and the tenth line-plane space angle is the space angle between the The angle between the line segment where the little finger joint point of the target character's hand is located and the second surface; the second surface is the z-axis and y-axis in the three-dimensional coordinate system established with the second palm root node as the origin. The plane formed.
即对于视频中角色手部模型集合点{Q1，Q2,Q3…Q20}，相应的空间角为和其中Q1对应目标对象手部模型中的对应点q1，Q2对应目标对象手部模型中的对应点q2，Q3对应目标对象手部模型中的对应点q3，依此类推。对应ω1，对应ω2，对应ω3，对应ω4。That is, for the character hand model collection points {Q1, Q2, Q3...Q20} in the video, the corresponding spatial angle is and Q1 corresponds to the corresponding point q1 in the target object's hand model, Q2 corresponds to the corresponding point q2 in the target object's hand model, Q3 corresponds to the corresponding point q3 in the target object's hand model, and so on. Corresponding to ω 1 , Corresponding to ω 2 , Corresponding to ω 3 , Corresponds to ω 4 .
这样，可首先调整视频中角色手部Q2、Q3、Q4、Q5、Q6点的坐标，已知当前Q5，Q6坐标值即调整前的坐标，可令线段Q1Q6为向量线段Q1Q5为向量固定点Q6坐标，利用夹角公示和ω1值可以求出n个向量解的集合固定点Q5坐标，利用夹角公示和ω1值可以 求出m个向量解的集合集合D和集合E中的解向量分别是以原点为起点的向量，通过将当前集合D中的向量坐标减去原点坐标，由此可以计算出所有可能的Q5坐标点的集合，这样的坐标集合记为M。同理，也可以计算所有可能的Q6坐标点的集合记为集合G。In this way, you can first adjust the coordinates of Q2, Q3, Q4, Q5, and Q6 points of the character's hand in the video. It is known that the current Q5 and Q6 coordinate values are the coordinates before adjustment, so Line segment Q1Q6 is a vector Line segments Q1Q5 are vectors Fixed point Q6 coordinates, the set of n vector solutions can be found using the angle expression and the value of ω 1 The coordinates of the fixed point Q5 can be determined by using the included angle expression and the value of ω 1 Find the set of m vector solutions The solution vectors in set D and set E are vectors starting from the origin. By subtracting the origin coordinates from the vector coordinates in the current set D, the set of all possible Q5 coordinate points can be calculated. Such a coordinate set Marked as M. In the same way, the set of all possible Q6 coordinate points can also be calculated and recorded as set G.
然后，可依次计算当前点Q6与集合G中的所有点的空间距离，记为L1。假设原Q6＝(xq,yq,zq)，集合G中任一点为K＝(x′,y′,z′)，则点Q6与点K的空间距离的计算公示如下所示：
Then, the spatial distance between the current point Q6 and all points in the set G can be calculated sequentially, recorded as L 1 . Assume that the original Q6 = (x q , y q , z q ), and any point in the set G is K = (x′, y′, z′), then the calculation of the spatial distance between point Q6 and point K is as follows:
同理，依次计算Q5与集合M中所有坐标点空间距离值L2，然后可确定满足L1+L2的值最小，且保证空间夹角为ω1的两个坐标点，分别记为Q6′和Q5′，其中Q6′是Q6调整之后的坐标，Q5′是Q5调整之后的坐标。In the same way, calculate the spatial distance values L 2 between Q5 and all coordinate points in the set M in sequence, and then determine the two coordinate points that satisfy the minimum value of L 1 + L 2 and ensure that the space angle is ω 1 , respectively, recorded as Q6 ′ and Q5′, where Q6′ is the coordinate after Q6 adjustment, and Q5′ is the coordinate after Q5 adjustment.
已知Q5′坐标，再令求出一系列满足条件的坐标点集合，记为集合J，然后计算原Q4点与集合J中所有坐标点的空间距离，取最小的空间距离值所对应的坐标值，该坐标值即为调整后的Q4坐标值，记为Q4′。同理可以依次求得调整后的Q3和Q2值。Given the coordinates of Q5′, let Find a series of coordinate point sets that meet the conditions, recorded as set J, then calculate the spatial distance between the original Q4 point and all coordinate points in set J, and take the coordinate value corresponding to the smallest spatial distance value. This coordinate value is the adjustment The final Q4 coordinate value is recorded as Q4′. In the same way, the adjusted Q3 and Q2 values can be obtained in sequence.
接下来，可调整剩下其他点的位置，包括{Q7，Q8，Q9…Q20}。具体地，可令Q6′Q7与面zQ1y的夹角等于γ1，求解得到Q6′Q7的一组向量解。将向量中的坐标减去Q6′的坐标得到所有可能的Q7点的坐标值，该集合记为R。计算原Q7点的坐标与R中所有值的空间距离，得到最小的空间距离值对应的坐标点，即为调整后的Q7的坐标点，记为Q7′。同理利用已知的上一个节点的坐标值，比如Q7的上一个节点是Q6，Q8的上一个节点是Q5，以及线面空间角，通过求得最小空间距离来确定最终的调整坐标值，依次调整剩下的所有坐标点的值。Next, you can adjust the positions of the remaining points, including {Q7, Q8, Q9…Q20}. Specifically, the angle between Q6′Q7 and surface zQ1y can be equal to γ 1 , and a set of vector solutions for Q6′Q7 can be obtained. Subtract the coordinates of Q6′ from the coordinates in the vector to obtain the coordinate values of all possible Q7 points. This set is marked R. Calculate the spatial distance between the coordinates of the original Q7 point and all values in R, and obtain the coordinate point corresponding to the smallest spatial distance value, which is the adjusted coordinate point of Q7, recorded as Q7′. In the same way, use the known coordinate value of the previous node, for example, the previous node of Q7 is Q6, the previous node of Q8 is Q5, and the line-plane space angle, and determine the final adjustment coordinate value by finding the minimum spatial distance. Adjust the values of all remaining coordinate points in turn.
调整之后的手部互动效果如图11所示，用户手势可以和视频中角色手势形成隔屏幕触碰的互动体验。The adjusted hand interaction effect is shown in Figure 11. User gestures can form a screen-touch interactive experience with the character gestures in the video.
这样，通过该实施方式，可依次对视频中角色手部各关节点的位置进行调整，最终形成用户手势与视频中角色手势形成隔屏幕触碰的互动体验，增加互动趣味性。In this way, through this implementation, the positions of the joints of the character's hand in the video can be adjusted sequentially, ultimately forming an interactive experience in which the user's gestures and the character's gestures in the video are touched across the screen, increasing the fun of interaction.
可选地，所述方法还包括： Optionally, the method also includes:
在用户选择进入演绎模式的情况下，显示视频配音界面，并在所述视频配音界面中显示辅助频谱图，其中，所述辅助频谱图是根据所述目标视频中的原声生成的频谱图；When the user chooses to enter the interpretation mode, display the video dubbing interface, and display an auxiliary spectrogram in the video dubbing interface, wherein the auxiliary spectrogram is a spectrogram generated based on the original sound in the target video;
获取用户根据所述辅助频谱图录入的配音音频；Obtain the dubbing audio recorded by the user based on the auxiliary spectrogram;
将所述配音音频融入所述目标视频中进行播放。The dubbing audio is integrated into the target video for playback.
一种实施方式中，当用户在创建的观影小室内观看视频时，小室内成员可以发起角色配音，由小室内成员完成视频中角色台词，将自己带入到剧情角色中。且可以引入了声纹匹配规则，可用于辅助角色台词配音，使得用户在声音演绎时有声纹参考。In one implementation, when a user watches a video in a created viewing booth, members of the booth can initiate character dubbing, and the members of the booth complete the character lines in the video, bringing themselves into the role of the plot. And voiceprint matching rules can be introduced, which can be used to assist the dubbing of character lines, so that users can have voiceprint reference when performing voice interpretation.
在成功创建观影小室后，可自动将视频中的音频提取并且生成声纹示意图也即辅助频谱图，如图12中的实线所示。After the viewing room is successfully created, the audio in the video can be automatically extracted and a voiceprint schematic diagram, also known as an auxiliary spectrogram, is generated, as shown by the solid line in Figure 12.
用户可启动情景带入，选择喜欢的角色和视频片段。此时可以选择多人合作演绎也可以选择单人独自演绎。多人合作时，每个人可选择各自的角色完成各自的声音部分即可。单人演绎时候，除自己挑选的角色，其他角色声音可以选择视频原音，也可以选择其他用户上传的配音。Users can activate scene introduction and select their favorite characters and video clips. At this time, you can choose to perform collaboratively with multiple people or perform alone. When multiple people cooperate, everyone can choose their own character to complete their own voice part. When performing a solo performance, in addition to the character you choose, you can choose the original video sound for other characters' voices, or you can choose the dubbing uploaded by other users.
此时视频画面从观影中切换出来，进入视频配音界面，原生辅助声纹即辅助频谱图在界面中弹出。用户在配音过程中可以参考辅助的配音声纹来调整自己的语调、语速，从而更好地完成声音演绎。At this time, the video screen is switched out of the movie viewing and the video dubbing interface is entered. The native auxiliary voiceprint, that is, the auxiliary spectrogram pops up in the interface. During the dubbing process, users can refer to the auxiliary dubbing voiceprint to adjust their intonation and speaking speed to better complete the voice interpretation.
在多人合作模式下分别获取多人声音整合成完整视频，单人演绎模式下，将单人与用户选择的其他角色原音或者其他用户声音合成，得到用户配音的视频。这样，实现了观影小室内用户之间，用户和视频直接的互动，增加观影的趣味性。In the multi-player cooperation mode, the voices of multiple people are obtained and integrated into a complete video. In the single-player interpretation mode, the original voices of a single person and other characters selected by the user or the voices of other users are synthesized to obtain a video dubbed by the user. In this way, direct interaction between users in the viewing booth and between users and videos is achieved, making watching movies more interesting.
进一步地，所述获取用户根据所述辅助频谱图录入的配音音频之后，所述方法还包括：Further, after obtaining the dubbing audio recorded by the user based on the auxiliary spectrogram, the method further includes:
对所述配音音频进行调整，使调整后的所述配音音频与所述辅助频谱图匹配；Adjust the dubbing audio so that the adjusted dubbing audio matches the auxiliary spectrogram;
所述将所述配音音频融入所述目标视频中进行播放，包括：The step of integrating the dubbing audio into the target video for playback includes:
将调整后的所述配音音频融入所述目标视频中进行播放。The adjusted dubbing audio is integrated into the target video for playback.
该实施方式中，还可引入声纹匹配规则用于修饰用户声音，对已经录入 系统的用户声音，通过声纹自动匹配来改善声纹，使得用户的声音更加贴合角色。In this embodiment, voiceprint matching rules can also be introduced to modify the user's voice. The system's user voice improves the voiceprint through automatic voiceprint matching, making the user's voice more suitable for the character.
具体地，在用户结束配音后，系统可根据录入的声纹和原声声纹匹配。如图12所示，用户录入的声纹即频谱如虚线所示，原声声纹如实线所示。用户声纹和原视频声纹必然存在偏差，该实施方式中，可通过计算声域内方差来匹配原视频声纹声域值，提高用户声音贴合度。Specifically, after the user finishes dubbing, the system can match the recorded voiceprint with the original voiceprint. As shown in Figure 12, the voiceprint entered by the user, that is, the spectrum, is shown as a dotted line, and the original voiceprint is shown as a solid line. There is bound to be a deviation between the user's voiceprint and the original video's voiceprint. In this embodiment, the voice domain value of the original video's voiceprint can be matched by calculating the variance within the voice domain to improve the fit of the user's voice.
进一步地，所述获取用户根据所述辅助频谱图录入的配音音频之后，所述方法还包括：Further, after obtaining the dubbing audio recorded by the user based on the auxiliary spectrogram, the method further includes:
根据所述配音音频与所述辅助频谱图中各频点的差值，计算所述配音音频的得分；Calculate the score of the dubbing audio according to the difference between each frequency point in the dubbing audio and the auxiliary spectrogram;
所述将所述配音音频融入所述目标视频中进行播放，包括：The step of integrating the dubbing audio into the target video for playback includes:
将所述配音音频融入所述目标视频中进行播放，并显示所述得分。The dubbing audio is integrated into the target video for playback, and the score is displayed.
一种实施方式中，还可引入声纹匹配规则用于用户评分机制，对同一个角色的所有用户配音进行打分，得分靠前的将被收录在榜单集合中。In one implementation, voiceprint matching rules can also be introduced for the user scoring mechanism to score all users' dubbing of the same character, and those with the top scores will be included in the list collection.
具体地，用户在完成演绎后，可计算用户配音音频中每个声纹点与原声纹的差值。假设原声纹共有n个声纹点，当用户声纹与原声声纹的差值占原声纹值为0％时，可得一个单位分，当差值占原声纹值在50％以上时，得0个单位分，当差值占原声纹值介于0～50％时，每升高10个百分点，降低0.2个单位分。每个声纹点的单位分在集合λ中，用户录入的声纹集合为X，原声纹集合为N。用户声纹集合中的某个用户声纹值为xi，对应的原声纹值为ni，则该用户的总得分R的计算公式如下所示：
Specifically, after the user completes the interpretation, the difference between each voiceprint point in the user's dubbing audio and the original voiceprint can be calculated. Assume that the original voiceprint has n voiceprint points. When the difference between the user's voiceprint and the original voiceprint accounts for 0% of the original voiceprint value, one unit point can be obtained. When the difference accounts for more than 50% of the original voiceprint value, we get 0 unit points, when the difference accounts for 0 to 50% of the original voiceprint value, it will decrease by 0.2 unit points for every 10 percentage points increase. The unit of each voiceprint point is divided into a set λ, the voiceprint set entered by the user is X, and the original voiceprint set is N. If a user's voiceprint value in the user voiceprint set is x i and the corresponding original voiceprint value is n i , then the calculation formula of the user's total score R is as follows:
这样，在对用户配音音频进行打分后，可在用户配音视频中显示其得分，在多人合作模式下，可在合成的配音视频中显示各用户的配音得分，并按得分高低进行排序。In this way, after the user's dubbing audio is scored, the score can be displayed in the user's dubbing video. In the multi-player cooperation mode, the dubbing score of each user can be displayed in the synthesized dubbing video and sorted by score.
这样，通过这种情景带入模式，能够很好地促进小室内视频和用户之间，以及用户与用户之间的互动。In this way, through this situational introduction mode, the interaction between indoor video and users, as well as between users can be well promoted.
可选地，所述方法还包括：Optionally, the method also includes:
获取用户在所述观影小室中观看所述目标视频时于第一时段内发布的弹 幕和弹幕数量；Obtain the bullets posted by the user in the first period when watching the target video in the viewing room. Number of scenes and barrages;
确定所述第一时段内发布的各弹幕对应的情感词，并统计各情感词的数量；Determine the emotional words corresponding to each barrage posted within the first period, and count the number of each emotional word;
在第一情感词的数量占所述弹幕数量的比值大于预设阈值的情况下，在所述观影小室中添加与所述第一情感词相关的特效挂件，或者，播放与所述第一情感词相关的音效，其中，所述第一情感词为任一情感词。When the ratio of the number of first emotional words to the number of comments is greater than a preset threshold, add a special effect pendant related to the first emotional word in the viewing room, or play a special effect pendant related to the first emotional word. A sound effect related to an emotional word, wherein the first emotional word is any emotional word.
一种实施方式中，还可基于观影小室内的弹幕对视频情景做渲染，通过对弹幕语义分析提取相应的情感词，根据不同的情感词配备特效挂件、音效等。In one implementation, the video scene can also be rendered based on the barrages in the viewing booth, the corresponding emotional words are extracted through semantic analysis of the barrages, and special effect pendants, sound effects, etc. are equipped according to different emotional words.
具体实现方式为，观影小室内用户可在观看视频时发布弹幕；统计某个时间段内的弹幕数量M，如统计10分钟内的弹幕数量；对该时段内的每一个弹幕抽取现有情感词，比如恐怖，欢快，悲伤等，抽取方式可以是关键词匹配和自然语言处理算法中的语义理解；统计情感词数量N，当N/M大于某个阈值时触发情景渲染，其中，由于不同观众对于同一个剧情有不同观感从而产生不同情感，这里先对情感词做相似性分析，计算一个或几个具有代表性的情感词，然后统计各情感词的数量；结合计算出的情感词，在观影小室内播放的视频画面中自动添加相关的特效挂件，还可以情感词为搜索匹配词来获取并播放相应的音效。例如，对于欢乐的情感词，可以在视频画面中添加小星星掉落的特效，还可调取欢快的音效进行播放。The specific implementation method is that users in the viewing booth can post barrages while watching videos; count the number M of barrages within a certain period of time, such as counting the number of barrages within 10 minutes; count each barrage within this period Extract existing emotional words, such as horror, joy, sadness, etc. The extraction method can be keyword matching and semantic understanding in natural language processing algorithms; count the number of emotional words N, and trigger scene rendering when N/M is greater than a certain threshold. Among them, since different viewers have different perceptions of the same plot and thus produce different emotions, here we first perform a similarity analysis on the emotional words, calculate one or several representative emotional words, and then count the number of each emotional word; the combination is calculated Emotional words can be used to automatically add relevant special effects widgets to the video played in the viewing room. Emotional words can also be used as search matching words to obtain and play corresponding sound effects. For example, for happy emotional words, you can add the special effect of falling stars to the video screen, and you can also call up cheerful sound effects for playback.
这样，通过该实施方式中的情景渲染，可以让剧情更加灵动，进一步增加互动趣味性。In this way, through the scene rendering in this embodiment, the plot can be made more flexible and the interactive interest further increased.
相关的大众观影技术缺乏针对某些用户群体的个性化定制。本公开提出了一种基于弹幕的观影小室互动方法。用户可以选择志趣相同的用户群体构建观影小室，比如某个明星追随者群体同步观影，小室内的观影私密性更好。在观影小室内还可以进行各种互动，比如肢体互动，情景带入以及弹幕渲染等，大大增加了视频观看的趣味性，将观众从观看者转变成了视频演绎的参与者。Relevant mass viewing technology lacks personalized customization for certain user groups. This disclosure proposes a barrage-based interactive method for viewing rooms. Users can choose user groups with the same interests to build movie-watching cabins. For example, if a group of celebrity followers watches movies simultaneously, the viewing privacy in the cabin will be better. Various interactions can also be carried out in the viewing booth, such as physical interaction, scene introduction and barrage rendering, etc., which greatly increases the fun of video watching and transforms the audience from a viewer into a participant in the video interpretation.
本公开实施例的观影互动方法，在用户观看目标视频的过程中，获取用户图像；识别所述用户图像中的目标对象的姿态；基于所述目标对象的姿态， 确定所述目标视频中的目标角色的回应姿态；基于所述目标对象的目标部位的姿态，对所述目标角色的目标部位的回应姿态进行调整；基于调整后的所述目标角色的目标部位的回应姿态，在所述目标视频的视频画面中显示所述目标角色的回应姿态。这样，用户可以与视频中角色进行互动，使得互动方式不限于弹幕，增加了互动的趣味性。The movie-viewing interaction method of the embodiment of the present disclosure acquires the user image while the user watches the target video; identifies the posture of the target object in the user image; based on the posture of the target object, Determine the response posture of the target character in the target video; adjust the response posture of the target part of the target character based on the posture of the target part of the target object; based on the adjusted target part of the target character The response gesture displays the response gesture of the target character in the video screen of the target video. In this way, users can interact with the characters in the video, so that the interaction method is not limited to barrages, making the interaction more interesting.
本公开实施例还提供了一种观影互动装置。参见图13，图13是本公开实施例提供的观影互动装置的结构图。由于观影互动装置解决问题的原理与本公开实施例中观影互动方法相似，因此该观影互动装置的实施可以参见方法的实施，重复之处不再赘述。An embodiment of the present disclosure also provides a movie viewing interactive device. Referring to Figure 13, Figure 13 is a structural diagram of an interactive device for watching movies provided by an embodiment of the present disclosure. Since the problem-solving principle of the interactive viewing device is similar to the interactive viewing method in the embodiment of the present disclosure, the implementation of the interactive viewing device can be referred to the implementation of the method, and repeated details will not be repeated.
如图13所示，观影互动装置1300包括：As shown in Figure 13, the movie viewing interactive device 1300 includes:
第一获取模块1301，用于在用户观看目标视频的过程中，获取用户图像；The first acquisition module 1301 is used to acquire user images while the user is watching the target video;
识别模块1302，用于识别所述用户图像中的目标对象的姿态；Recognition module 1302, used to recognize the posture of the target object in the user image;
第一确定模块1303，用于基于所述目标对象的姿态，确定所述目标视频中的目标角色的回应姿态；The first determination module 1303 is configured to determine the response posture of the target character in the target video based on the posture of the target object;
第一调整模块1304，用于基于所述目标对象的目标部位的姿态，对所述目标角色的目标部位的回应姿态进行调整；The first adjustment module 1304 is configured to adjust the response posture of the target part of the target character based on the posture of the target part of the target object;
第一显示模块1305，用于基于调整后的所述目标角色的目标部位的回应姿态，在所述目标视频的视频画面中显示所述目标角色的回应姿态。The first display module 1305 is configured to display the response gesture of the target character in the video screen of the target video based on the adjusted response gesture of the target part of the target character.
可选地，识别模块1302包括：Optionally, the identification module 1302 includes:
识别单元，用于通过目标检测算法识别所述用户图像中的目标对象；An identification unit configured to identify the target object in the user image through a target detection algorithm;
构建单元，用于构建所述目标对象的骨骼模型；A construction unit used to construct a skeleton model of the target object;
第一确定单元，用于基于所述目标对象的骨骼模型，确定所述目标对象的姿态。A first determination unit configured to determine the posture of the target object based on the skeletal model of the target object.
可选地，观影互动装置1300还包括：Optionally, the interactive viewing device 1300 also includes:
第二获取模块，用于获取所述目标视频中的目标角色的骨骼模型；The second acquisition module is used to acquire the skeleton model of the target character in the target video;
匹配模块，用于对所述目标对象的骨骼模型与所述目标角色的骨骼模型进行骨骼关键点匹配，以调整所述目标角色的回应姿态。A matching module, configured to match skeletal key points between the skeletal model of the target object and the skeletal model of the target character to adjust the response posture of the target character.
可选地，第一显示模块1305还用于基于所述目标对象的目标部位的姿态和调整后的所述目标角色的目标部位的回应姿态，在所述视频画面中显示所 述目标对象的目标部位与所述目标角色的目标部位的互动效果。Optionally, the first display module 1305 is further configured to display the target part in the video screen based on the posture of the target part of the target object and the adjusted response posture of the target part of the target character. The interactive effect between the target part of the target object and the target part of the target character.
可选地，第一调整模块1304包括：Optionally, the first adjustment module 1304 includes:
第二确定单元，用于确定包围所述目标对象的目标部位的三维模型的第一最小外接球；a second determination unit configured to determine the first minimum circumscribed sphere of the three-dimensional model surrounding the target part of the target object;
第三确定单元，用于确定包围所述目标角色的目标部位的三维模型的第二最小外接球；a third determination unit configured to determine the second minimum circumscribed sphere of the three-dimensional model surrounding the target part of the target character;
投影单元，用于根据所述第一最小外接球的球心位置，将所述第一最小外接球投影到所述目标视频的视频画面中；A projection unit configured to project the first minimum circumscribed ball into the video picture of the target video according to the center position of the first minimum circumscribed ball;
第一调整单元，用于根据所述第一最小外接球投影在所述视频画面中的位置，对所述第二最小外接球的位置进行调整；A first adjustment unit configured to adjust the position of the second minimum circumscribed ball according to the position of the first minimum circumscribed ball projected in the video screen;
第二调整单元，用于根据调整后的所述第二最小外接球的位置，调整所述目标角色的目标部位的回应姿态。The second adjustment unit is configured to adjust the response posture of the target part of the target character according to the adjusted position of the second minimum external ball.
可选地，所述第一调整单元包括：Optionally, the first adjustment unit includes:
第一调整子单元，用于根据所述第一最小外接球投影在所述视频画面中的球心位置，调整所述第二最小外接球的球心位置；A first adjustment subunit configured to adjust the center position of the second minimum circumscribed ball according to the center position of the first minimum circumscribed ball projected in the video screen;
确定子单元，用于根据所述目标角色的目标部位的三维模型中任两点间的夹角关系，确定所述目标角色的目标部位中各点的位置；Determining subunit, used to determine the position of each point in the target part of the target character based on the angle relationship between any two points in the three-dimensional model of the target part of the target character;
所述第二调整单元用于根据调整后的所述第二最小外接球的球心位置，以及所述目标角色的目标部位中各点的位置，调整所述目标角色的目标部位的回应姿态。The second adjustment unit is configured to adjust the response posture of the target part of the target character according to the adjusted center position of the second minimum external ball and the positions of each point in the target part of the target character.
可选地，所述目标部位为手部；第一调整模块1304包括：Optionally, the target part is the hand; the first adjustment module 1304 includes:
建立单元，用于以所述目标对象的手部的第一手掌根节点为原点建立三维坐标系，其中，所述目标对象的手部包括大拇指关节点、食指关节点、中指关节点、无名指关节点和小指关节点；An establishment unit configured to establish a three-dimensional coordinate system with the first palm heel node of the target object's hand as the origin, wherein the target object's hand includes a thumb joint point, an index finger joint point, a middle finger joint point, and a ring finger joint point. joints and little finger joints;
第一计算单元，用于计算第一空间角、第二空间角、第三空间角和第四空间角；其中，所述第一空间角是以所述第一手掌根节点为顶点且以第一线段和第二线段为边的空间夹角，所述第二空间角是以所述第一手掌根节点为顶点且以所述第二线段和第三线段为边的空间夹角，所述第三空间角是以所述第一手掌根节点为顶点且以所述第三线段和第四线段为边的空间夹角，所 述第四空间角是以所述第一手掌根节点为顶点且以所述第四线段和第五线段为边的空间夹角；所述第一线段为所述第一手掌根节点与相邻的大拇指关节点的线段，所述第二线段为所述第一手掌根节点与相邻的食指关节点的线段，所述第三线段为所述第一手掌根节点与相邻的中指关节点的线段，所述第四线段为所述第一手掌根节点与相邻的无名指关节点的线段，所述第五线段为所述第一手掌根节点与相邻的小指关节点的线段；The first calculation unit is used to calculate the first spatial angle, the second spatial angle, the third spatial angle and the fourth spatial angle; wherein the first spatial angle is based on the first palm heel node as the vertex and the third spatial angle. The first line segment and the second line segment are the angles in space between the sides, and the second space angle is the angle in space with the first palm heel node as the vertex and the second line segment and the third line segment as the sides, so The third space angle is the space angle with the first palm heel node as the vertex and the third line segment and the fourth line segment as the sides, so The fourth space angle is the space angle with the first palm heel node as the vertex and the fourth line segment and the fifth line segment as the sides; the first line segment is the first palm heel node and the corresponding space angle. The second line segment is the line segment between the first palm heel node and the adjacent index finger joint point, and the third line segment is the line segment between the first palm heel node and the adjacent middle finger joint. The line segment of the joint point, the fourth line segment is the line segment between the first palm heel node and the adjacent ring finger joint point, the fifth line segment is the line segment between the first palm heel node and the adjacent little finger joint point ;
第二计算单元，用于计算第一线面空间角、第二线面空间角、第三线面空间角、第四线面空间角和第五线面空间角；其中，所述第一线面空间角是以所述目标对象的手部的大拇指关节点所在线段与第一面的空间夹角，所述第二线面空间角是以所述目标对象的手部的食指关节点所在线段与所述第一面的空间夹角，所述第三线面空间角是以所述目标对象的手部的中指关节点所在线段与所述第一面的空间夹角，所述第四线面空间角是以所述目标对象的手部的无名指关节点所在线段与所述第一面的空间夹角，所述第五线面空间角是以所述目标对象的手部的小指关节点所在线段与所述第一面的空间夹角；所述第一面为所述三维坐标系中z轴和y轴所构成的平面；The second calculation unit is used to calculate the first line-plane space angle, the second line-plane space angle, the third line-plane space angle, the fourth line-plane space angle and the fifth line-plane space angle; wherein, the first line-plane space angle The angle is the space angle between the line segment where the thumb joint point of the target object's hand is located and the first plane, and the second line-plane space angle is the line segment where the index finger joint point of the target object's hand is located. The spatial angle between the third line and the first surface is the spatial angle between the line segment where the middle finger joint point of the target object's hand is and the first surface. The fourth line is The surface-space angle is the space angle between the line segment where the ring finger joint point of the target object's hand is located and the first surface, and the fifth line-surface space angle is the space angle between the little finger joint of the target object's hand. The spatial angle between the line segment where the point is located and the first surface; the first surface is the plane formed by the z-axis and the y-axis in the three-dimensional coordinate system;
第三调整单元，用于根据所述第一空间角、第二空间角、第三空间角和第四空间角，以及所述第一线面空间角、第二线面空间角、第三线面空间角、第四线面空间角和第五线面空间角，调整所述目标角色的手部的大拇指关节点、食指关节点、中指关节点、无名指关节点和小指关节点的位置。The third adjustment unit is used to adjust the first space angle, the second space angle, the third space angle and the fourth space angle according to the first line-plane space angle, the second line-plane space angle and the third line-plane space angle. angle, the fourth line-plane space angle and the fifth line-plane space angle, and adjust the positions of the thumb joint point, index finger joint point, middle finger joint point, ring finger joint point and little finger joint point of the target character's hand.
可选地，所述第三调整单元包括：Optionally, the third adjustment unit includes:
第二调整子单元，用于令第五空间角等于所述第一空间角，第六空间角等于所述第二空间角，第七空间角等于所述第三空间角，第八空间角等于所述第四空间角，调整所述目标角色的手部的第一大拇指关节点、第一食指关节点、第一中指关节点、第一无名指关节点和第一小指关节点的位置；The second adjustment subunit is used to make the fifth space angle equal to the first space angle, the sixth space angle equal to the second space angle, the seventh space angle equal to the third space angle, and the eighth space angle equal to The fourth space angle adjusts the positions of the first thumb joint point, first index finger joint point, first middle finger joint point, first ring finger joint point and first little finger joint point of the target character's hand;
第三调整子单元，用于令第六线面空间角等于所述第一线面空间角，第七线面空间角等于所述第二线面空间角，第八线面空间角等于所述第三线面空间角，第九线面空间角等于所述第四线面空间角，第十线面空间角等于所述第五线面空间角，调整所述目标角色的手部的除所述第一大拇指关节点外的其他大拇指关节点、除所述第一食指关节点外的其他食指关节点、除所述 第一中指关节点外的其他中指关节点、除所述第一无名指关节点外的其他无名指关节点和除所述第一小指关节点外的其他小指关节点的位置；The third adjustment subunit is used to make the sixth line-plane space angle equal to the first line-plane space angle, the seventh line-plane space angle equal to the second line-plane space angle, and the eighth line-plane space angle equal to the third line-plane space angle. Three line and plane space angles, the ninth line and plane space angle is equal to the fourth line and plane space angle, and the tenth line and plane space angle is equal to the fifth line and plane space angle. Adjust the angle of the hand of the target character except for the third line and plane space angle. Other thumb joint points except the first thumb joint point, other index finger joint points except the first index finger joint point, except the above-mentioned first index finger joint point The positions of other middle finger joint points except the first middle finger joint point, other ring finger joint points except the first ring finger joint point, and other little finger joint points except the first little finger joint point;
其中，所述第五空间角是以所述目标角色的手部的第二手掌根节点为顶点且以第六线段和第七线段为边的空间夹角，所述第六空间角是以所述第二手掌根节点为顶点且以所述第七线段和第八线段为边的空间夹角，所述第七空间角是以所述第二手掌根节点为顶点且以第八线段和第九线段为边的空间夹角，所述第八空间角是以所述第二手掌根节点为顶点且以第九线段和第十线段为边的空间夹角；所述第六线段为所述第二手掌根节点与相邻的所述第一大拇指关节点的线段，所述第七线段为所述第二手掌根节点与相邻的所述第一食指关节点的线段，所述第八线段为所述第二手掌根节点与相邻的所述第一中指关节点的线段，所述第九线段为所述第二手掌根节点与相邻的所述第一无名指关节点的线段，所述第十线段为所述第二手掌根节点与相邻的所述第一小指关节点的线段；Wherein, the fifth space angle is the space angle with the second palm root node of the target character's hand as the vertex and the sixth line segment and the seventh line segment as the sides, and the sixth space angle is based on the The space angle between the second palm heel node as a vertex and the seventh line segment and the eighth line segment as sides, and the seventh space angle is the second palm heel node as a vertex and the eighth line segment and the eighth line segment as sides. The nine line segments are the space angles of the sides, and the eighth space angle is the space angle with the second palm heel node as the vertex and the ninth line segment and the tenth line segment as the sides; the sixth line segment is the space angle. The line segment between the second palm heel node and the adjacent first thumb joint point, the seventh line segment is the line segment between the second palm heel node and the adjacent first index finger joint point, and the seventh line segment is the line segment between the second palm heel node and the adjacent first index finger joint point. The eighth line segment is a line segment between the second palm heel node and the adjacent first middle finger joint point, and the ninth line segment is a line segment between the second palm heel node and the adjacent first ring finger joint point. , the tenth line segment is a line segment between the second palm heel node and the adjacent first little finger joint point;
所述第六线面空间角以所述目标角色的手部的大拇指关节点所在线段与第二面的空间夹角，所述第七线面空间角是以所述目标角色的手部的食指关节点所在线段与所述第二面的空间夹角，所述第八线面空间角是以所述目标角色的手部的中指关节点所在线段与所述第二面的空间夹角，所述第九线面空间角是以所述目标角色的手部的无名指关节点所在线段与所述第二面的空间夹角，所述第十线面空间角是以所述目标角色的手部的小指关节点所在线段与所述第二面的空间夹角；所述第二面为以所述第二手掌根节点为原点建立的三维坐标系中z轴和y轴所构成的平面。The sixth line-plane space angle is the space angle between the line segment where the thumb joint point of the target character's hand is located and the second surface, and the seventh line-plane space angle is the space angle between the target character's hand and the second surface. The angle between the line segment where the joint point of the index finger is located and the second surface. The eighth line-plane space angle is the space between the line segment where the joint point of the middle finger of the hand of the target character is located and the second surface. The included angle, the ninth line-plane space angle is the space angle between the line segment where the ring finger joint point of the target character's hand is located and the second surface, and the tenth line-plane space angle is the space angle between the The angle between the line segment where the little finger joint point of the target character's hand is located and the second surface; the second surface is the z-axis and y-axis in the three-dimensional coordinate system established with the second palm root node as the origin. The plane formed.
可选地，观影互动装置1300还包括：Optionally, the movie viewing interactive device 1300 also includes:
接收模块，用于接收第一用户在观看所述目标视频中针对第一弹幕的输入，其中，所述输入用于指示对第二用户发起观影小室创建邀请，所述第二用户为发布所述第一弹幕的用户；A receiving module configured to receive input from the first user regarding the first barrage while watching the target video, wherein the input is used to instruct the second user to initiate an invitation to create a viewing room, and the second user is a publisher. The user of the first barrage;
创建模块，用于在所述第二用户接受所述观影小室创建邀请的情况下，创建所述第一用户与所述第二用户的观影小室，并在所述观影小室中播放所述目标视频；A creation module, configured to create a viewing cabin for the first user and the second user when the second user accepts the invitation to create the viewing cabin, and play all the viewing cabins in the viewing cabin. Describe the target video;
第一获取模块1301用于在用户在所述观影小室中观看所述目标视频的 过程中，获取用户图像。The first acquisition module 1301 is used to obtain the target video when the user watches the target video in the viewing room. During the process, the user image is obtained.
可选地，观影互动装置1300还包括：Optionally, the interactive viewing device 1300 also includes:
第二显示模块，用于在用户选择进入演绎模式的情况下，显示视频配音界面，并在所述视频配音界面中显示辅助频谱图，其中，所述辅助频谱图是根据所述目标视频中的原声生成的频谱图；The second display module is used to display the video dubbing interface when the user chooses to enter the interpretation mode, and to display an auxiliary spectrogram in the video dubbing interface, wherein the auxiliary spectrogram is based on the target video. Spectrogram generated from the original sound;
第三获取模块，用于获取用户根据所述辅助频谱图录入的配音音频；The third acquisition module is used to acquire the dubbing audio input by the user according to the auxiliary spectrogram;
播放模块，用于将所述配音音频融入所述目标视频中进行播放。A playback module is used to integrate the dubbing audio into the target video for playback.
可选地，观影互动装置1300还包括：Optionally, the interactive viewing device 1300 also includes:
第二调整模块，用于对所述配音音频进行调整，使调整后的所述配音音频与所述辅助频谱图匹配；The second adjustment module is used to adjust the dubbing audio so that the adjusted dubbing audio matches the auxiliary spectrogram;
所述播放模块用于将调整后的所述配音音频融入所述目标视频中进行播放。The playback module is used to integrate the adjusted dubbing audio into the target video for playback.
可选地，观影互动装置1300还包括：Optionally, the interactive viewing device 1300 also includes:
计算模块，用于根据所述配音音频与所述辅助频谱图中各频点的差值，计算所述配音音频的得分；A calculation module configured to calculate the score of the dubbing audio based on the difference between the dubbing audio and each frequency point in the auxiliary spectrogram;
所述播放模块用于将所述配音音频融入所述目标视频中进行播放，并显示所述得分。The playback module is used to integrate the dubbing audio into the target video for playback, and display the score.
可选地，观影互动装置1300还包括：Optionally, the interactive viewing device 1300 also includes:
第四获取模块，用于获取用户在所述观影小室中观看所述目标视频时于第一时段内发布的弹幕和弹幕数量；The fourth acquisition module is used to obtain the barrages and the number of barrages posted by the user in the first period when watching the target video in the viewing room;
第二确定模块，用于确定所述第一时段内发布的各弹幕对应的情感词，并统计各情感词的数量；The second determination module is used to determine the emotional words corresponding to each barrage released within the first period, and count the number of each emotional word;
处理模块，用于在第一情感词的数量占所述弹幕数量的比值大于预设阈值的情况下，在所述观影小室中添加与所述第一情感词相关的特效挂件，或者，播放与所述第一情感词相关的音效，其中，所述第一情感词为任一情感词。A processing module configured to add a special effect pendant related to the first emotional word in the viewing room when the ratio of the number of first emotional words to the number of barrages is greater than a preset threshold, or, Play sound effects related to the first emotional word, where the first emotional word is any emotional word.
本公开实施例提供的观影互动装置1300，可以执行上述方法实施例，其实现原理和技术效果类似，本实施例此处不再赘述。The movie viewing interactive device 1300 provided by the embodiment of the present disclosure can execute the above method embodiments, and its implementation principles and technical effects are similar, and will not be described again in this embodiment.
本公开实施例的观影互动装置1300，在用户观看目标视频的过程中，获 取用户图像；识别所述用户图像中的目标对象的姿态；基于所述目标对象的姿态，确定所述目标视频中的目标角色的回应姿态；基于所述目标对象的目标部位的姿态，对所述目标角色的目标部位的回应姿态进行调整；基于调整后的所述目标角色的目标部位的回应姿态，在所述目标视频的视频画面中显示所述目标角色的回应姿态。这样，用户可以与视频中角色进行互动，使得互动方式不限于弹幕，增加了互动的趣味性。The movie-viewing interactive device 1300 in the embodiment of the present disclosure obtains Obtain the user image; identify the posture of the target object in the user image; determine the response posture of the target character in the target video based on the posture of the target object; based on the posture of the target part of the target object, determine the response posture of the target character in the target video. The response posture of the target part of the target character is adjusted; based on the adjusted response posture of the target part of the target character, the response posture of the target character is displayed in the video screen of the target video. In this way, users can interact with the characters in the video, so that the interaction method is not limited to barrages, making the interaction more interesting.
此外，本公开实施例的计算机可读存储介质，用于存储计算机程序，所述计算机程序可被处理器执行实现图1所示方法实施例中各个步骤。In addition, the computer-readable storage medium of the embodiment of the present disclosure is used to store a computer program, and the computer program can be executed by a processor to implement each step in the method embodiment shown in Figure 1.
在本公开所提供的几个实施例中，应该理解到，所揭露方法和装置，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。In the several embodiments provided in this disclosure, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.
另外，在本公开各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理包括，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用硬件加软件功能单元的形式实现。In addition, each functional unit in various embodiments of the present disclosure may be integrated into one processing unit, each unit may be physically included separately, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units.
上述以软件功能单元的形式实现的集成的单元，可以存储在一个计算机可读取存储介质中。上述软件功能单元存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本公开各个实施例所述收发方法的部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute some steps of the sending and receiving methods described in various embodiments of the present disclosure. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program code. .
以上所述是本公开的优选实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本公开所述原理的前提下，还可以作出若干改进和润饰，这些改进和润饰也应视为本公开的保护范围。 The above are the preferred embodiments of the present disclosure. It should be noted that for those of ordinary skill in the art, several improvements and modifications can be made without departing from the principles described in the present disclosure. These improvements and modifications can also be made. should be regarded as the scope of protection of this disclosure.

Claims (10)

  1. 一种观影互动方法，包括：An interactive method for watching movies, including:
    在用户观看目标视频的过程中，获取用户图像；While the user is watching the target video, the user image is obtained;
    识别所述用户图像中的目标对象的姿态；identifying the pose of the target object in the user image;
    基于所述目标对象的姿态，确定所述目标视频中的目标角色的回应姿态；Based on the posture of the target object, determine the response posture of the target character in the target video;
    基于所述目标对象的目标部位的姿态，对所述目标角色的目标部位的回应姿态进行调整；Based on the posture of the target part of the target object, adjust the response posture of the target part of the target character;
    基于调整后的所述目标角色的目标部位的回应姿态，在所述目标视频的视频画面中显示所述目标角色的回应姿态。Based on the adjusted response posture of the target part of the target character, the response posture of the target character is displayed in the video frame of the target video.
  2. 根据权利要求1所述的方法，其中，所述识别所述用户图像中的目标对象的姿态，包括：The method according to claim 1, wherein the identifying the gesture of the target object in the user image includes:
    通过目标检测算法识别所述用户图像中的目标对象；Identify the target object in the user image through a target detection algorithm;
    构建所述目标对象的骨骼模型；Construct a skeletal model of the target object;
    基于所述目标对象的骨骼模型，确定所述目标对象的姿态；Determine the posture of the target object based on the skeletal model of the target object;
    所述基于所述目标对象的姿态，确定所述目标视频中的目标角色的回应姿态之后，所述方法还包括：After determining the response posture of the target character in the target video based on the posture of the target object, the method further includes:
    获取所述目标视频中的目标角色的骨骼模型；Obtain the skeleton model of the target character in the target video;
    对所述目标对象的骨骼模型与所述目标角色的骨骼模型进行骨骼关键点匹配，以调整所述目标角色的回应姿态。Skeleton key point matching is performed on the skeletal model of the target object and the skeletal model of the target character to adjust the response posture of the target character.
  3. 根据权利要求1所述的方法，其中，所述基于所述目标对象的目标部位的姿态，对所述目标角色的目标部位的回应姿态进行调整，包括：The method according to claim 1, wherein adjusting the response posture of the target part of the target character based on the posture of the target part of the target object includes:
    确定包围所述目标对象的目标部位的三维模型的第一最小外接球；Determine the first minimum circumscribed sphere of the three-dimensional model surrounding the target part of the target object;
    确定包围所述目标角色的目标部位的三维模型的第二最小外接球；Determine the second minimum circumscribed sphere of the three-dimensional model surrounding the target part of the target character;
    根据所述第一最小外接球的球心位置，将所述第一最小外接球投影到所述目标视频的视频画面中；Project the first minimum circumscribed ball into the video frame of the target video according to the center position of the first minimum circumscribed ball;
    根据所述第一最小外接球投影在所述视频画面中的位置，对所述第二最小外接球的位置进行调整；Adjust the position of the second minimum circumscribed ball according to the projected position of the first minimal circumscribed ball in the video screen;
    根据调整后的所述第二最小外接球的位置，调整所述目标角色的目标部 位的回应姿态。Adjust the target part of the target character according to the adjusted position of the second smallest external ball. bit of response.
  4. 根据权利要求3所述的方法，其中，所述根据所述第一最小外接球投影在所述视频画面中的位置，对所述第二最小外接球的位置进行调整，包括：The method according to claim 3, wherein adjusting the position of the second minimum circumscribed ball according to the projected position of the first minimal circumscribed ball in the video screen includes:
    根据所述第一最小外接球投影在所述视频画面中的球心位置，调整所述第二最小外接球的球心位置；Adjust the center position of the second minimum circumscribed ball according to the center position of the first minimum circumscribed ball projected in the video screen;
    根据所述目标角色的目标部位的三维模型中任两点间的夹角关系，确定所述目标角色的目标部位中各点的位置；Determine the position of each point in the target part of the target character based on the angle relationship between any two points in the three-dimensional model of the target part of the target character;
    所述根据调整后的所述第二最小外接球的位置，调整所述目标角色的目标部位的回应姿态，包括：Adjusting the response posture of the target part of the target character according to the adjusted position of the second minimum external ball includes:
    根据调整后的所述第二最小外接球的球心位置，以及所述目标角色的目标部位中各点的位置，调整所述目标角色的目标部位的回应姿态。The response posture of the target part of the target character is adjusted according to the adjusted center position of the second minimum external ball and the positions of each point in the target part of the target character.
  5. 根据权利要求1所述的方法，其中，所述目标部位为手部；The method according to claim 1, wherein the target part is a hand;
    所述基于所述目标对象的目标部位的姿态，对所述目标角色的目标部位的回应姿态进行调整，包括：Adjusting the response posture of the target part of the target character based on the posture of the target part of the target object includes:
    以所述目标对象的手部的第一手掌根节点为原点建立三维坐标系，其中，所述目标对象的手部包括大拇指关节点、食指关节点、中指关节点、无名指关节点和小指关节点；A three-dimensional coordinate system is established with the first palm heel node of the target object's hand as the origin, wherein the target object's hand includes a thumb joint point, an index finger joint point, a middle finger joint point, a ring finger joint point and a little finger joint point. point;
    计算第一空间角、第二空间角、第三空间角和第四空间角；其中，所述第一空间角是以所述第一手掌根节点为顶点且以第一线段和第二线段为边的空间夹角，所述第二空间角是以所述第一手掌根节点为顶点且以所述第二线段和第三线段为边的空间夹角，所述第三空间角是以所述第一手掌根节点为顶点且以所述第三线段和第四线段为边的空间夹角，所述第四空间角是以所述第一手掌根节点为顶点且以所述第四线段和第五线段为边的空间夹角；所述第一线段为所述第一手掌根节点与相邻的大拇指关节点的线段，所述第二线段为所述第一手掌根节点与相邻的食指关节点的线段，所述第三线段为所述第一手掌根节点与相邻的中指关节点的线段，所述第四线段为所述第一手掌根节点与相邻的无名指关节点的线段，所述第五线段为所述第一手掌根节点与相邻的小指关节点的线段；Calculate the first space angle, the second space angle, the third space angle and the fourth space angle; wherein, the first space angle is based on the first palm heel node as the vertex and the first line segment and the second line segment is the space angle of the side, the second space angle is the space angle with the first palm heel node as the vertex and the second line segment and the third line segment as the side, and the third space angle is The space angle between the first palm heel node as a vertex and the third line segment and the fourth line segment as sides, and the fourth space angle is a space angle with the first palm heel node as a vertex and the fourth line segment as an edge. The line segment and the fifth line segment are the spatial angles of the sides; the first line segment is the line segment between the first palm heel node and the adjacent thumb joint point, and the second line segment is the first palm heel node. and the adjacent index finger joint point, the third line segment is the line segment between the first palm heel node and the adjacent middle finger joint point, and the fourth line segment is the line segment between the first palm heel node and the adjacent middle finger joint point. The line segment of the ring finger joint point, the fifth line segment is the line segment of the first palm heel node and the adjacent little finger joint point;
    计算第一线面空间角、第二线面空间角、第三线面空间角、第四线面空 间角和第五线面空间角；其中，所述第一线面空间角是以所述目标对象的手部的大拇指关节点所在线段与第一面的空间夹角，所述第二线面空间角是以所述目标对象的手部的食指关节点所在线段与所述第一面的空间夹角，所述第三线面空间角是以所述目标对象的手部的中指关节点所在线段与所述第一面的空间夹角，所述第四线面空间角是以所述目标对象的手部的无名指关节点所在线段与所述第一面的空间夹角，所述第五线面空间角是以所述目标对象的手部的小指关节点所在线段与所述第一面的空间夹角；所述第一面为所述三维坐标系中z轴和y轴所构成的平面；Calculate the first line and surface space angle, the second line and surface space angle, the third line and surface space angle, and the fourth line and surface space angle. angle and the fifth line-plane space angle; wherein, the first line-plane space angle is the space angle between the line segment where the thumb joint point of the target object's hand is located and the first surface, and the second line-plane space angle is The surface space angle is the space angle between the line segment where the index finger joint point of the target object's hand is located and the first surface, and the third line surface space angle is the space angle between the middle finger joint point of the target object's hand. The spatial angle between the line segment and the first surface, and the fourth line-plane spatial angle is the spatial angle between the line segment where the ring finger joint point of the target object's hand is and the first surface, so The fifth line-plane space angle is the space angle between the line segment where the little finger joint point of the target object's hand is located and the first surface; the first surface is the z-axis and y-axis in the three-dimensional coordinate system The plane formed by the axis;
    根据所述第一空间角、第二空间角、第三空间角和第四空间角，以及所述第一线面空间角、第二线面空间角、第三线面空间角、第四线面空间角和第五线面空间角，调整所述目标角色的手部的大拇指关节点、食指关节点、中指关节点、无名指关节点和小指关节点的位置。According to the first space angle, the second space angle, the third space angle and the fourth space angle, as well as the first line-plane space angle, the second line-plane space angle, the third line-plane space angle and the fourth line-plane space angle. angle and the fifth line-plane space angle, adjust the positions of the thumb joint point, index finger joint point, middle finger joint point, ring finger joint point and little finger joint point of the target character's hand.
  6. 根据权利要求5所述的方法，其中，所述根据所述第一空间角、第二空间角、第三空间角和第四空间角，以及所述第一线面空间角、第二线面空间角、第三线面空间角、第四线面空间角和第五线面空间角，调整所述目标角色的手部的大拇指关节点、食指关节点、中指关节点、无名指关节点和小指关节点的位置，包括：The method according to claim 5, wherein the first space angle, the second space angle, the third space angle and the fourth space angle, and the first line-plane space angle, the second line-plane space angle angle, the third line-plane space angle, the fourth line-plane space angle and the fifth line-plane space angle, adjust the thumb joint point, index finger joint point, middle finger joint point, ring finger joint point and little finger joint of the target character's hand The location of the point, including:
    令第五空间角等于所述第一空间角，第六空间角等于所述第二空间角，第七空间角等于所述第三空间角，第八空间角等于所述第四空间角，调整所述目标角色的手部的第一大拇指关节点、第一食指关节点、第一中指关节点、第一无名指关节点和第一小指关节点的位置；Let the fifth space angle be equal to the first space angle, the sixth space angle be equal to the second space angle, the seventh space angle be equal to the third space angle, and the eighth space angle be equal to the fourth space angle, adjust The positions of the first thumb joint point, first index finger joint point, first middle finger joint point, first ring finger joint point and first little finger joint point of the target character's hand;
    令第六线面空间角等于所述第一线面空间角，第七线面空间角等于所述第二线面空间角，第八线面空间角等于所述第三线面空间角，第九线面空间角等于所述第四线面空间角，第十线面空间角等于所述第五线面空间角，调整所述目标角色的手部的除所述第一大拇指关节点外的其他大拇指关节点、除所述第一食指关节点外的其他食指关节点、除所述第一中指关节点外的其他中指关节点、除所述第一无名指关节点外的其他无名指关节点和除所述第一小指关节点外的其他小指关节点的位置；Let the sixth line-plane space angle be equal to the first line-plane space angle, the seventh line-plane space angle be equal to the second line-plane space angle, the eighth line-plane space angle be equal to the third line-plane space angle, and the ninth line-plane space angle be equal to the third line-plane space angle. The plane space angle is equal to the fourth line plane space angle, and the tenth line plane space angle is equal to the fifth line plane space angle. Adjust other points of the target character's hand except the first thumb joint point. The thumb joint point, the other index finger joint points except the first index finger joint point, the other middle finger joint points except the first middle finger joint point, the other ring finger joint points except the first ring finger joint point, and The positions of other little finger joint points except the first little finger joint point;
    其中，所述第五空间角是以所述目标角色的手部的第二手掌根节点为顶 点且以第六线段和第七线段为边的空间夹角，所述第六空间角是以所述第二手掌根节点为顶点且以所述第七线段和第八线段为边的空间夹角，所述第七空间角是以所述第二手掌根节点为顶点且以第八线段和第九线段为边的空间夹角，所述第八空间角是以所述第二手掌根节点为顶点且以第九线段和第十线段为边的空间夹角；所述第六线段为所述第二手掌根节点与相邻的所述第一大拇指关节点的线段，所述第七线段为所述第二手掌根节点与相邻的所述第一食指关节点的线段，所述第八线段为所述第二手掌根节点与相邻的所述第一中指关节点的线段，所述第九线段为所述第二手掌根节点与相邻的所述第一无名指关节点的线段，所述第十线段为所述第二手掌根节点与相邻的所述第一小指关节点的线段；Wherein, the fifth spatial angle is based on the second palm root node of the target character's hand. point and has the sixth line segment and the seventh line segment as the sides. The sixth space angle is the space angle with the second palm heel node as the vertex and the seventh line segment and the eighth line segment as the sides. angle, the seventh space angle is the space angle with the second palm heel node as the vertex and the eighth line segment and the ninth line segment as the sides, and the eighth space angle is the space angle with the second palm heel node as the vertex and the eighth line segment and the ninth line segment as the sides. is the vertex and the space angle with the ninth line segment and the tenth line segment as sides; the sixth line segment is the line segment between the second palm heel node and the adjacent first thumb joint point, and the seventh line segment is the vertex. The line segment is a line segment between the second palm heel node and the adjacent first index finger joint point, and the eighth line segment is a line segment between the second palm heel node and the adjacent first middle finger joint point, The ninth line segment is a line segment between the second palm heel node and the adjacent first ring finger joint, and the tenth line segment is a line segment between the second palm heel node and the adjacent first little finger joint. line segment of point;
    所述第六线面空间角以所述目标角色的手部的大拇指关节点所在线段与第二面的空间夹角，所述第七线面空间角是以所述目标角色的手部的食指关节点所在线段与所述第二面的空间夹角，所述第八线面空间角是以所述目标角色的手部的中指关节点所在线段与所述第二面的空间夹角，所述第九线面空间角是以所述目标角色的手部的无名指关节点所在线段与所述第二面的空间夹角，所述第十线面空间角是以所述目标角色的手部的小指关节点所在线段与所述第二面的空间夹角；所述第二面为以所述第二手掌根节点为原点建立的三维坐标系中z轴和y轴所构成的平面。The sixth line-plane space angle is the space angle between the line segment where the thumb joint point of the target character's hand is located and the second surface, and the seventh line-plane space angle is the space angle between the target character's hand and the second surface. The angle between the line segment where the joint point of the index finger is located and the second surface. The eighth line-plane space angle is the space between the line segment where the joint point of the middle finger of the hand of the target character is located and the second surface. The included angle, the ninth line-plane space angle is the space angle between the line segment where the ring finger joint point of the target character's hand is located and the second surface, and the tenth line-plane space angle is the space angle between the The angle between the line segment where the little finger joint point of the target character's hand is located and the second surface; the second surface is the z-axis and y-axis in the three-dimensional coordinate system established with the second palm root node as the origin. The plane formed.
  7. 根据权利要求1所述的方法，其中，所述在用户观看目标视频的过程中，获取用户图像之前，所述方法还包括：The method according to claim 1, wherein before the user's image is acquired while the user is watching the target video, the method further includes:
    接收第一用户在观看所述目标视频中针对第一弹幕的输入，其中，所述输入用于指示对第二用户发起观影小室创建邀请，所述第二用户为发布所述第一弹幕的用户；Receive input from the first user regarding the first bullet while watching the target video, wherein the input is used to instruct the second user to initiate an invitation to create a viewing room, and the second user is responsible for posting the first bullet. screen user;
    在所述第二用户接受所述观影小室创建邀请的情况下，创建所述第一用户与所述第二用户的观影小室，并在所述观影小室中播放所述目标视频；When the second user accepts the invitation to create a viewing room, create a viewing room for the first user and the second user, and play the target video in the viewing room;
    所述在用户观看目标视频的过程中，获取用户图像，包括：The process of obtaining the user's image while the user is watching the target video includes:
    在用户在所述观影小室中观看所述目标视频的过程中，获取用户图像。During the process of the user watching the target video in the viewing booth, the user image is acquired.
  8. 根据权利要求7所述的方法，所述方法还包括：The method of claim 7, further comprising:
    在用户选择进入演绎模式的情况下，显示视频配音界面，并在所述视频 配音界面中显示辅助频谱图，其中，所述辅助频谱图是根据所述目标视频中的原声生成的频谱图；获取用户根据所述辅助频谱图录入的配音音频；将所述配音音频融入所述目标视频中进行播放；When the user chooses to enter the interpretation mode, the video dubbing interface is displayed and the video An auxiliary spectrogram is displayed in the dubbing interface, wherein the auxiliary spectrogram is a spectrogram generated according to the original sound in the target video; the dubbing audio entered by the user according to the auxiliary spectrogram is obtained; the dubbing audio is integrated into the Play in the target video;
    和/或，and / or,
    获取用户在所述观影小室中观看所述目标视频时于第一时段内发布的弹幕和弹幕数量；确定所述第一时段内发布的各弹幕对应的情感词，并统计各情感词的数量；在第一情感词的数量占所述弹幕数量的比值大于预设阈值的情况下，在所述观影小室中添加与所述第一情感词相关的特效挂件，或者，播放与所述第一情感词相关的音效，其中，所述第一情感词为任一情感词。Obtain the barrages and the number of barrages posted by the user in the first period when watching the target video in the viewing room; determine the emotional words corresponding to each barrage posted in the first period, and count each emotion The number of words; when the ratio of the number of first emotional words to the number of barrages is greater than the preset threshold, add special effects pendants related to the first emotional words in the viewing room, or play Sound effects related to the first emotional word, wherein the first emotional word is any emotional word.
  9. 一种观影互动装置，包括：An interactive device for viewing movies, including:
    第一获取模块，用于在用户观看目标视频的过程中，获取用户图像；The first acquisition module is used to acquire the user's image while the user is watching the target video;
    识别模块，用于识别所述用户图像中的目标对象的姿态；A recognition module, used to recognize the posture of the target object in the user image;
    第一确定模块，用于基于所述目标对象的姿态，确定所述目标视频中的目标角色的回应姿态；A first determination module, configured to determine the response posture of the target character in the target video based on the posture of the target object;
    第一调整模块，用于基于所述目标对象的目标部位的姿态，对所述目标角色的目标部位的回应姿态进行调整；A first adjustment module, configured to adjust the response posture of the target part of the target character based on the posture of the target part of the target object;
    第一显示模块，用于基于调整后的所述目标角色的目标部位的回应姿态，在所述目标视频的视频画面中显示所述目标角色的回应姿态。The first display module is configured to display the response posture of the target character in the video frame of the target video based on the adjusted response posture of the target part of the target character.
  10. 一种计算机可读存储介质，用于存储计算机程序，其中，所述计算机程序被处理器执行时实现如权利要求1至8中任一项所述的观影互动方法中的步骤。 A computer-readable storage medium used to store a computer program, wherein when the computer program is executed by a processor, the steps in the interactive method for viewing movies according to any one of claims 1 to 8 are implemented.
PCT/CN2023/097743 2022-06-02 2023-06-01 Film-watching interaction method and apparatus, and computer-readable storage medium WO2023232103A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210624394.2A CN114900738B (en) 2022-06-02 2022-06-02 Video watching interaction method and device and computer readable storage medium
CN202210624394.2 2022-06-02

Publications (1)

Publication Number Publication Date
WO2023232103A1 true WO2023232103A1 (en) 2023-12-07

Family

ID=82726145

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/097743 WO2023232103A1 (en) 2022-06-02 2023-06-01 Film-watching interaction method and apparatus, and computer-readable storage medium

Country Status (2)

Country Link
CN (1) CN114900738B (en)
WO (1) WO2023232103A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114900738B (en) * 2022-06-02 2024-07-16 咪咕文化科技有限公司 Video watching interaction method and device and computer readable storage medium

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130201276A1 (en) * 2012-02-06 2013-08-08 Microsoft Corporation Integrated interactive space
US20190095393A1 (en) * 2017-03-31 2019-03-28 Nanning Fugui Precision Industrial Co., Ltd. Sharing method and device for video and audio data presented in interacting fashion
CN109543644A (en) * 2018-06-28 2019-03-29 济南大学 A kind of recognition methods of multi-modal gesture
CN110288703A (en) * 2019-06-28 2019-09-27 斑马网络技术有限公司 Image processing method, device, equipment and storage medium
CN110308792A (en) * 2019-07-01 2019-10-08 北京百度网讯科技有限公司 Control method, device, equipment and the readable storage medium storing program for executing of virtual role
CN110675474A (en) * 2019-08-16 2020-01-10 咪咕动漫有限公司 Virtual character model learning method, electronic device and readable storage medium
CN111541951A (en) * 2020-05-08 2020-08-14 腾讯科技（深圳）有限公司 Video-based interactive processing method and device, terminal and readable storage medium
CN114900738A (en) * 2022-06-02 2022-08-12 咪咕文化科技有限公司 Film viewing interaction method and device and computer readable storage medium

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108667798A (en) * 2018-03-27 2018-10-16 上海临奇智能科技有限公司 A kind of method and system of virtual viewing
CN112637692B (en) * 2019-10-09 2023-04-07 阿里巴巴集团控股有限公司 Interaction method, device and equipment
KR102387730B1 (en) * 2020-11-10 2022-04-18 주식회사 조이펀 User Information Based on Interface automatic control system Using Projection Mapping
CN112827162A (en) * 2020-12-10 2021-05-25 桑亚杰 Three-dimensional demonstration animation interactive experience method and device
CN113705520A (en) * 2021-09-03 2021-11-26 广州虎牙科技有限公司 Motion capture method and device and server

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130201276A1 (en) * 2012-02-06 2013-08-08 Microsoft Corporation Integrated interactive space
US20190095393A1 (en) * 2017-03-31 2019-03-28 Nanning Fugui Precision Industrial Co., Ltd. Sharing method and device for video and audio data presented in interacting fashion
CN109543644A (en) * 2018-06-28 2019-03-29 济南大学 A kind of recognition methods of multi-modal gesture
CN110288703A (en) * 2019-06-28 2019-09-27 斑马网络技术有限公司 Image processing method, device, equipment and storage medium
CN110308792A (en) * 2019-07-01 2019-10-08 北京百度网讯科技有限公司 Control method, device, equipment and the readable storage medium storing program for executing of virtual role
CN110675474A (en) * 2019-08-16 2020-01-10 咪咕动漫有限公司 Virtual character model learning method, electronic device and readable storage medium
CN111541951A (en) * 2020-05-08 2020-08-14 腾讯科技（深圳）有限公司 Video-based interactive processing method and device, terminal and readable storage medium
CN114900738A (en) * 2022-06-02 2022-08-12 咪咕文化科技有限公司 Film viewing interaction method and device and computer readable storage medium

Also Published As

Publication number Publication date
CN114900738B (en) 2024-07-16
CN114900738A (en) 2022-08-12

Similar Documents

Publication Publication Date Title
WO2022095467A1 (en) Display method and apparatus in augmented reality scene, device, medium and program
US20200294297A1 (en) Telepresence of Users in Interactive Virtual Spaces
US11736756B2 (en) Producing realistic body movement using body images
CN107911724B (en) Live broadcast interaction method, device and system
US9996979B2 (en) Augmented reality technology-based handheld viewing device and method thereof
TWI752502B (en) Method for realizing lens splitting effect, electronic equipment and computer readable storage medium thereof
US20090202114A1 (en) Live-Action Image Capture
JP2019536131A (en) Controls and interfaces for user interaction in virtual space
US20110154266A1 (en) Camera navigation for presentations
US20140068526A1 (en) Method and apparatus for user interaction
TWI255141B (en) Method and system for real-time interactive video
US20180197345A1 (en) Augmented reality technology-based handheld viewing device and method thereof
US20240257484A1 (en) 3d upper garment tracking
WO2023232103A1 (en) Film-watching interaction method and apparatus, and computer-readable storage medium
WO2023201937A1 (en) Human-machine interaction method and apparatus based on story scene, device, and medium
Nguyen et al. Real-time 3D human capture system for mixed-reality art and entertainment
US20240155074A1 (en) Movement Tracking for Video Communications in a Virtual Environment
CN108815845A (en) The information processing method and device of human-computer interaction, computer equipment and readable medium
WO2024031882A1 (en) Video processing method and apparatus, and computer readable storage medium
WO2023076648A1 (en) Extraction of user representation from video stream to a virtual environment
Pettersson et al. A perceptual evaluation of social interaction with emotes and real-time facial motion capture
US20240048780A1 (en) Live broadcast method, device, storage medium, electronic equipment and product
US20240013488A1 (en) Groups and Social In Artificial Reality
US20230334790A1 (en) Interactive reality computing experience using optical lenticular multi-perspective simulation
Ballin et al. Personal virtual humans—inhabiting the TalkZone and beyond

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23815280

Country of ref document: EP

Kind code of ref document: A1