WO2020107904A1 - 一种视频特效添加方法、装置、终端设备及存储介质 - Google Patents

一种视频特效添加方法、装置、终端设备及存储介质 Download PDF

Info

Publication number
WO2020107904A1
WO2020107904A1 PCT/CN2019/097094 CN2019097094W WO2020107904A1 WO 2020107904 A1 WO2020107904 A1 WO 2020107904A1 CN 2019097094 W CN2019097094 W CN 2019097094W WO 2020107904 A1 WO2020107904 A1 WO 2020107904A1
Authority
WO
WIPO (PCT)
Prior art keywords
video
image frame
condition
target human
joint point
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/097094
Other languages
English (en)
French (fr)
Inventor
唐堂
孟宇
陈曼仪
林晔
陈晔
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing ByteDance Network Technology Co Ltd
Original Assignee
Beijing ByteDance Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing ByteDance Network Technology Co Ltd filed Critical Beijing ByteDance Network Technology Co Ltd
Priority to SG11202009646XA priority Critical patent/SG11202009646XA/en
Publication of WO2020107904A1 publication Critical patent/WO2020107904A1/zh
Priority to US17/021,090 priority patent/US20210029305A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2621—Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H04N21/4312—Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/633—Control of cameras or camera modules by using electronic viewfinders for displaying additional information relating to control or operation of the camera
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • H04N5/765—Interface circuits between an apparatus for recording and another apparatus
    • H04N5/77—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • H04N5/765—Interface circuits between an apparatus for recording and another apparatus
    • H04N5/77—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera
    • H04N5/772—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera the recording apparatus and the television camera being placed in the same enclosure
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/61—Control of cameras or camera modules based on recognised objects
    • H04N23/611—Control of cameras or camera modules based on recognised objects where the recognised objects include parts of the human body

Definitions

  • Embodiments of the present disclosure relate to data technology, such as a method, device, terminal device, and storage medium for adding video special effects.
  • video interaction applications can recognize the user's face, and add a static image on the user's head (for example, add a headdress to the hair) or increase facial expression to cover the user's face.
  • This method of adding images is too limited, and at the same time the application scenario is too single to meet the diverse needs of users.
  • Embodiments of the present disclosure provide a method, device, terminal device, and storage medium for adding video special effects.
  • an embodiment of the present disclosure provides a method for adding a video special effect, the method includes: acquiring at least one image frame in a video, and identifying at least one target human joint point of a user in the image frame; When the position of the target human joint point in the image frame satisfies the joint position condition in the preset video effect conditions, use the image frame as the target image frame, and obtain at least two of the target image frame before Continuous image frames; determine the motion state of the target human joint points based on the target human joint points identified in the at least two continuous image frames; the motion state of the target human joint points meet the video special effect conditions In the case of a joint action condition that matches the joint position condition, obtain a video special effect that matches the video special effect condition; add the video to the video position associated with the target image frame in the video Video effects with matching effects.
  • an embodiment of the present disclosure also provides an apparatus for adding video special effects, including: a target human joint point recognition module configured to acquire at least one image frame in a video and identify at least one user in the image frame Target human joint point; the joint position condition judgment module is set to use the image frame as the target image when the position of the identified target human joint point in the image frame satisfies the joint position condition in the preset video effect conditions Frame and obtain at least two consecutive image frames before the target image frame; the joint point motion state detection module is set to determine the target joint point according to the target human joint point identified in the at least two continuous image frames The motion state of the target human joint point; the joint action condition judgment module is set to obtain when the motion state of the target human joint point satisfies the joint motion condition matching the joint position condition in the video special effect condition A video effect matching the video effect condition; a video effect adding module is set to add a video effect matching the video effect condition at a video position associated with the target image frame in the video.
  • an embodiment of the present disclosure also provides a terminal device including: at least one processor; a memory configured to store at least one program; where the at least one program is executed by the at least one processor, Causing the at least one processor to implement the video effect adding method as described in the embodiments of the present disclosure.
  • an embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, a method for adding a video special effect as described in an embodiment of the present disclosure is implemented.
  • FIG. 1a is a flowchart of a method for adding video special effects according to an embodiment of the present disclosure
  • FIG. 1b is a schematic diagram of a human joint point provided by an embodiment of the present disclosure.
  • FIG. 1c is a schematic diagram of a set position range provided by an embodiment of the present disclosure.
  • FIG. 1d is a schematic diagram of another setting position range provided by an embodiment of the present disclosure.
  • 2a is a flowchart of a method for adding video special effects provided by an embodiment of the present disclosure
  • 2b is a schematic diagram of a human joint point provided by an embodiment of the present disclosure.
  • 2c is a schematic diagram of another human joint point provided by an embodiment of the present disclosure.
  • FIG. 3 is a schematic structural diagram of a video effect adding device according to an embodiment of the present disclosure.
  • FIG. 4 is a schematic structural diagram of a terminal device according to an embodiment of the present disclosure.
  • FIG. 1a is a flowchart of a method for adding video special effects according to an embodiment of the present disclosure. This embodiment can be applied to the case of adding video special effects to a video.
  • the method can be performed by a video special effect adding device, which can be used It is implemented in at least one of software and hardware, and the device may be configured in a terminal device, for example, a computer or the like. As shown in FIG. 1a, the method includes steps S110 to S180.
  • step S110 at least one image frame in the video is acquired, and at least one target human joint point of the user is identified in the image frame.
  • the video is formed by a series of static image frames continuously displayed at a very fast speed.
  • the video can be split into a series of image frames, and the image frames can be edited, thereby realizing the editing operation of the video.
  • Human joint points are used to determine the user's motion state in the image frame, such as standing, bowing or jumping, and to determine the user's position information, such as the distance between the user and the terminal device, the user and the terminal device Location information such as the relative position of other objects or the user’s position in the screen shot by the terminal device.
  • the human body contour is as shown in the figure, where the circle in the human body contour represents the recognized human joint point, and the line between the two human joint points is used Represents the body parts of the human body, for example, the line between the wrist joint point and the elbow joint point is used to represent the arm between the wrist and the elbow.
  • all human body regions can be identified in the image frame. For example, it can be performed on the image frame based on the depth information contained in the image frame (depth information can be obtained through an infrared camera). Image segmentation to identify all human regions in the image frame. Select a human body area from all human body areas to identify human joint points. For example, the human body area with the shortest distance may be selected as the user who needs to identify the human joint point according to the distance between the human body area and the display screen of the terminal device. Choose other methods to determine, no restrictions on this. After the human body region is determined, human body joint points are identified on the human body region, all human body joint points belonging to the user are determined, and at least one target human body joint point can be selected from all human body joint points of the user as needed.
  • the method for identifying the joint points of the human body may be: determining the body part areas (arms, hands, thighs, and feet) belonging to the human body area in the human body area, and calculating the joint points (elbow, (Wrists, knees, etc.), and finally generate a human skeleton system according to the position of each joint point identified, and determine the target human joint point from it as needed.
  • you can use the connection between two target human joint points (such as the connection between the wrist joint point and the elbow joint point to represent the arm between the wrist and elbow), for example, through the two target human joint points
  • the coordinates determine the vector of the line segment composed of two points, and determine the user's body part area's operating state or position.
  • the above-mentioned human body recognition, body part area recognition, and joint point position calculation in the body part area can all use pre-training
  • the deep learning model is implemented, and the deep learning model can be trained according to the deep features extracted from the human body depth information.
  • step S120 in an image frame selected from at least one image frame, it is determined whether the position of the identified target human joint point in the image frame satisfies the joint position condition in the preset video effect conditions until at least one image frame If all the judgments are completed, if the position of the identified target human joint point in the image frame satisfies the joint position condition in the preset video effect conditions, step S130 is executed; when the identified target human joint point in the image frame If the position does not satisfy the joint position condition in the preset video effect conditions, step S140 is executed.
  • the position judgment of the target human joint points identified by all the image frames in the video may be performed by selecting the image frames one by one according to the playback order of the video.
  • Video special effect conditions may refer to conditions for adding video special effects, and may include joint position conditions and joint motion conditions.
  • the joint position condition may refer to the position requirement of at least one target human joint point, which is used to start adding video effects, for example, placing the left hand at the center of the camera shooting screen, or the relative position requirement of two target human joint points.
  • the left hand is placed in the area where the left eye joint point is located, and there are other position areas, which are not limited in the embodiments of the present disclosure.
  • setting the position range includes setting a plane position range or setting a spatial position range, where setting the plane position range may refer to a position range in a plane that is the same as or parallel to the video shooting plane.
  • setting the plane position range may refer to a position range in a plane that is the same as or parallel to the video shooting plane.
  • the set plane position range does not contain depth information
  • the set space position range contains depth information.
  • the three boxes are the set plane position range
  • the square body is the set space position range.
  • a joint position condition corresponds to a set position range, and at the same time corresponds to a video effect. If there are two joint position conditions, and the two target human joint points satisfy the settings corresponding to the above two joint position conditions, respectively In the case of a position range, two video effects corresponding to two joint position conditions can be added to the image frame at the same time.
  • a joint position condition can also correspond to a target human joint point, for example, the target human joint point corresponding to the foot joint position condition is an ankle joint point or not a wrist joint point, a head joint point or a shoulder joint point Wait.
  • step S130 the image frame corresponding to the target human joint point that meets the joint position condition is used as the target image frame, and at least two consecutive image frames before the target image frame are acquired, and step S150 is executed.
  • At least two consecutive image frames may refer to n consecutive image frames, where n is greater than or equal to 2, wherein consecutive may refer to the video positions of n consecutive image frames in the video being continuous, or It means that the playback sequence of n consecutive image frames in the video is continuous.
  • Acquiring at least two consecutive image frames before the target image frame may refer to acquiring consecutive n image frames before the target image frame according to the playback order (or shooting order) of the multiple image frames in the video.
  • the first two image frames, the first three image frames, or the first n image frames of the target image frame are acquired, and the image frames acquired at the same time are continuous.
  • step S140 the next image frame is acquired, and step S120 is returned to.
  • step S150 the motion state of the target human joint point is determined according to the target human joint point identified in at least two consecutive image frames.
  • At least one target human joint point is identified in at least two consecutive image frames respectively, and for the position of each target human joint point in at least two continuous image frames, the Displacement.
  • the movement direction and the moving distance of the target human joint point can be known, and the movement speed of the target human joint point can be determined according to the duration of at least two consecutive image frames, so that The information about the movement direction, movement distance and movement speed of the target human joint point determines the movement state of the target human joint point. For example, according to the positions of the wrist joint points in 30 consecutive image frames, it is determined that the wrist joint point is continuously translated to the right by 10 pixels.
  • step S160 it is determined whether the motion state of the target human joint point satisfies the joint action condition matching the joint position condition in the video special effect condition, and the motion state of the target human joint point meets the joint action match the joint position condition in the video special effect condition.
  • step S170 is executed; in the case where the motion state of the target human joint point does not satisfy the joint action condition matching the joint position condition in the video special effect condition, step S140 is executed.
  • the joint motion condition may refer to the motion of a preset joint point, and may include at least one of the direction of motion, the speed of the motion, and the distance of the motion, for example, the wrist moves downward to the set area range, and the wrist starts at 1 per frame
  • the speed of one pixel moves to the right, or multiple joint points (such as the head, shoulders, and elbows) move downward, and the movement distance of the head joint point is greater than the movement distance of the shoulder joint point, and the movement distance of the shoulder joint point
  • this embodiment of the present disclosure is not limited.
  • the video special effect conditions include multiple joint position conditions and multiple joint action conditions, and also include the correspondence between the joint position conditions and the joint action conditions.
  • the target can be determined by satisfying the joint position conditions at the target human joint point Whether the joint point of the human body meets the joint motion conditions.
  • the palm joint point of the user when it is determined that the user has a drumming action, it is necessary to first determine whether the palm joint point of the user enters the drumhead area, and after determining that the palm joint point enters the drumhead area, determine whether the palm exists in multiple consecutive image frames before the current image frame The movement state from top to bottom, if there is, it is determined that there is an action of hitting the head surface area of the palm joint point of the user, so that the music effect and animation effect corresponding to the hit of the head surface can be correspondingly added.
  • step S170 video effects matching the video effect conditions are acquired.
  • the video effects matching the video effect conditions are added starting from the current image frame that meets the video effect conditions.
  • Video effects are used to add special effects matching the user's actions in the target image frame to achieve interaction with the user.
  • it can refer to at least one of animation effects and music effects.
  • Adding animation effects is used in the process of displaying the target image frame
  • at least one of the static and dynamic images is overlaid on the original content of the target image frame, and a music special effect is added to play music simultaneously during the display in the target image frame.
  • step S180 at the video position associated with the target image frame in the video, a video effect matching the video effect conditions is added.
  • the video position is used to indicate the position of the image frame in the video. Since the image frames split from the video can be arranged according to the video playback sequence, the video position can also be used to indicate the playback time of the image frame during the video playback, which can refer to the relative start time of the video playback time.
  • a series of image frames that can be split into videos can be numbered according to the playback order. For example, the first image frame to be played is the first frame, the image frame to be played after the first image frame is the second frame, and so on, Number all image frames split in the video. For example, a video can be split into 100 frames, each image frame corresponds to a serial number, and the target image frame can be the 50th frame.
  • the video effects can be expressed in the form of codes, and the video effects are added at the video position, that is, the code fragments corresponding to the video effects are added to the code fragments corresponding to the target image frame, so that the video effects are added to the target image frame .
  • FIG. 2a is a flowchart of a method for adding video special effects according to an embodiment of the present disclosure. This embodiment is refined based on the solution in the above embodiment.
  • the at least one image frame in the acquired video is refined into: during the video recording process, at least one image frame in the video is acquired in real time.
  • add the video effects matching the joint position conditions to refine as follows: use the video position of the target image frame as the starting point for adding special effects; according to the video effects matching the video effect conditions
  • the duration of the special effect starts from the starting point of adding the special effect, and the video special effect is added to the image frame in the video that matches the duration of the special effect.
  • the method of this embodiment may include steps S210 to S290.
  • step S210 during video recording, at least one image frame in the video is acquired in real time, and at least one target human joint point of the user is identified in the image frame.
  • step S220 in the image frame selected from the at least one image frame, it is determined whether the position of the identified target human joint point in the image frame satisfies the joint position condition in the preset video effect conditions until at least one image frame All the judgments are completed.
  • step S230 is executed; when the identified target human joint point is in the image frame If the position does not satisfy the joint position condition in the preset video effect conditions, step S240 is executed.
  • step S230 the image frame corresponding to the target human joint point that meets the joint position condition is taken as the target image frame, and at least two consecutive image frames before the target image frame are acquired, and step S250 is executed.
  • determining the motion state of the target human joint point based on the target human joint points identified in at least two consecutive image frames may include: according to the video position associated with at least two continuous image frames in the video And the position of the target human joint point in at least two consecutive image frames to determine the motion state of the target human joint point.
  • the chronological sequence of at least two consecutive image frames during video playback can be determined, according to the position of the target human joint point in at least two consecutive image frames, Determine the movement direction and movement distance of the target human joint point in any two adjacent consecutive image frames, and then determine the movement direction and movement distance of the target human joint point in the video segment formed by at least two consecutive image frames, thereby determining The motion state of the target human joint.
  • determining that the motion state of the target human joint point satisfies the joint action condition matching the joint position condition in the video special effect condition may include: the direction of change of the position of the target human joint point in at least two consecutive image frames When the change direction in the joint motion condition is satisfied, it is determined that the motion state of the target human joint point satisfies the joint motion condition.
  • the direction of change of the target human joint point in at least two consecutive image frames is determined according to the position of the target human joint point in at least two continuous image frames.
  • the position may be coordinate information.
  • the position of the target human joint point in at least two consecutive image frames gradually moves from the coordinate (x 1 , y 1 ) to the coordinate (x 2 , y 2 ), and the direction of change of the position of the target human joint point is The coordinate (x 1 , y 1 ) points to the direction of the coordinate (x 2 , y 2 ).
  • the position of the target human joint point in at least two consecutive image frames changes continuously with time, but the target human joint point identified in each continuous image frame has an error, which can identify the mutation or discrete
  • the position of is used as the error culling, and the position where the target human joint point continuously changes in at least two consecutive image frames is retained as the basis for determining the direction of the position change.
  • the identified position of the target human joint point in the image frame satisfies the joint position condition in the preset video special effect conditions, including: the position of the target human joint point in the image frame matches the joint position condition If the position of the target human joint point in the previous image frame of the image frame is not within the set position range, the image frame is determined as the target image frame, and the target image frame is identified. The target human joint point meets the joint position condition.
  • the target human joint point enters the set position range as the preset joint position condition
  • the target human joint point is within the set position range in the current image frame and not in the previous image frame of the current image frame
  • determine that the target human joint point enters the set position range from outside the set position range thereby determining that the target human joint point exists for the set position range
  • determine the target human joint The point meets the preset joint position condition.
  • the position range is set to 5 dashed rectangles. The size of these 5 rectangular areas may not be the same.
  • the video effects corresponding to these 5 rectangular areas may be the same. It may not be the same.
  • the left wrist joint point in Figure 2b is outside the set position range, and the left wrist joint point in Figure 2c is within the set position range.
  • the position of the left wrist joint point in the user is determined by the position shown in Figure 2b When the position shown in FIG. 2c is reached, it is determined that the left wrist joint point of the user enters the set position range from outside the set position range, thereby determining that the left wrist joint point of the user satisfies the preset joint position condition.
  • step S240 the next image frame is acquired, and step S220 is returned to.
  • step S250 the motion state of the target human joint point is determined according to the target human joint point identified in at least two consecutive image frames.
  • step S260 it is determined whether the motion state of the target human joint point meets the joint motion condition matching the joint position condition in the video special effect condition, and the motion state of the target human joint point meets the joint motion matching the joint position condition in the video special effect condition
  • step S270 is executed; in the case where the motion state of the target human joint point does not satisfy the joint action condition matching the joint position condition in the video special effect condition, step S240 is executed.
  • step S270 video effects matching the video effect conditions are acquired.
  • step S280 the video position of the target image frame is used as the starting point of the special effect addition.
  • the starting point of the effect addition can refer to the starting position of the video effect addition.
  • step S290 the video effect is added to the image frame in the video that matches the duration of the effect, starting from the starting point of the effect addition, according to the effect duration of the video effect matching the video effect conditions.
  • the effect duration may refer to the time elapsed from the start position to the end position of the video effect.
  • the image frames that match the duration of the effect can refer to all image frames in the video from the beginning of the effect addition, that is, from the target image frame to the end image frame corresponding to the end of the video effect.
  • the video effect is a music effect. If the duration of a music effect is 3s, in this video, 30 image frames are played in 1s.
  • 90 image frames (including the target image frame) starting from the target image frame ) Is the image frame that matches the duration of the effect.
  • the embodiment of the present disclosure captures a series of image frames in real time by capturing video in real time, so as to determine in real time whether the target human joint point of the user in the captured video meets the joint position condition in the video special effect condition and matches the joint position condition Joint motion conditions, and in the case of meeting the video special effects conditions, add video special effects in real time, you can achieve the video recording while adding video special effects, improve the efficiency of adding video special effects.
  • the video effect adding method further includes: during the recording process of the video, presenting the image frames in the video in real time in the video preview interface; matching the duration of the effect in the video
  • it also includes: in the video preview interface, real-time rendering of the image frames with added special video effects.
  • the video preview interface may refer to an interface of a terminal device for users to browse videos, where the terminal device may include a server-side device or a client-side device. While shooting the video in real time, the video is displayed in the video preview interface in real time, whereby the user can browse the content of the captured video in real time.
  • While adding video effects in real time display the video effects along with the video in the video preview interface, so that users can browse to the video after adding the video effect in real time, improve the efficiency of adding video effects, and improve the user experience.
  • the video special effects include: at least one of dynamic animation special effects and music special effects; in the video preview interface, real-time rendering of image frames with added video special effects, including: in the video preview interface, real-time in video image frames Draw dynamic animation special effects and play music special effects.
  • the dynamic animation effects are drawn in the image frame displayed in real time, for example, at least one image of musical instruments, backgrounds, characters, etc. is drawn.
  • the video effects include music effects
  • the music effects are played while the image frames are displayed in real time.
  • the user selects the drumming scene and starts to record the video, and the recorded video and the added special effects are presented to the user in real time through the video preview interface.
  • the animation of the drum is rendered at the user's lower limbs in the video.
  • the animation effect of the hit head surface is rendered in the video.
  • the head surface Presents a concave shape, while playing the sound of drumming.
  • the user selects a dance pad (eg Jiugongge dance pad) scene and starts recording a video.
  • the animation effect of the dancing mat is rendered at the user's foot area in the video.
  • the animation of stepping is rendered on the middle area of the dance mat in the video, for example, the right ankle joint point falls on the middle area of the dance mat
  • the upper half of the area presents a smoke ring shape and spreads outward, while playing the sound effect corresponding to the middle grid of the dance blanket.
  • FIG. 3 is a schematic structural diagram of an apparatus for adding video special effects according to an embodiment of the present disclosure.
  • the device can be implemented in at least one of software and hardware, and the device can be configured in a terminal device. As shown in FIG. 3, the device may include: a target human joint point recognition module 310, a joint position condition judgment module 320, a joint point motion state detection module 330, a joint action condition judgment module 340, and a video effect addition module 350.
  • the target human joint point recognition module 310 is set to acquire at least one image frame in the video and identify at least one target human joint point of the user in the image frame;
  • the joint position condition judgment module 320 is set to use the image frame as the target image frame and obtain the target when the position of the identified target human joint point in the image frame satisfies the joint position condition in the preset video special effect conditions At least two consecutive image frames before the image frame;
  • the node motion state detection module 330 is configured to determine the motion state of the target human joint point based on the target human joint points identified in at least two consecutive image frames;
  • the joint action condition judgment module 340 is configured to obtain a video special effect matching the video special effect condition when the motion state of the target human joint point satisfies the joint motion condition matching the joint position condition in the video special effect condition;
  • the video effect adding module 350 is set to add a video effect matching the video effect condition at the video position associated with the target image frame in the video.
  • the video is added to match the video special effect condition Action effects, avoiding the situation that the video effects of the video interaction application are too single, realize the increase of video effects for the position of the user's joint points and the motion state of the joint points, increase the richness of the video interactive applications, and increase the flexibility of adding video effects Sex.
  • the target human joint recognition module 310 includes: a real-time image frame acquisition module, which is configured to acquire at least one image frame in the video in real time during video recording; a video special effect addition module 350, including: special effect addition
  • the starting point determination module is set to use the video position of the target image frame as the starting point for adding special effects; the video effect real-time adding module is set to be based on the duration of the video effect matching the joint position conditions, starting from the starting point of the special effect, in the video Add video effects to the image frames that match the duration of the effects.
  • the joint point motion state detection module includes: a motion state determination module set to be based on a video position associated with at least two consecutive image frames in the video, and the target human joint point is in at least two consecutive image frames The position in the position determines the motion state of the target human joint point.
  • the joint action condition determination module 340 includes a position change direction detection module set to change the direction of the position of the target human joint point in at least two consecutive image frames to meet the change direction in the joint action condition Next, it is determined that the motion state of the target human joint point satisfies the joint action condition.
  • the joint position condition determination module 320 includes a position detection module set to set the position of the target human joint point in the image frame within the set position range where the joint position condition matches, and the target human joint point is at When the position in the previous image frame of the image frame is not within the set position range, the image frame is determined as the target image frame, and at the same time, it is determined that the target human joint point identified in the target image frame satisfies the joint position condition.
  • the device for adding video effects further includes: a real-time image frame rendering module, which is set to present real-time image frames in the video in the video preview interface during video recording; a real-time video effect rendering module, which is set to In the video preview interface, image frames with added video effects are presented in real time.
  • the video effects include: at least one of dynamic animation effects and music effects;
  • the video effects real-time rendering module includes: special effects display and playback modules, which are set to be in real time in the video image frames of the video preview interface Draw dynamic animation special effects and play music special effects.
  • the video effect adding device provided by the embodiment of the present disclosure belongs to the same inventive concept as the above video effect adding method.
  • the technical details not described in detail in the embodiments of the present disclosure please refer to the above video effect adding method.
  • FIG. 4 shows a schematic structural diagram of an electronic device (eg, client or server) 400 suitable for implementing the embodiment of the present disclosure.
  • Terminal devices in the embodiments of the present disclosure may include, but are not limited to, such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (Personal Digital Assistant (PDA), tablet computers (Portable Android Device, PAD), portable multimedia players (Portable Media Player, PMP), mobile terminals such as in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital television (TV), desktop computers, and so on.
  • PDA Personal Digital Assistant
  • PMP portable multimedia players
  • mobile terminals such as in-vehicle terminals (such as in-vehicle navigation terminals)
  • fixed terminals such as digital television (TV), desktop computers, and so on.
  • the electronic device shown in FIG. 4 is only an example, and should not bring any limitation to the functions and use scope of the embodiments of the present disclosure.
  • the electronic device 400 may include a processing device (such as a central processing unit, a graphics processor, etc.) 401, which may be stored in a read-only memory (Read-Only Memory, ROM) 402 program or from a storage device 408 loads the program in the random access memory (Random Access Memory, RAM) 403 to perform various appropriate actions and processes.
  • ROM Read-Only Memory
  • RAM Random Access Memory
  • various programs and data necessary for the operation of the electronic device 400 are also stored.
  • the processing device 401, ROM 402, and RAM 403 are connected to each other via a bus 404.
  • the input/output (Input/Output, I/O) interface 405 is also connected to the bus 404.
  • the following devices can be connected to the I/O interface 405: including input devices 406 such as touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, Liquid Crystal Display (LCD) , An output device 407 of a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409.
  • the communication device 409 may allow the electronic device 400 to perform wireless or wired communication with other devices to exchange data.
  • FIG. 4 shows an electronic device 400 having various devices, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or provided instead.
  • the process described above with reference to the flowchart may be implemented as a computer software program.
  • embodiments of the present disclosure include a computer program product that includes a computer program carried on a computer-readable medium, the computer program containing program code for performing the method shown in the flowchart.
  • the computer program may be downloaded and installed from the network through the communication device 409, or from the storage device 408, or from the ROM 402.
  • the processing device 401 When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiments of the present disclosure are executed.
  • Embodiments of the present disclosure also provide a computer-readable storage medium.
  • the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two.
  • the computer-readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above.
  • Computer-readable storage media may include, but are not limited to: electrical connections with at least one wire, portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable only Read memory (Erasable Programmable Read-Only Memory, EPROM or flash memory), optical fiber, portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage device, magnetic storage device, or any suitable combination.
  • the computer-readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.
  • the computer-readable signal medium may include a data signal that is propagated in baseband or as part of a carrier wave, in which computer-readable program code is carried.
  • This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, and the computer-readable signal medium may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device .
  • the program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: electric wires, optical cables, radio frequency (RF), etc., or any suitable combination of the foregoing.
  • the computer-readable medium may be included in the above-mentioned electronic device; or it may exist alone without being assembled into the electronic device.
  • the computer-readable medium carries at least one program, and when the at least one program is executed by the electronic device, the electronic device is caused to: acquire at least one image frame in the video, and identify at least one target human joint of the user in the image frame Point; in the case where the position of the identified target human joint point in the image frame satisfies the joint position condition in the preset video effect conditions, the image frame is used as the target image frame, and at least two before the target image frame are acquired Continuous image frames; determine the motion state of the target human joint points based on the target human joint points identified in at least two continuous image frames; the joints that match the joint position conditions in the video special effect conditions at the target human joint point motion state In the case of action conditions, the video effects matching the video effect conditions are obtained; at the video position associated with the target image frame in the video, the video effects matching the video effect conditions are added.
  • the computer program code for performing the operations of the present disclosure can be written in one or more programming languages or a combination thereof.
  • the above programming languages include object-oriented programming languages such as Java, Smalltalk, C++, as well as conventional Procedural programming language-such as "C" language or similar programming language.
  • the program code may be executed entirely on the user's computer, partly on the user's computer, as an independent software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (Local Area Network, LAN) or a wide area network (Wide Area Network, WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).
  • LAN Local Area Network
  • WAN Wide Area Network
  • each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of the code contains at least one of the Executable instructions.
  • the functions noted in the block may occur out of the order noted in the figures. For example, two blocks represented in succession may actually be executed in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.
  • each block in the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts can be implemented with dedicated hardware-based systems that perform specified functions or operations Or, it can be realized by a combination of dedicated hardware and computer instructions.
  • the modules described in the embodiments of the present disclosure may be implemented in software or hardware. Among them, the name of the module does not constitute a limitation on the module itself under certain circumstances.
  • the target human joint recognition module can also be described as "acquiring at least one image frame in the video and identifying the user in the image frame A module of at least one target joint point".

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Social Psychology (AREA)
  • Human Computer Interaction (AREA)
  • Psychiatry (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Processing Or Creating Images (AREA)
  • Television Signal Processing For Recording (AREA)

Abstract

本公开公开了一种视频特效添加方法、装置、终端设备及存储介质。该方法包括：获取视频中的至少一个图像帧，并在所述图像帧中识别用户的至少一个目标人体关节点；在识别出的目标人体关节点在图像帧中的位置满足预设的视频特效条件中关节位置条件的情况下，将所述图像帧作为目标图像帧，并获取所述目标图像帧之前的至少两个连续图像帧；根据在所述至少两个连续图像帧中识别出的目标人体关节点，确定所述目标人体关节点的运动状态；在所述目标人体关节点满足关节动作条件的情况下，获取与所述视频特效条件匹配的视频特效；在所述视频中与所述目标图像帧关联的视频位置处，添加与所述视频特效条件匹配的视频特效。

Description

一种视频特效添加方法、装置、终端设备及存储介质
本公开要求在2018年11月29日提交中国专利局、申请号为201811447962.6的中国专利申请的优先权，该公开的全部内容通过引用结合在本公开中。
技术领域
本公开实施例涉及数据技术，例如一种视频特效添加方法、装置、终端设备及存储介质。
背景技术
随着通信技术和终端设备设备的发展，各种终端设备例如手机、平板电脑等已经成为了人们工作和中不可或缺的一部分，而且随着终端设备的日益普及，视频交互应用成为一种沟通和娱乐的主要渠道。
目前，视频交互应用能够识别出用户面部，并在用户头部上增加静态图像(例如在头发上增加头饰)或者增加面部表情覆盖在用户面部上。这种增加图像的方法过于局限，同时应用场景过于单一，无法满足用户的多样化需求。
发明内容
本公开实施例提供一种视频特效添加方法、装置、终端设备及存储介质。
第一方面，本公开实施例提供了一种视频特效添加方法，该方法包括：获取视频中的至少一个图像帧，并在所述图像帧中识别用户的至少一个目标人体关节点；在识别出的目标人体关节点在所述图像帧中的位置满足预设的视频特效条件中关节位置条件的情况下，将所述图像帧作为目标图像帧，并获取所述目标图像帧之前的至少两个连续图像帧；根据在所述至少两个连续图像帧中识别出的目标人体关节点，确定所述目标人体关节点的运动状态；在所述目标人体关节点的运动状态满足所述视频特效条件中与所述关节位置条件匹配的关节动作条件的情况下，获取与所述视频特效条件匹配的视频特效；在所述视频中与所述目标图像帧关联的视频位置处，添加与所述视频特效条件匹配的视频特效。
第二方面，本公开实施例还提供了一种视频特效添加装置，包括：目标人体关节点识别模块，设置为获取视频中的至少一个图像帧，并在所述图像帧中识别用户的至少一个目标人体关节点；关节位置条件判断模块，设置为在识别出的目标人体关节点在图像帧中的位置满足预设的视频特效条件中关节位置条件的情况下，将所述图像帧作为目标图像帧，并获取所述目标图像帧之前的至少两个连续图像帧；关节点运动状态检测模块，设置为根据在所述至少两个连续图像帧中识别出的所述目标人体关节点，确定所述目标人体关节点的运动状态；关节动作条件判断模块，设置为在所述目标人体关节点的运动状态满足所述视频特效条件中与所述关节位置条件匹配的关节动作条件的情况下，获取与所述视频特效条件匹配的视频特效；视频特效添加模块，设置为在所述视频中 与所述目标图像帧关联的视频位置处，添加与所述视频特效条件匹配的视频特效。
第三方面，本公开实施例还提供了一种终端设备，该终端设备包括：至少一个处理器；存储器，设置为存储至少一个程序；在所述至少一个程序被所述至少一个处理器执行，使得所述至少一个处理器实现如本公开实施例所述的视频特效添加方法。
第四方面，本公开实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，所述程序被处理器执行时实现如本公开实施例所述的视频特效添加方法。
附图说明
图1a是本公开一实施例提供的一种视频特效添加方法的流程图；
图1b是本公开一实施例提供的一种人体关节点的示意图；
图1c是本公开一实施例提供的一种设定位置范围的示意图；
图1d是本公开一实施例提供的另一种设定位置范围的示意图；
图2a是本公开一实施例提供的一种视频特效添加方法的流程图；
图2b是本公开一实施例提供的一种人体关节点的示意图；
图2c是本公开一实施例提供的另一种人体关节点的示意图；
图3是本公开一实施例提供的一种视频特效添加装置的结构示意图；
图4是本公开一实施例提供的一种终端设备的结构示意图。
具体实施方式
图1a为本公开一实施例提供的一种视频特效添加方法的流程图，本实施例可适用于在视频中添加视频特效的情况，该方法可以由视频特效添加装置来执行，该装置可以采用软件和硬件中至少之一的方式实现，该装置可以配置于终端设备中，例如典型的是计算机等。如图1a所示，该方法包括步骤S110至步骤S180。
在步骤S110中，获取视频中的至少一个图像帧，并在图像帧中识别用户的至少一个目标人体关节点。
一般来说，视频是由一系列静态的图像帧以极快的速度连续放映形成。由此，可以将视频拆分成一系列图像帧，并对图像帧进行编辑操作，从而实现对视频的编辑操作。图像帧中存在多个用户时，可以根据每个用户的关节点的识别完整度、置信度或者每个用户与拍摄视频的设备的距离，选择其中一个用户作为后续需要添加视频特效的对象。人体关节点用于确定图像帧中用户的动作状态，例如站立、鞠躬或跳跃等动作状态，以及用于确定用户的位置信息，例如用户与终端设备之间的距离、用户与终端设备所拍摄到的其他物体的相对位置或用户在终端设备所拍摄到的画面中的位置等位置信息。
在一个例子中，如图1b所示，在移动终端中，人体轮廓如图所示，其中，人体轮廓中的圆圈表示识别到的人体关节点，两个人体关节点之间的连线用于 表示人体的身体部位，例如，手腕关节点和手肘关节点之间的连线用于表示手腕和手肘之间的手臂。
对每个图像帧进行人体关节点识别操作，首先可以在图像帧中识别出所有人体区域，例如可以是根据图像帧所包含的深度信息(深度信息可以通过红外线摄像机获取)，对该图像帧进行图像分割，识别出图像帧中所有人体区域。从所有人体区域中选择一个人体区域用于识别人体关节点，例如可以是根据人体区域与终端设备显示屏幕之间的距离，选择距离最短的人体区域作为需要识别人体关节点的用户，此外还可以选择其他方式确定，对此不做限制。在确定人体区域之后，对该人体区域进行人体关节点识别，确定属于该用户的所有人体关节点，并可以根据需要从该用户的所有人体关节点中筛选出至少一个目标人体关节点。
其中，识别人体关节点的方法可以是：在人体区域中确定属于该人体区域中的身体部位区域(手臂、手、大腿和脚等)，并在每个身体部位区域计算关节点(手肘、手腕和膝盖等)位置，最后根据识别到的每个关节点位置，生成人体骨架系统，并从中根据需要确定目标人体关节点。此外，可以通过采用两个目标人体关节点的连线(如手腕关节点和手肘关节点之间的连线用于表示手腕和手肘之间的手臂)，例如通过两个目标人体关节点的坐标确定两点构成的线段的向量，判断用户某个身体部位区域的动作状态或位置，上述涉及到的人体识别、身体部位区域识别和身体部位区域中的关节点位置计算均可以采用预先训练的深度学习模型实现，而深度学习模型可以根据由人体深度信息提取出来的深度特征进行训练。
需要说明的是，识别人体关节点的方法还有其他方法，对此本公开实施例不作限制。
在步骤S120中，在从至少一个图像帧中选取的图像帧中，判断识别出的目标人体关节点在图像帧中的位置是否满足预设的视频特效条件中关节位置条件，直至至少一个图像帧全部判断完成，在识别出的目标人体关节点在图像帧中的位置满足预设的视频特效条件中关节位置条件的情况下，执行步骤S130；在识别出的目标人体关节点在图像帧中的位置不满足预设的视频特效条件中关节位置条件的情况下，执行步骤S140。
在一实施例中，对视频中所有图像帧识别到的目标人体关节点进行位置判断，可以是以按照视频的播放顺序逐一选取图像帧进行判断的方式。
视频特效条件可以是指用于添加视频特效的条件，可以包括关节位置条件和关节动作条件。
关节位置条件可以是指至少一个目标人体关节点的位置要求，用于开始添加视频特效，例如将左手放置在摄像头拍摄画面中心位置处，此外还可以是两个目标人体关节点的相对位置要求，例如将左手放置在左眼关节点所在区域，还有其他位置区域，对此本公开实施例不作限制。
其中，目标人体关节点满足预设的关节位置条件，可以是指该目标人体关节点持续位于设定位置范围内，或者是指该目标人体关节点进入或退出设定位 置范围内。在一实施例中，设定位置范围包括设定平面位置范围或者设定空间位置范围，其中，设定平面位置范围可以是指在与视频拍摄平面相同或平行的平面中的位置范围，在拍摄到的物体映射到该平面时的位置在设定位置范围内的情况下，确定该物体满足设定平面位置范围；而设定空间位置范围是指视频中所拍摄到的空间中的位置范围，在拍摄到的物体在设定空间位置范围内的情况下，确定该物体满足设定空间位置范围。也就是说设定平面位置范围不包含深度信息，而设定空间位置范围包含深度信息。在一个例子中，如图1c所示，三个方框为设定平面位置范围，如图1d所示，正方体内为设定空间位置范围。
需要说明的是，一个关节位置条件对应一个设定位置范围，同时对应一个视频特效，若存在两个关节位置条件，且在两个目标人体关节点分别满足上述两个关节位置条件对应的设定位置范围的情况下，可以同时在该图像帧中添加两个关节位置条件对应的两个视频特效。
此外，一个关节位置条件还可以对应一个目标人体关节点，例如，脚部关节位置条件对应的目标人体关节点为脚腕关节点或，而非手腕关节点、头部关节点或肩部关节点等。
在步骤S130中，将满足关节位置条件的目标人体关节点对应的图像帧作为目标图像帧，并获取目标图像帧之前的至少两个连续图像帧，执行步骤S150。
在一实施例中，至少两个连续图像帧可以是指连续的n个图像帧，n大于等于2，其中，连续可以是指n个连续图像帧在视频中的视频位置是连续的，或者是指n个连续图像帧在视频中的播放顺序是连续的。
获取目标图像帧之前的至少两个连续图像帧可以是指，根据视频中多个图像帧的播放顺序(或拍摄顺序)，获取目标图像帧之前的连续的n个图像帧。实际上是获取目标图像帧的前两个图像帧、前三个图像帧或前n个图像帧等，同时获取的图像帧是连续的。
在步骤S140中，获取下一个图像帧，返回执行步骤S120。
在步骤S150中，根据在至少两个连续图像帧中识别出的目标人体关节点，确定目标人体关节点的运动状态。
在一实施例中，分别在至少两个连续图像帧中识别至少一个目标人体关节点，针对每个目标人体关节点在至少两个连续图像帧中的位置，可以确定每个目标人体关节点的位移。在一实施例中，根据目标人体关节点的位移，可以知道目标人体关节点的运动方向和运动距离，并根据至少两个连续图像帧的持续时间，确定目标人体关节点的运动速度，从而根据目标人体关节点的运动方向、运动距离和运动速度等信息，确定目标人体关节点的运动状态。例如，根据手腕关节点分别在30个连续图像帧中的位置，确定手腕关节点是向右持续平移10个像素。
在步骤S160中，判断目标人体关节点的运动状态是否满足视频特效条件中与关节位置条件匹配的关节动作条件，在目标人体关节点的运动状态满足视频特效条件中与关节位置条件匹配的关节动作条件的情况下，执行步骤S170；在目标人体关节点的运动状态不满足视频特效条件中与关节位置条件匹配的关节 动作条件的情况下，执行步骤S140。
关节动作条件可以是指预设的关节点的动作，可以包括运动的方向、运动的速度和运动的距离等中的至少一种，例如手腕向下运动到设定区域范围，手腕以每帧1个像素的速度向右移动，或者多个关节点(如头部、肩膀和手肘)均向下移动，同时头部关节点的运动距离大于肩膀关节点的运动距离，肩膀关节点的运动距离大于手肘关节点的运动距离等，还有其他动作，对此本公开实施例不作限制。
需要说明的是，视频特效条件中包括多个关节位置条件和多个关节动作条件，同时包括关节位置条件和关节动作条件的对应关系，可以通过在目标人体关节点满足关节位置条件后，判断目标人体关节点是否满足关节动作条件。例如，确定用户有击鼓动作时，需要先确定用户的手掌关节点是否进入鼓面区域，并在确定手掌关节点进入鼓面区域之后，判断当前图像帧之前多个连续图像帧中手掌是否存在从上向下的运动状态，如果存在，则确定该用户的手掌关节点存在击打该鼓面区域的动作，从而可以对应添加该鼓面被击打对应的音乐效果和动画效果。
在步骤S170中，获取与视频特效条件匹配的视频特效。
在视频中从满足视频特效条件的当前图像帧开始增加与视频特效条件匹配的视频特效。视频特效用于在目标图像帧中添加根据用户动作匹配的特殊效果，以实现与用户交互，例如可以是指动画特效和音乐特效中至少一种，添加动画特效用于目标图像帧在显示的过程中同时绘制静态和动态图像中至少一种覆盖于目标图像帧原有内容上，添加音乐特效用于在目标图像帧中显示的过程中，同时播放音乐。
在步骤S180中，在视频中与目标图像帧关联的视频位置处，添加与视频特效条件匹配的视频特效。
视频位置用于表示图像帧在视频中的位置。由于视频拆分出的图像帧可以按照视频播放顺序进行排列，从而，视频位置还可以用于表示图像帧在视频播放过程中的播放时刻，该播放时刻可以是指相对视频播放的起始时刻的时刻。可以将视频拆分的一系列图像帧按照播放顺序进行编号，例如，第一个播放的图像帧为第1帧，在第1帧图像帧之后播放的图像帧为第2帧，以此类推，将该视频中拆分的所有图像帧进行编号。例如，视频可拆分成100帧，每个图像帧对应有一个序号，目标图像帧可以是第50帧。
确定目标图像帧的视频位置之后，在该视频位置处添加视频特效。实际上，视频特效可以以代码形式表示，在该视频位置处添加视频特效，也即在该目标图像帧对应的代码片段中添加视频特效对应的代码片段，从而实现在目标图像帧中添加视频特效。
本公开实施例通过在视频的图像帧中识别到的目标人体关节点满足视频特效条件中关节位置条件以及满足与关节位置条件匹配的关节动作条件时，为该视频添加与视频特效条件匹配的动作特效，避免了视频交互应用的视频特效过于单一的情况，实现针对用户的关节点的位置以及关节点的运动状态增加视频 特效，提高视频交互应用的丰富度，同时提高视频增加特效的灵活性。图2a为本公开一实施例提供的一种视频特效添加方法的流程图。本实施例以上述实施例中的方案为基础进行细化。在本实施例中，将获取视频中的至少一个图像帧细化为：在视频录制过程中，实时获取视频中的至少一个图像帧。同时，将在视频中与目标图像帧关联的视频位置处，添加与关节位置条件匹配的视频特效细化为：将目标图像帧的视频位置作为特效添加起点；根据与视频特效条件匹配的视频特效的特效持续时间，从特效添加起点开始，在视频中的与特效持续时间匹配的图像帧中，添加视频特效。
相应的，本实施例的方法可以包括步骤S210至步骤S290。
在步骤S210中，在视频录制过程中，实时获取视频中的至少一个图像帧，并在图像帧中识别用户的至少一个目标人体关节点。
可以实时拍摄视频，并实时获取视频中的每个图像帧。
本实施例中的视频、图像帧、目标人体关节点、视频特效条件、关节位置条件、关节动作条件、视频位置和视频特效等均可以参考上述实施例中的描述。
在步骤S220中，在从至少一个图像帧中选取的图像帧中，判断识别出的目标人体关节点在图像帧中的位置是否满足预设的视频特效条件中关节位置条件，直至至少一个图像帧全部判断完成，在识别出的目标人体关节点在图像帧中的位置满足预设的视频特效条件中关节位置条件的情况下，执行步骤S230；在识别出的目标人体关节点在图像帧中的位置不满足预设的视频特效条件中关节位置条件的情况下，执行步骤S240。
在步骤S230中，将满足关节位置条件的目标人体关节点对应的图像帧作为目标图像帧，并获取目标图像帧之前的至少两个连续图像帧，执行步骤S250。
在一实施例中，根据在至少两个连续图像帧中识别出的目标人体关节点，确定目标人体关节点的运动状态，可以包括：根据在视频中与至少两个连续图像帧关联的视频位置，以及目标人体关节点在至少两个连续图像帧中的位置，确定目标人体关节点的运动状态。
例如，根据至少两个连续图像帧在视频中关联的视频位置，可以确定至少两个连续图像帧在视频播放过程中的时间顺序，根据目标人体关节点在至少两个连续图像帧中的位置，确定任意两个相邻的连续图像帧中该目标人体关节点的运动方向和运动距离，进而确定目标人体关节点在至少两个连续图像帧形成的视频片段中的运动方向和运动距离，从而确定目标人体关节点的运动状态。
在一实施例中，确定目标人体关节点的运动状态满足视频特效条件中与关节位置条件匹配的关节动作条件，可以包括：在目标人体关节点在至少两个连续图像帧中的位置的变化方向满足关节动作条件中变化方向的情况下，确定目标人体关节点的运动状态满足关节动作条件。
例如，根据目标人体关节点在至少两个连续图像帧中位置，该位置可以是坐标信息，确定目标人体关节点在至少两个连续图像帧中的变化方向。在一个例子中，目标人体关节点在至少两个连续图像帧中的位置由坐标(x 1，y 1)逐渐移动到坐标(x 2，y 2)，目标人体关节点的位置的变化方向为坐标(x 1，y 1)指向 坐标(x 2，y 2)的方向。
此外，通常目标人体关节点在至少两个连续图像帧中的位置是随时间顺序连续变化，但在每个连续图像帧中识别到的目标人体关节点存在误差，可以将识别到的突变或离散的位置作为误差剔除，保留目标人体关节点在至少两个连续图像帧中连续变化的位置，作为确定判断位置变化方向的依据。
在一实施例中，识别出的目标人体关节点在图像帧中的位置满足预设的视频特效条件中关节位置条件，包括：在目标人体关节点在图像帧中的位置处于关节位置条件匹配的设定位置范围内，且目标人体关节点在图像帧的前一图像帧中的位置不在设定位置范围内的情况下，将图像帧确定为目标图像帧，同时确定在目标图像帧中识别出的目标人体关节点满足关节位置条件。
例如，将目标人体关节点进入设定位置范围的进入状态作为预设的关节位置条件，在目标人体关节点在当前图像帧中的设定位置范围内，且不在当前图像帧的前一个图像帧中设定位置范围内的情况下，确定目标人体关节点由设定位置范围外，进入到设定位置范围内，从而确定目标人体关节点针对设定位置范围存在进入状态，进而确定目标人体关节点满足预设的关节位置条件。在一个例子中，如图2b-图2c所示，设定位置范围为5个虚线矩形，这5个矩形区域的尺寸可以不全相同，相应的，这5个矩形区域对应的视频特效可以相同，也可以不全相同，图2b中的左手手腕关节点在设定位置范围外，图2c中的左手手腕关节点在设定位置范围内，在用户左手手腕关节点的位置由图2b所示的位置变为图2c所示的位置的情况下，确定用户左手手腕关节点从设定位置范围外进入到设定位置范围内，从而确定用户左手手腕关节点满足预设的关节位置条件。
在步骤S240中，获取下一个图像帧，返回执行步骤S220。
在步骤S250中，根据在至少两个连续图像帧中识别出的目标人体关节点，确定目标人体关节点的运动状态。
在步骤S260中，判断目标人体关节点的运动状态是否满足视频特效条件中与关节位置条件匹配的关节动作条件，在目标人体关节点的运动状态满足视频特效条件中与关节位置条件匹配的关节动作条件的情况下，执行步骤S270；在目标人体关节点的运动状态不满足视频特效条件中与关节位置条件匹配的关节动作条件的情况下，执行步骤S240。
在步骤S270中，获取与视频特效条件匹配的视频特效。
在步骤S280中，将目标图像帧的视频位置作为特效添加起点。
由于视频位置可以用于表示图像帧在视频中的位置，特效添加起点可以是指视频特效添加的起始位置。
在步骤S290中，根据与视频特效条件匹配的视频特效的特效持续时间，从特效添加起点开始，在视频中的与特效持续时间匹配的图像帧中，添加视频特效。
特效持续时间可以是指视频特效的起始位置到结束位置之间经历的时间。与特效持续时间匹配的图像帧可以是指在视频中从特效添加起点开始，也就是 从目标图像帧开始，一直到该视频特效结束时对应的结束图像帧之间的所有图像帧。例如，视频特效为音乐特效，若一个音乐特效的持续时间为3s，在该视频中，1s播放30个图像帧，按视频播放顺序，从目标图像帧开始的90个图像帧(包括目标图像帧)即为与特效持续时间匹配的图像帧。
本公开实施例通过实时拍摄视频，并实时获取视频拆分的一系列图像帧，从而实时判断拍摄的视频中用户的目标人体关节点是否满足视频特效条件中关节位置条件以及与关节位置条件匹配的关节动作条件，并在满足视频特效条件的情况下，实时添加视频特效，可以实现在视频录制的同时添加视频特效，提高视频特效的添加效率。
在上述实施例的基础上，在一实施例中，视频特效添加方法，还包括：在视频的录制过程中，在视频预览界面中实时呈现视频中的图像帧；在视频中与特效持续时间匹配的图像帧中，添加视频特效的同时，还包括：在视频预览界面中，实时呈现添加视频特效的图像帧。
其中，视频预览界面可以是指用于用户浏览视频的终端设备的界面，其中，终端设备可以包括服务器端设备或客户端设备。在实时拍摄视频的同时，将视频实时显示在视频预览界面中，由此，用户可以实时浏览到拍摄的视频的内容。
在实时添加视频特效的同时，将视频特效跟随视频一起显示在视频预览界面中，以使用户实时浏览到添加视频效果后的视频，提高视频特效添加的效率，提高用户体验。
在一实施例中视频特效包括：动态动画特效和音乐特效中至少一种；在视频预览界面中，实时呈现添加视频特效的图像帧，包括：在视频预览界面中，在视频的图像帧中实时绘制动态动画特效，并播放音乐特效。
例如，在视频特效包括动态动画特效的情况下，在实时显示的图像帧中绘制动态动画特效，例如，绘制乐器、背景和人物等中至少一种图像。在视频特效包括音乐特效的情况下，在图像帧实时显示的同时播放音乐特效。通过设置视频特效包括动态动画特效和音乐特效中至少一种，提高视频特效的多样性。
在一个例子中，用户选择击鼓场景，并开始录制视频，通过视频预览界面将录制的视频以及添加的特效实时呈现给用户。根据用户的初始动作姿态，在视频中用户下肢部位处渲染鼓的动画效果。在用户左手手掌关节点从上到下落在鼓面区域的情况下，在视频中渲染被击打的鼓面的动画效果，例如，在左手手掌关节点落在鼓面区域的情况下，鼓面呈现凹面形状，同时播放击鼓的音效。
在另一个例子中，用户选择跳舞毯(例如九宫格跳舞毯)场景，并开始录制视频。根据用户的初始动作姿态，在视频中用户的脚部区域处渲染跳舞毯的动画效果。在用户右脚脚腕关节点从上到下落在跳舞毯中间格区域的情况下，在视频中跳舞毯中间格区域上渲染踩踏的动画效果，例如，在右脚脚腕关节点落在跳舞毯中间格区域的情况下，该区域上半部呈现烟圈形状，并向外扩散，同时播放跳舞毯中间格对应的音效。图3为本公开一实施例提供的一种视频特效添加装置的结构示意图，本实施例可适用于在视频中添加视频特效的情况。该装置可以采用软件和硬件中至少之一的方式实现，该装置可以配置于终端设 备中。如图3所示，该装置可以包括：目标人体关节点识别模块310、关节位置条件判断模块320、关节点运动状态检测模块330、关节动作条件判断模块340和视频特效添加模块350。
目标人体关节点识别模块310，设置为获取视频中的至少一个图像帧，并在图像帧中识别用户的至少一个目标人体关节点；
关节位置条件判断模块320，设置为在识别出的目标人体关节点在图像帧中的位置满足预设的视频特效条件中关节位置条件的情况下，将该图像帧作为目标图像帧，并获取目标图像帧之前的至少两个连续图像帧；
关节点运动状态检测模块330，设置为根据在至少两个连续图像帧中识别出的目标人体关节点，确定目标人体关节点的运动状态；
关节动作条件判断模块340，设置为在目标人体关节点的运动状态满足视频特效条件中与关节位置条件匹配的关节动作条件的情况下，获取与视频特效条件匹配的视频特效；
视频特效添加模块350，设置为在视频中与目标图像帧关联的视频位置处，添加与视频特效条件匹配的视频特效。
本公开实施例通过在视频的图像帧中识别到的目标人体关节点满足视频特效条件中关节位置条件以及满足与关节位置条件匹配的关节动作条件的情况下，为该视频添加与视频特效条件匹配的动作特效，避免了视频交互应用的视频特效过于单一的情况，实现针对用户的关节点的位置以及关节点的运动状态增加视频特效，提高视频交互应用的丰富度，同时提高视频增加特效的灵活性。
在一实施例中，目标人体关节点识别模块310，包括：图像帧实时获取模块，设置为在视频录制过程中，实时获取视频中的至少一个图像帧；视频特效添加模块350，包括：特效添加起点确定模块，设置为将目标图像帧的视频位置作为特效添加起点；视频特效实时添加模块，设置为根据与关节位置条件匹配的视频特效的特效持续时间，从特效添加起点开始，在视频中的与特效持续时间匹配的图像帧中，添加视频特效。
在一实施例中，关节点运动状态检测模块，包括：运动状态确定模块，设置为根据在视频中与至少两个连续图像帧关联的视频位置，以及目标人体关节点在至少两个连续图像帧中的位置，确定目标人体关节点的运动状态。
在一实施例中，关节动作条件判断模块340，包括：位置变化方向检测模块，设置为在目标人体关节点在至少两个连续图像帧中的位置的变化方向满足关节动作条件中变化方向的情况下，确定目标人体关节点的运动状态满足关节动作条件。
在一实施例中，关节位置条件判断模块320，包括：位置检测模块，设置为在目标人体关节点在图像帧中的位置处于关节位置条件匹配的设定位置范围内，且目标人体关节点在该图像帧的前一图像帧中的位置不在设定位置范围内的情况下，将该图像帧确定为目标图像帧，同时确定在目标图像帧中识别出的目标人体关节点满足关节位置条件。
在一实施例中，视频特效添加装置，还包括：图像帧实时呈现模块，设置 为在视频的录制过程中，在视频预览界面中实时呈现视频中的图像帧；视频特效实时呈现模块，设置为在视频预览界面中，实时呈现添加视频特效的图像帧。
在一实施例中，视频特效包括：动态动画特效和音乐特效中至少一种；视频特效实时呈现模块，包括：特效展示和播放模块，设置为在视频预览界面中，在视频的图像帧中实时绘制动态动画特效，并播放音乐特效。
本公开实施例提供的视频特效添加装置，与上述的视频特效添加方法属于同一发明构思，未在本公开实施例中详尽描述的技术细节可参见上述视频特效添加方法。
本公开实施例提供了一种终端设备，下面参考图4，其示出了适于用来实现本公开实施例的电子设备(例如客户端或服务器端)400的结构示意图。本公开实施例中的终端设备可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、个人数字助理(Personal Digital Assistant，PDA)、平板电脑(Portable Android Device，PAD)、便携式多媒体播放器(Portable Media Player，PMP)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字电视(Television，TV)、台式计算机等等的固定终端。图4示出的电子设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图4所示，电子设备400可以包括处理装置(例如中央处理器、图形处理器等)401，其可以根据存储在只读存储器(Read-Only Memory，ROM)402中的程序或者从存储装置408加载到随机访问存储器(Random Access Memory，RAM)403中的程序而执行各种适当的动作和处理。在RAM 403中，还存储有电子设备400操作所需的各种程序和数据。处理装置401、ROM 402以及RAM403通过总线404彼此相连。输入/输出(Input/Output，I/O)接口405也连接至总线404。
通常，以下装置可以连接至I/O接口405：包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置406；包括例如液晶显示器(Liquid Crystal Display，LCD)、扬声器、振动器等的输出装置407；包括例如磁带、硬盘等的存储装置408；以及通信装置409。通信装置409可以允许电子设备400与其他设备进行无线或有线通信以交换数据。虽然图4示出了具有各种装置的电子设备400，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置409从网络上被下载和安装，或者从存储装置408被安装，或者从ROM 402被安装。在该计算机程序被处理装置401执行时，执行本公开实施例的方法中限定的上述功能。
本公开实施例还提供了一种计算机可读存储介质，计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、 或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有至少一个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(Erasable Programmable Read-Only Memory，EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(Compact Disc Read-Only Memory，CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、射频(Radio Frequency，RF)等等，或者上述的任意合适的组合。
上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。
上述计算机可读介质承载有至少一个程序，在上述至少一个程序被该电子设备执行时，使得该电子设备：获取视频中的至少一个图像帧，并在图像帧中识别用户的至少一个目标人体关节点；在识别出的目标人体关节点在图像帧中的位置满足预设的视频特效条件中关节位置条件的情况下，将该图像帧作为目标图像帧，并获取目标图像帧之前的至少两个连续图像帧；根据在至少两个连续图像帧中识别出的目标人体关节点，确定目标人体关节点的运动状态；在目标人体关节点的运动状态满足视频特效条件中与关节位置条件匹配的关节动作条件的情况下，获取与视频特效条件匹配的视频特效；在视频中与目标图像帧关联的视频位置处，添加与视频特效条件匹配的视频特效。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码，上述程序设计语言包括面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(Local Area Network，LAN)或广域网(Wide Area Network，WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、 或代码的一部分包含至少一个个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的模块可以通过软件的方式实现，也可以通过硬件的方式来实现。其中，模块的名称在某种情况下并不构成对该模块本身的限定，例如，目标人体关节点识别模块还可以被描述为“获取视频中的至少一个图像帧，并在图像帧中识别用户的至少一个目标人体关节点的模块”。

Claims (16)

  1. 一种视频特效添加方法，包括：
    获取视频中的至少一个图像帧，并在所述图像帧中识别用户的至少一个目标人体关节点；
    在识别出的目标人体关节点在所述图像帧中的位置满足预设的视频特效条件中关节位置条件的情况下，将所述图像帧作为目标图像帧，并获取所述目标图像帧之前的至少两个连续图像帧；
    根据在所述至少两个连续图像帧中识别出的所述目标人体关节点，确定所述目标人体关节点的运动状态；
    在所述目标人体关节点的运动状态满足所述视频特效条件中与所述关节位置条件匹配的关节动作条件的情况下，获取与所述视频特效条件匹配的视频特效；
    在所述视频中与所述目标图像帧关联的视频位置处，添加与所述视频特效条件匹配的视频特效。
  2. 根据权利要求1所述的方法，其中，所述获取视频中的至少一个图像帧，包括：
    在视频录制过程中，实时获取所述视频中的至少一个图像帧；
    所述在所述视频中与所述目标图像帧关联的视频位置处，添加与所述视频特效条件匹配的视频特效，包括：
    将所述目标图像帧的视频位置作为特效添加起点；
    根据与所述视频特效条件匹配的视频特效的特效持续时间，从所述特效添加起点开始，在所述视频中的与所述特效持续时间匹配的图像帧中，添加所述视频特效。
  3. 根据权利要求2所述的方法，其中，所述根据在所述至少两个连续图像帧中识别出的目标人体关节点，确定所述目标人体关节点的运动状态，包括：
    根据在所述视频中与所述至少两个连续图像帧关联的视频位置，以及所述目标人体关节点在所述至少两个连续图像帧中的位置，确定所述目标人体关节点的运动状态。
  4. 根据权利要求3所述的方法，其中，所述目标人体关节点的运动状态满足所述视频特效条件中与所述关节位置条件匹配的关节动作条件，包括：
    在所述目标人体关节点在所述至少两个连续图像帧中的位置的变化方向满足所述关节动作条件中变化方向的情况下，确定所述目标人体关节点的运动状态满足所述关节动作条件。
  5. 根据权利要求2所述的方法，其中，所述识别出的目标人体关节点在所述图像帧中的位置满足预设的视频特效条件中关节位置条件，包括：
    在所述目标人体关节点在所述图像帧中的位置处于所述关节位置条件匹配的设定位置范围内，且所述目标人体关节点在所述图像帧的前一图像帧中的位置不在所述设定位置范围内的情况下，将所述图像帧确定为目标图像帧，同时确定在所述目标图像帧中识别出的目标人体关节点满足所述关节位置条件。
  6. 根据权利要求2-5任一项所述的方法，还包括：
    在所述视频的录制过程中，在视频预览界面中实时呈现所述视频中的图像帧；
    在所述视频中的与所述特效持续时间匹配的图像帧中，添加所述视频特效的同时，还包括：
    在所述视频预览界面中，实时呈现添加所述视频特效的图像帧。
  7. 根据权利要求6所述的方法，其中，所述视频特效包括：动态动画特效和音乐特效中的至少一种；
    所述在所述视频预览界面中，实时呈现添加所述视频特效的图像帧，包括：
    在所述视频预览界面中，在所述视频的图像帧中实时绘制动态动画特效，并播放音乐特效。
  8. 一种视频特效添加装置，包括：
    目标人体关节点识别模块，设置为获取视频中的至少一个图像帧，并在所述图像帧中识别用户的至少一个目标人体关节点；
    关节位置条件判断模块，设置为在识别出的目标人体关节点在所述图像帧中的位置满足预设的视频特效条件中关节位置条件的的情况下，将所述图像帧作为目标图像帧，并获取所述目标图像帧之前的至少两个连续图像帧；
    关节点运动状态检测模块，设置为根据在所述至少两个连续图像帧中识别出的所述目标人体关节点，确定所述目标人体关节点的运动状态；
    关节动作条件判断模块，设置为在所述目标人体关节点的运动状态满足所述视频特效条件中与所述关节位置条件匹配的关节动作条件的情况下，获取与所述视频特效条件匹配的视频特效；
    视频特效添加模块，设置为在所述视频中与所述目标图像帧关联的视频位置处，添加与所述视频特效条件匹配的视频特效。
  9. 根据权利要求8所述的装置，其中，所述目标人体关节点识别模块，包括：
    图像帧实时获取模块，设置为在视频录制过程中，实时获取所述视频中的至少一个图像帧；
    所述视频特效添加模块，包括：
    特效添加起点确定模块，设置为将所述目标图像帧的视频位置作为特效添加起点；
    视频特效实时添加模块，设置为根据与所述关节位置条件匹配的视频特效的特效持续时间，从所述特效添加起点开始，在所述视频中的与所述特效持续时间匹配的图像帧中，添加所述视频特效。
  10. 根据权利要求9所述的装置，其中，所述关节点运动状态检测模块，包括：
    运动状态确定模块，设置为根据在所述视频中与所述至少两个连续图像帧关联的视频位置，以及所述目标人体关节点在所述至少两个连续图像帧中的位置，确定所述目标人体关节点的运动状态。
  11. 根据权利要求10所述的装置，其中，所述关节动作条件判断模块，包 括：
    位置变化方向检测模块，设置为在所述目标人体关节点在所述至少两个连续图像帧中的位置的变化方向满足所述关节动作条件中变化方向的情况下，确定所述目标人体关节点的运动状态满足所述关节动作条件。
  12. 根据权利要求9所述的装置，其中，所述关节位置条件判断模块，包括：
    位置检测模块，设置为在所述目标人体关节点在所述图像帧中的位置处于所述关节位置条件匹配的设定位置范围内，且所述目标人体关节点在所述图像帧的前一图像帧中的位置不在所述设定位置范围内的情况下，将所述图像帧确定为目标图像帧，同时确定在所述目标图像帧中识别出的目标人体关节点满足所述关节位置条件。
  13. 根据权利要求9-12任一项所述的装置，还包括：
    图像帧实时呈现模块，设置为在所述视频的录制过程中，在视频预览界面中实时呈现所述视频中的图像帧；
    视频特效实时呈现模块，设置为在所述视频预览界面中，实时呈现添加所述视频特效的图像帧。
  14. 根据权利要求13所述的装置，其中，所述视频特效包括：动态动画特效和音乐特效中的至少一种；
    所述视频特效实时呈现模块，包括：
    特效展示和播放模块，设置为在所述视频预览界面中，在所述视频的图像帧中实时绘制动态动画特效，并播放音乐特效。
  15. 一种终端设备，包括：
    至少一个处理器；
    存储器，设置为存储至少一个程序；
    所述至少一个程序被所述至少一个处理器执行，使得所述至少一个处理器实现如权利要求1-7任一项所述的视频特效添加方法。
  16. 一种计算机可读存储介质，其上存储有计算机程序，所述程序被处理器执行时实现如权利要求1-7任一项所述的视频特效添加方法。
PCT/CN2019/097094 2018-11-29 2019-07-22 一种视频特效添加方法、装置、终端设备及存储介质 Ceased WO2020107904A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
SG11202009646XA SG11202009646XA (en) 2018-11-29 2019-07-22 Method and apparatus for adding a video special effect, terminal device and storage medium
US17/021,090 US20210029305A1 (en) 2018-11-29 2020-09-15 Method and apparatus for adding a video special effect, terminal device and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811447962.6 2018-11-29
CN201811447962.6A CN109618183B (zh) 2018-11-29 2018-11-29 一种视频特效添加方法、装置、终端设备及存储介质

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/021,090 Continuation US20210029305A1 (en) 2018-11-29 2020-09-15 Method and apparatus for adding a video special effect, terminal device and storage medium

Publications (1)

Publication Number Publication Date
WO2020107904A1 true WO2020107904A1 (zh) 2020-06-04

Family

ID=66005512

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/097094 Ceased WO2020107904A1 (zh) 2018-11-29 2019-07-22 一种视频特效添加方法、装置、终端设备及存储介质

Country Status (4)

Country Link
US (1) US20210029305A1 (zh)
CN (1) CN109618183B (zh)
SG (1) SG11202009646XA (zh)
WO (1) WO2020107904A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111586444A (zh) * 2020-06-05 2020-08-25 广州繁星互娱信息科技有限公司 视频处理方法、装置、电子设备及存储介质
CN112333464A (zh) * 2020-10-30 2021-02-05 北京字跳网络技术有限公司 一种交互数据生成方法、装置以及计算机存储介质
CN112333473A (zh) * 2020-10-30 2021-02-05 北京字跳网络技术有限公司 一种交互方法、装置以及计算机存储介质
EP3968626A1 (en) * 2020-09-09 2022-03-16 Beijing Xiaomi Mobile Software Co., Ltd. Photography method, photography apparatus, electronic device, and storage medium
CN115278041A (zh) * 2021-04-29 2022-11-01 北京字跳网络技术有限公司 图像处理方法、装置、电子设备以及可读存储介质

Families Citing this family (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109618183B (zh) * 2018-11-29 2019-10-25 北京字节跳动网络技术有限公司 一种视频特效添加方法、装置、终端设备及存储介质
CN111507142A (zh) * 2019-01-31 2020-08-07 北京字节跳动网络技术有限公司 人脸表情图像处理方法、装置和电子设备
CN109889893A (zh) * 2019-04-16 2019-06-14 北京字节跳动网络技术有限公司 视频处理方法、装置及设备
CN109889892A (zh) * 2019-04-16 2019-06-14 北京字节跳动网络技术有限公司 视频效果添加方法、装置、设备及存储介质
CN110084204B (zh) * 2019-04-29 2020-11-24 北京字节跳动网络技术有限公司 基于目标对象姿态的图像处理方法、装置和电子设备
CN110298327B (zh) * 2019-07-03 2021-09-03 北京字节跳动网络技术有限公司 一种视觉特效处理方法及装置、存储介质与终端
US11138414B2 (en) * 2019-08-25 2021-10-05 Nec Corporation Of America System and method for processing digital images
CN112106347A (zh) * 2019-08-30 2020-12-18 深圳市大疆创新科技有限公司 一种图像生成方法、设备、可移动平台及存储介质
CN110827789B (zh) * 2019-10-12 2023-05-23 平安科技（深圳）有限公司 音乐生成方法、电子装置及计算机可读存储介质
CN111192349B (zh) * 2019-11-21 2024-02-20 深圳国匠云职业教育科技有限公司 一种发动机内燃运作的图像显示方法、系统及终端
CN113628097A (zh) * 2020-05-09 2021-11-09 北京字节跳动网络技术有限公司 图像特效配置方法、图像识别方法、装置及电子设备
CN111611941B (zh) * 2020-05-22 2023-09-19 腾讯科技（深圳）有限公司 特效处理方法及相关设备
CN112560605B (zh) * 2020-12-02 2023-04-18 北京字节跳动网络技术有限公司 交互方法、装置、终端、服务器和存储介质
CN112637665B (zh) * 2020-12-23 2022-11-04 北京市商汤科技开发有限公司 增强现实场景下的展示方法、装置、电子设备及存储介质
CN114758037B (zh) * 2020-12-28 2025-11-21 北京小米移动软件有限公司 影像生成方法、装置、终端设备及存储介质
CN112822544B (zh) * 2020-12-31 2023-10-20 广州酷狗计算机科技有限公司 视频素材文件生成方法、视频合成方法、设备及介质
CN112766214A (zh) * 2021-01-29 2021-05-07 北京字跳网络技术有限公司 一种人脸图像处理方法、装置、设备及存储介质
CN112887796B (zh) * 2021-02-10 2022-07-22 北京字跳网络技术有限公司 视频生成方法、装置、设备及介质
CN113012229A (zh) * 2021-03-26 2021-06-22 北京华捷艾米科技有限公司 人体关节点的定位方法及装置
US11961537B2 (en) 2021-04-30 2024-04-16 Lemon Inc. Content creation based on rhythm
CN113382275B (zh) 2021-06-07 2023-03-07 广州博冠信息科技有限公司 直播数据的生成方法、装置、存储介质及电子设备
CN115454313A (zh) * 2021-06-09 2022-12-09 脸萌有限公司 触碰动画显示方法、装置、设备及介质
CN115619960A (zh) * 2021-07-15 2023-01-17 北京小米移动软件有限公司 图像处理的方法、装置及电子设备
US20240324906A1 (en) * 2021-07-28 2024-10-03 Panasonic Intellectual Property Management Co., Ltd. Determination method, determination device, and determination system
CN113920167B (zh) * 2021-11-01 2025-09-05 广州博冠信息科技有限公司 一种图像处理方法、装置、存储介质及计算机系统
CN114429506B (zh) * 2022-01-28 2024-02-06 北京字跳网络技术有限公司 图像处理方法、装置、设备、存储介质和程序产品
CN114531553B (zh) 2022-02-11 2024-02-09 北京字跳网络技术有限公司 生成特效视频的方法、装置、电子设备及存储介质
CN114567805B (zh) * 2022-02-24 2024-06-14 北京字跳网络技术有限公司 确定特效视频的方法、装置、电子设备及存储介质
CN114581566A (zh) * 2022-03-10 2022-06-03 北京字跳网络技术有限公司 一种动画特效生成方法、装置、设备及介质
CN114697568B (zh) * 2022-04-07 2024-02-20 脸萌有限公司 特效视频确定方法、装置、电子设备及存储介质
CN115019228A (zh) * 2022-05-18 2022-09-06 合肥的卢深视科技有限公司 视频后期处理方法、装置、电子设备和存储介质
US12356076B2 (en) * 2022-05-24 2025-07-08 Canon Kabushiki Kaisha Image capture control device, image capture device, image capture control method, and non-transitory computer-readable storage medium
CN115766974B (zh) * 2022-10-24 2025-02-07 珠海金山数字网络科技有限公司 特效生成方法及装置
CN116489331A (zh) * 2023-04-21 2023-07-25 深圳市吉屋网络技术有限公司 一种视频特效处理方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2006117374A2 (fr) * 2005-05-03 2006-11-09 France Telecom Procédé de reconstruction tridimensionnelle d'un membre ou d'un ensemble de membres articulés
CN102855462A (zh) * 2011-07-01 2013-01-02 富士通株式会社 确定人体动作周期及识别人体动作的方法和装置
CN104623910A (zh) * 2015-01-15 2015-05-20 西安电子科技大学 舞蹈辅助特效伴侣系统及实现方法
CN108289180A (zh) * 2018-01-30 2018-07-17 广州市百果园信息技术有限公司 根据肢体动作处理视频的方法、介质和终端装置
CN109618183A (zh) * 2018-11-29 2019-04-12 北京字节跳动网络技术有限公司 一种视频特效添加方法、装置、终端设备及存储介质

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20020045123A (ko) * 2000-12-08 2002-06-19 김한주 애니메이션 교육용 촬영시스템
JP2002298557A (ja) * 2001-03-30 2002-10-11 Fantekkusu:Kk ノンリニアビデオ編集システムと記録媒体及び写真ホルダー
US9448636B2 (en) * 2012-04-18 2016-09-20 Arb Labs Inc. Identifying gestures using gesture data compressed by PCA, principal joint variable analysis, and compressed feature matrices
EP2674913B1 (en) * 2012-06-14 2014-07-23 Softkinetic Software Three-dimensional object modelling fitting & tracking.
JP2015061577A (ja) * 2013-01-18 2015-04-02 株式会社東芝 動作情報処理装置
CN203405772U (zh) * 2013-09-09 2014-01-22 北京诺亦腾科技有限公司 一种基于运动捕捉的浸入式虚拟现实系统
CN103488291B (zh) * 2013-09-09 2017-05-24 北京诺亦腾科技有限公司 一种基于运动捕捉的浸入式虚拟现实系统
CN104902212B (zh) * 2015-04-30 2019-05-10 努比亚技术有限公司 一种视频通信方法及装置
CN106250867B (zh) * 2016-08-12 2017-11-14 南京华捷艾米软件科技有限公司 一种基于深度数据的人体骨骼跟踪系统的实现方法
CN106599770A (zh) * 2016-10-20 2017-04-26 江苏清投视讯科技有限公司 一种基于体感动作识别及抠像的滑雪场景显示方法
CN107728782A (zh) * 2017-09-21 2018-02-23 广州数娱信息科技有限公司 交互方法及交互系统、服务器

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2006117374A2 (fr) * 2005-05-03 2006-11-09 France Telecom Procédé de reconstruction tridimensionnelle d'un membre ou d'un ensemble de membres articulés
CN102855462A (zh) * 2011-07-01 2013-01-02 富士通株式会社 确定人体动作周期及识别人体动作的方法和装置
CN104623910A (zh) * 2015-01-15 2015-05-20 西安电子科技大学 舞蹈辅助特效伴侣系统及实现方法
CN108289180A (zh) * 2018-01-30 2018-07-17 广州市百果园信息技术有限公司 根据肢体动作处理视频的方法、介质和终端装置
CN109618183A (zh) * 2018-11-29 2019-04-12 北京字节跳动网络技术有限公司 一种视频特效添加方法、装置、终端设备及存储介质

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111586444A (zh) * 2020-06-05 2020-08-25 广州繁星互娱信息科技有限公司 视频处理方法、装置、电子设备及存储介质
EP3968626A1 (en) * 2020-09-09 2022-03-16 Beijing Xiaomi Mobile Software Co., Ltd. Photography method, photography apparatus, electronic device, and storage medium
CN112333464A (zh) * 2020-10-30 2021-02-05 北京字跳网络技术有限公司 一种交互数据生成方法、装置以及计算机存储介质
CN112333473A (zh) * 2020-10-30 2021-02-05 北京字跳网络技术有限公司 一种交互方法、装置以及计算机存储介质
CN112333464B (zh) * 2020-10-30 2022-08-02 北京字跳网络技术有限公司 一种交互数据生成方法、装置以及计算机存储介质
CN112333473B (zh) * 2020-10-30 2022-08-23 北京字跳网络技术有限公司 一种交互方法、装置以及计算机存储介质
CN115278041A (zh) * 2021-04-29 2022-11-01 北京字跳网络技术有限公司 图像处理方法、装置、电子设备以及可读存储介质
CN115278041B (zh) * 2021-04-29 2024-02-27 北京字跳网络技术有限公司 图像处理方法、装置、电子设备以及可读存储介质

Also Published As

Publication number Publication date
CN109618183A (zh) 2019-04-12
US20210029305A1 (en) 2021-01-28
SG11202009646XA (en) 2020-10-29
CN109618183B (zh) 2019-10-25

Similar Documents

Publication Publication Date Title
WO2020107904A1 (zh) 一种视频特效添加方法、装置、终端设备及存储介质
CN109462776B (zh) 一种视频特效添加方法、装置、终端设备及存储介质
CN109525891B (zh) 多用户视频特效添加方法、装置、终端设备及存储介质
CN111857923B (zh) 特效展示方法、装置、电子设备及计算机可读介质
CN109600559B (zh) 一种视频特效添加方法、装置、终端设备及存储介质
CN109474850B (zh) 运动像素视频特效添加方法、装置、终端设备及存储介质
CN109348277B (zh) 运动像素视频特效添加方法、装置、终端设备及存储介质
WO2020186935A1 (zh) 虚拟对象的显示方法、装置、电子设备和计算机可读存储介质
CN112199016B (zh) 图像处理方法、装置、电子设备及计算机可读存储介质
EP4170589B1 (en) Music playing method and apparatus based on user interaction, and device and storage medium
WO2020207080A1 (zh) 视频拍摄方法、装置、电子设备及存储介质
WO2021197020A1 (zh) 音频处理方法、装置、可读介质及电子设备
WO2022007565A1 (zh) 增强现实的图像处理方法、装置、电子设备及存储介质
US12469233B2 (en) Video processing method and apparatus, and device, and storage medium
WO2024016924A1 (zh) 视频处理方法、装置、电子设备及存储介质
WO2023151554A1 (zh) 视频图像处理方法、装置、电子设备及存储介质
WO2023011104A1 (zh) 视频交互方法及设备
CN116782018A (zh) 视频生成方法、装置、电子设备及存储介质
CN111915744A (zh) 增强现实图像的交互方法、终端和存储介质
US12417574B2 (en) Animation processing method and apparatus, electronic device and storage medium
CN114425162B (zh) 一种视频处理方法和相关装置
CN117806448A (zh) 一种数据处理方法、装置、设备及介质
WO2022260589A1 (zh) 触碰动画显示方法、装置、设备及介质
CN114797096A (zh) 虚拟对象的控制方法、装置、设备及存储介质
WO2020147598A1 (zh) 模型动作方法、装置、带屏音箱、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19890989

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 29.09.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19890989

Country of ref document: EP

Kind code of ref document: A1