WO2020007183A1 - 视频数据处理方法、装置和存储介质 - Google Patents

视频数据处理方法、装置和存储介质 Download PDF

Info

Publication number
WO2020007183A1
WO2020007183A1 PCT/CN2019/091340 CN2019091340W WO2020007183A1 WO 2020007183 A1 WO2020007183 A1 WO 2020007183A1 CN 2019091340 W CN2019091340 W CN 2019091340W WO 2020007183 A1 WO2020007183 A1 WO 2020007183A1
Authority
WO
WIPO (PCT)
Prior art keywords
video frame
human body
current video
point set
point
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/091340
Other languages
English (en)
French (fr)
Inventor
周景锦
程培
曾毅榕
傅斌
高雨
朱晓龙
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Publication of WO2020007183A1 publication Critical patent/WO2020007183A1/zh
Priority to US16/993,061 priority Critical patent/US11461876B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/23—Recognition of whole body movements, e.g. for sport training
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20182—Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person

Definitions

  • the present application relates to the field of communication technologies, and in particular, to a video data processing method, device, and storage medium.
  • the existing manual adjustment of the human body in the video usually requires offline manual adjustment for each frame of the image, which not only has low processing efficiency, but also cannot meet real-time playback. (Such as video chat or live broadcast).
  • the embodiments of the present application provide a video data processing method, device, and storage medium; the robustness and adaptability of video data processing can be enhanced, the video data processing effect can be improved, and the display effect of the video screen can be improved.
  • An embodiment of the present application provides a video data processing method, including:
  • Obtaining a point set of a human body in a historical video frame where the historical video frame is a video frame in the video stream with the current video frame as a starting point and a past preset time range;
  • an embodiment of the present application further provides a video data processing apparatus, including:
  • a detection unit configured to collect a current video frame from a video stream, and detect positions of key points of the human body in the current video frame to obtain a point set of the human body in the current video frame;
  • An obtaining unit configured to obtain a point set of a human body in a historical video frame, where the historical video frame is a video frame in the video stream with the current video frame as a starting point and a past preset time range;
  • An analysis unit configured to analyze the jitter amplitude of a key point of the human body in the current video frame according to the point set of the human body in the current video frame and the point set of the human body in the historical video frame;
  • a determining unit configured to determine an adjusted reference position of a human body in the current video frame according to a jitter amplitude
  • the stretching unit is configured to adjust a human body part corresponding to the adjusted reference position in the current video frame, and output the adjusted current video frame.
  • An embodiment of the present application further provides an electronic device.
  • the electronic device includes:
  • an embodiment of the present application further provides a non-volatile machine-readable storage medium, where the storage medium stores machine-readable instructions, and the machine-readable instructions can be executed by a processor to perform the operations described in the embodiments of the present application. Steps in any of the provided video data processing methods.
  • the positions of the key points of the human body in the current video frame can be detected to obtain the point set of the human body in the current video frame, and then the point set of the human body in the historical video frame is obtained.
  • the jitter amplitude of the key points of the human body in the current video frame according to the point set of the human body in the current video frame and the point set of the human body in the historical video frame, and determine the adjusted reference position of the human body in the current video frame according to the jitter amplitude.
  • FIG. 1a is a schematic diagram of a scenario of a video data processing method provided by some embodiments of the present application.
  • FIG. 1b is a flowchart of a video data processing method provided by some embodiments of the present application.
  • FIG. 1c is a schematic diagram of key points of a human body in a video data processing method provided by some embodiments of the present application.
  • 1d is a flowchart of obtaining a point set of a human body in a historical video frame in some embodiments of the present application
  • 1e is a flowchart of analyzing a jitter amplitude of a key point of a human body in a current video frame in some embodiments of the present application;
  • FIG. 1f is a flowchart of smoothing a point set of a human body in a current video frame in some embodiments of the present application
  • FIG. 1g is a flowchart of determining an adjusted reference position of a human body in a current video frame according to a jitter amplitude in some embodiments of the present application;
  • FIG. 2a is another flowchart of a video data processing method provided by some embodiments of the present application.
  • 2b is a schematic diagram of detecting a position of a key point of a human body in a video data processing method provided by some embodiments of the present application;
  • FIG. 2c is a schematic diagram of a slope change in a stretching region in a method provided by some embodiments of the present application.
  • 2d is a schematic diagram of a slope change of a stretched region after smoothing in a method provided by some embodiments of the present application;
  • 2e is an example diagram before and after stretching in a video data processing method provided by some embodiments of the present application.
  • 2f is an example diagram of a user interface of a video data processing method provided by some embodiments of the present application.
  • FIG. 3a is a schematic structural diagram of a video data processing apparatus according to some embodiments of the present application.
  • FIG. 3b is another schematic structural diagram of a video data processing apparatus according to some embodiments of the present application.
  • FIG. 4 is a schematic structural diagram of a network device provided by some embodiments of the present application.
  • Some embodiments of the present application provide a video data processing method, device, and storage medium.
  • the video data processing apparatus may be specifically integrated in a network device, and the network device may be a terminal or a server.
  • the network device may collect the current video frame from the video stream, and detect the position of the key point of the human body in the current video frame to obtain the point set of the human body in the current video frame; then, obtain the video.
  • the point set of the human body in the historical video frame of the stream, and the jitter amplitude of the key points of the human body in the current video frame is analyzed according to the point set of the human body in the current video frame and the point set of the human body in the historical video frame, and according to the current video frame
  • the human body's point set and the historical video frame's human point set analyze the jitter amplitude of the key points of the human body in the current video frame.
  • a preset fixed value is used as the current video Adjust the reference position of the frame.
  • the adjusted reference position of the current video frame according to the point set of the human body in the current video frame (that is, use the flexible adjustment of the reference position). Adjust the human body part corresponding to the reference position to stretch and output the current video frame after stretching, for example, the adjusted reference position So the waist can be carried out according to the golden ratio (i.e., length of the legs is 0.618) for the current video frame portion below the waist position of the stretching frame to obtain a result, and then, outputs the resulting frame, and the like.
  • the golden ratio i.e., length of the legs is 0.618
  • FIG. 1b is a flowchart of a video data processing method provided by some embodiments of the present application.
  • the video data processing device may be specifically integrated into a network device.
  • the network device may be a terminal or a server.
  • the terminal may be Including mobile phones, tablets, laptops or personal computers (PC, Personal Computer).
  • the specific process of the video data processing method may be as follows:
  • the video image can be collected by the local camera component to obtain the current video frame; or the current video frame can also be obtained by receiving video images sent by other devices, such as other terminals or camera devices, and so on.
  • the pose estimation model is used to detect the positions of the key points of the human body in the current video frame to obtain the point set of the human body in the current video frame.
  • the pose estimation model may be established in advance according to actual application requirements, may be provided to the video data processing device after being established by other equipment, or may be established by the video data processing device itself. For example, a plurality of image samples that have been labeled with key points of the human body can be specifically collected, and then, using these image samples to train the initial pose estimation initial model, the pose estimation model can be obtained.
  • the training method can be as follows:
  • the key points of the human body refer to characteristic points that can identify key parts of the human body, and the specific points can be determined according to actual application requirements.
  • the key points of the human body may include the feature points shown in Table 1:
  • the method may further include:
  • the point set of the human body in the current video frame is stored.
  • the saved point set can be directly read. For details, see step 103.
  • the historical video frame refers to a video frame in the video stream that uses the current video frame as a starting point and a preset time range in the past. For example, if the current frame is the third frame of a video stream, the historical video frame is The frames can be the first and second frames of the video stream, and so on.
  • step 103 may also include:
  • step 1034 Determine whether the point set of the human body in the historical video frame is saved. If it is, execute step 1035 to read the point set of the human body in the saved historical video frame. If not, perform step 1036 to the history The positions of key points of the human body in the video frame are detected to obtain the point set of the human body in the historical video frame.
  • the jitter amplitude of the key points of the human body in the current video frame can be analyzed according to the point set of the human body in the current video frame and the point set of the human body in the historical video frame. For example, Referring to FIG. 1e, the details can be as follows:
  • the “right ⁇ ” human key point and the “left ⁇ ” human key point can be obtained from the point set of the human body in the current video frame to obtain the point position of the human crotch in the current video frame.
  • the “right ⁇ ” human key point and the “left ⁇ ” human key point can be obtained from the point set of the human body in the historical video frame to obtain the point position of the human crotch in the historical video frame.
  • the average of the "right ⁇ " point (point 8 in Figure 1c) and the "left ⁇ " point (point 11 in Figure 1c) in the current video frame can be calculated to obtain the waist of the human body in the current video frame.
  • step 105 it is determined whether the amplitude of the jitter is less than a preset threshold, if yes, it is determined that the current video frame is in a stable state, and then step 105 is performed; if not, it is determined that the current video frame is in an unstable state, and then step 106 is performed.
  • the preset threshold can be set according to the requirements of the actual application.
  • the waist line of the upper and lower frames can be set to be within a stable state of "10%", while exceeding "10%” is an unstable state, of course. It can also be set that when the waist line of consecutive X1 frames floats within a certain range such as "10%", it is determined as a stable state, and when the waist line of consecutive X2 frames floats outside a certain amplitude such as "10%”, then Is unstable, and so on.
  • the X1 and X2 may also be determined according to actual application requirements. For example, X1 may be 60 frames, X2 may be 100 frames, and so on, and details are not described herein.
  • the video data processing method may further include:
  • step 104 may specifically be: analyzing the jitter amplitude of the key points of the human body in the current video frame according to the smoothed point set of the human body in the current video frame and the human point set in the historical video frame.
  • the point position of the human crotch in the current video frame can be obtained from the smooth point position of the human body in the current video frame
  • the position of the human crotch in the historical video frame can be obtained from the point position of the human body in the historical video frame. The point position, and then, according to the point position of the human crotch in the current video frame and the point position of the human crotch in the historical video frame, analyze the jitter amplitude of the key point of the human body in the current video frame, and so on.
  • step 104 ' may include:
  • N frames can be selected from the historical video frames (N is a positive integer greater than 1), and then, in the time dimension, the point set of the human body in the current video frame is smoothed between the N frames,
  • the time vector of the N frames can be weighted (that is, weighted sum), as follows:
  • M frames can be selected from the historical video frames (M is a positive integer greater than 1), and then, in the confidence dimension, the point set of the human body in the current video frame is smoothed between the M frames.
  • the confidence of the M frame can be weighted (that is, the weighted sum), as follows:
  • ⁇ k score k / ⁇ score.
  • the confidence level can be obtained when detecting the position of a key point of the human body. For details, see step 102. The greater the confidence, the more accurate the detected position of the key point of the human body.
  • P i is the total smoothing result of a key point S of a human body in the point set of the current video frame (i.e. frame), and so on, to calculate the total smoothing result of all human key points in the current video frame point set.
  • the smoothed point set includes the total smoothing result of all human key points in the current video frame point set.
  • ⁇ 1 , ⁇ 2 and ⁇ 3 Are with The weight can be set according to the requirements of the actual application.
  • the adjusted reference position may be a waistline position
  • the preset fixed value may be a preset waistline value
  • the waist line position of the current video frame can be calculated according to the point set of the human body in the current video frame, which can be as follows:
  • the point position of the human crotch in the current video frame is obtained from the point position of the human body in the current video frame. Determine the waistline position of the current video frame.
  • the “right ⁇ ” point and “left ⁇ ” point of the human body in the current video frame can be obtained from the point set of the human body in the current video frame, and then the The average value is used to obtain the waist line of the human body in the current video frame.
  • the position of the waist line (such as coordinates) can be directly used as the waist line position of the current video frame, or the position of the waist line can be fine-tuned according to a preset strategy As the waistline position of the current video frame.
  • the preset strategy may be determined according to actual application requirements, and details are not described herein.
  • step 106 when the adjusted reference position is the waistline position, step 106 may be: stretching a human body part below the waistline position in the current video frame, and outputting the stretched current video frame.
  • stretching a human body part below the waistline position in the current video frame For example, it can be as follows:
  • the human body part below the waist line position in the current video frame may be stretched according to the user's preferences or settings to obtain a result frame and output the result frame.
  • the video data processing method may further include:
  • the slope of the waist position in the current video frame after stretching is smoothed to obtain a result frame.
  • the smoothing strategy may be determined according to actual application requirements. For example, a transition zone may be set at the waistline position, such as a 20% transition zone, and then the slope in the transition zone is adjusted downward. To make the stretch junction smoother and softer.
  • a waistline position can be selected as the target waistline position for stretching. For example, the highest position can be selected.
  • the target waistline position and so on. That is, the step "stretch the human body part below the waistline position in the current video frame and output the stretched current video frame" may include:
  • the human body part below the target waistline position in the current video frame may be specifically stretched according to the golden ratio, and the stretched current video frame may be output.
  • the slope of the target waistline position portion in the current video frame after stretching can be smoothed before output. That is, the step "stretch the human body part below the target waistline position in the current video frame and output the stretched current video frame" may include;
  • the degree of stretching can be determined according to the requirements of the actual application.
  • the average value of the stretching degrees of the multiple human bodies can be used for stretching.
  • the optimal stretching degree of each human body determine the optimal stretching degree of each human body, and then calculate the average value of the stretching degrees of all human bodies in the current video frame. Stretch below the target waistline position in the frame, and so on.
  • stretching the human body part below the waistline position may include only human body images below the waistline position (or target waistline position) in the video frame. Stretching the region may also include stretching all image regions below the waistline position (or target waistline position) in the video frame as a whole.
  • the human body image area can be extracted from the video frame according to the detected human key points as the foreground part, and the remaining parts after extraction are used as the background part, and then, The image area below the waistline position in the foreground part is stretched, and then the stretched foreground part and the background part are superimposed to obtain the current video frame after the stretch, which is not repeated here.
  • step 106 After performing step 106, you can return to the step of collecting the current video frame (that is, step 101) to adjust the next video frame until the end of video collection, so that the human body in the video can be adjusted in real time, such as The ability to adjust the human body in the video.
  • this embodiment can detect the positions of the key points of the human body in the current video frame, obtain the point set of the human body in the current video frame, and then obtain the human body in the historical video frame.
  • Point set and analyze the jitter amplitude of the key points of the human body in the current video frame according to the point set of the human body in the current video frame and the point set of the human body in the historical video frame, and determine the adjustment reference of the human body in the current video frame according to the jitter amplitude
  • a preset fixed value is used as the adjustment reference position of the current video frame.
  • the position of the human body in the current video frame is used.
  • the human body part corresponding to the adjusted reference position in the current video frame can be adjusted and output, thereby realizing the purpose of real-time adjustment of the human body in the current frame.
  • the scheme only adjusts the body part corresponding to the adjustment reference position, and also for video frames with small jitter amplitude
  • the fixed reference position is used.
  • the reference position is adjusted flexibly. Therefore, real-time adjustment of the human body in the video frame can be achieved, and the adjustment effect can be improved while ensuring a greater degree of assurance.
  • the natural smoothing of the background deformation in the video frame greatly improves the robustness and adaptability of the real-time adjustment function, which is conducive to improving the display effect of the video picture.
  • a video data processing method As shown in FIG. 2a, a video data processing method, the specific process can be as follows:
  • a network device collects a current video frame from a video stream.
  • a network device can collect a video image through a local camera component to obtain the current video frame; or it can also receive a current video frame by receiving video images sent by other devices, such as other terminals or camera devices, and so on.
  • Other interfaces can be set in the user interface for invoking other preset function functions to implement the functions corresponding to the function functions.
  • a "selection box" 282 is set, and when the user selects the “selection box” 282, a preset function function can be triggered, so that the "stretch” degree set by the user in the current video frame is applied to other video frames by default. (Of course, if the user needs to reset the stretching degree of the human leg in a certain frame, the default item can also be adjusted through the interface for setting the "stretching" degree in the current video frame);
  • the user can flexibly set the stretching degree of the human leg in the video frame frame by frame. For example, referring to FIG. 2f, the user can also use the "slider" in the user interface. "283 to make adjustments, and will not repeat them here.
  • the network device detects a position of a key point of the human body in the current video frame by using a pose estimation model to obtain a point set of the human body in the current video frame.
  • the pose estimation model can be established in advance according to the needs of practical applications. For example, multiple image samples that have been labeled with key points of the human body can be collected. Then, the image samples are used to train the initial pose estimation initial model.
  • the posture estimation model can be obtained. For details, refer to the foregoing embodiments, and details are not described herein.
  • the key points of the human body refer to characteristic points that can identify key parts of the human body, and the specific points can be determined according to actual application requirements.
  • it can specifically include the 22 key points of the human body as shown in Figure 1c and Table 1: nose tip, neck (Huajia point), right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right condyle, right knee , Right ankle, left condyle, left knee, left ankle, right eye, left eye, left eye, right ear, left ear, right palm, left palm, right toe and left toe, etc.
  • the network device may import the current video frame into a pose estimation model for detection.
  • the pose estimation model performs human body key point position detection on the current video frame, it may output detection
  • the positions of multiple key points of the human body are obtained, and the positions of the key points of the multiple human bodies are added to the same set to obtain the point set of the human body in the current video frame.
  • the network device may also save the point set of the human body in the current video frame. In this way, if the point set of the human body in the current video frame is subsequently needed (the current video The frame can be used as the historical video frame of subsequent video frames), and then the saved point set can be directly read. For details, see step 203.
  • the network device obtains a point set of a human body in a historical video frame of the video stream.
  • a network device may obtain a video frame in the past preset time range from the current video frame as a starting point, obtain a historical video frame, and then detect the position of a human key point in the historical video frame to obtain the historical video.
  • the point set of the human body in the frame may be obtained.
  • the network device can also directly read the point set of the human body in the historical video frame, that is, the step "Network device obtains the history"
  • the "point set of the human body in the video frame” may also include:
  • the network device obtains from the video stream the video frame starting from the current video frame and past a preset time frame to obtain a historical video frame, and determines whether the point set of the human body in the historical video frame is saved, and if so, then Read the point set of the human body in the saved historical video frame; if not, detect the position of the key point of the human body in the historical video frame to obtain the point set of the human body in the historical video frame.
  • the preset time range may be determined according to actual application requirements. For example, if the current video frame is specifically the 10th frame, and the preset time range is 5 frames, for example, then the network device may obtain the 5th time.
  • the video frames of frame 6, frame 7, frame 8 and frame 9 are used as historical video frames. Then, it is determined whether there is a point set of the human body in these historical video frames. If they exist, directly read the corresponding The point set, if it does not exist, the position of the human key point in the historical video frame is detected. For example, if the corresponding point set is not saved in the 9th frame, then the human key in the 9th frame needs to be detected at this time. The position of the point is detected to obtain the point set of the 9th frame, and so on.
  • the detection method of the position of the human key point in the historical video frame is similar to the detection method of the human key point in the current video frame, that is, the posture estimation model can be used for the human key point in the historical video frame.
  • the position of the camera is detected, and for the pose estimation model, reference may be made to the description in step 202, and details are not described herein.
  • the network device uses a preset point smoothing algorithm to smooth the point set of the human body in the current video frame according to the point set of the human body in the historical video frame to obtain a smoothed point set of the human body in the current video frame.
  • the point-smoothing algorithm can be set according to the needs of practical applications.
  • the point set of the human body in the current video frame can be smoothed from the dimensions of time, space, and confidence, as follows:
  • the network device smoothes the point set of the human body in the current video frame in the time dimension according to the point set of the human body in the historical video frame to obtain a time smoothing result.
  • the network device smoothes the point set of the human body in the current video frame in the spatial dimension according to the point set of the human body in the historical video frame to obtain a spatial smoothing result.
  • the network device performs smooth processing on the point set of the human body in the current video frame in the confidence dimension according to the point set of the human body in the historical video frame to obtain a confidence smoothing result.
  • the time smoothing result, the spatial smoothing result, and the confidence smoothing result can be weighted (that is, a weighted sum) operation to obtain the smoothness of the human body in the current video frame.
  • the point set refer to step 104'-D, which is not repeated here.
  • the network device analyzes the jitter amplitude of the key points of the human body in the current video frame according to the smoothed point set of the human body in the current video frame and the human body point set in the historical video frame. For example, referring to Table 1, it can be as follows:
  • the network device obtains the “right ⁇ ” human key point and the “left ⁇ ” human key point from the point set of the human body in the current video frame, obtains the point of the human crotch in the current video frame, and obtains the human body from the historical video frame.
  • the points are obtained from the "right ⁇ ” human key points and "left ⁇ ” human key points, and the points of the human crotch in the historical video frame are obtained.
  • the current video frame is calculated based on the points of the human crotch in the current video frame.
  • the waist line of the human body, the waist line of the human body in the historical video frame is calculated according to the points of the human crotch in the historical video frame, and the distance between the waist line of the human body in the current video frame and the waist line of the human body in the historical video frame is calculated to obtain the current The magnitude of the human body's key points in the video frame.
  • the network device determines whether the amplitude of the jitter is less than a preset threshold, and if so, determines that the current video frame is in a stable state, and then executes step 207; if not, determines that the current video frame is in an unstable state, and then executes step 208.
  • the preset threshold can be set according to actual application requirements. For example, taking the preset threshold as 10%, for example, at this time, the network device can determine whether the jitter amplitude (that is, the upper and lower frame waistlines float) is less than 10%. If yes, the current video frame is determined to be in a stable state, and step 207 is performed. If not, for example, if the upper and lower frames are floating by 20%, it may be determined that the current video frame is in an unstable state, and then step 208 is performed.
  • the jitter amplitude that is, the upper and lower frame waistlines float
  • a preset waistline value is used as the waistline position of the current video frame.
  • the preset waistline value can be set according to actual application requirements.
  • the average waistline position of previous historical video frames of the current frame can be used as the waistline value, or the current frame can also be used.
  • the waistline position of the previous frame is used as the waistline value, and so on, and is not repeated here.
  • the waist position of the current video frame is calculated according to the point set of the human body in the current video frame.
  • the point position of the human crotch in the current video frame is obtained from the point position of the human body in the current video frame. Determine the waistline position of the current video frame.
  • the “right ⁇ ” point and “left ⁇ ” point of the human body in the current video frame can be obtained from the point set of the human body in the current video frame, and then the The average value is used to obtain the waist line of the human body in the current video frame.
  • the position of the waist line (such as coordinates) can be directly used as the waist line position of the current video frame, or the position of the waist line can be fine-tuned according to a preset strategy As the waistline position of the current video frame.
  • the preset strategy may be determined according to actual application requirements, and details are not described herein.
  • the network device stretches the human body part below the waist line position in the current video frame.
  • the stretching when stretching, can be performed according to the golden ratio (that is, the leg length is 0.618), or it can be stretched according to the user's preferences or settings. For example, the portion below the waistline position can be stretched. Stretch 1.2 times or 1.5 times the original, etc.
  • the network device can select one of the waistline positions as the target waistline position for stretching.
  • the waistline position with the highest position is selected as the target waistline position, and then the human body part below the target waistline position in the current video frame is stretched.
  • the specific stretching degree can be determined according to the needs of practical applications. For example, the average value of the stretching degrees of the multiple human bodies can be used for stretching, as follows:
  • the network device stretches a human image area below a waistline position (or a target waistline position) in a video frame.
  • the human body image area can be extracted from the video frame according to the detected human key points as the foreground part, and the remaining parts after extraction are used as the background part. Then, the image area below the waist line position in the foreground part is performed. Stretch, and then overlay the stretched foreground and background parts to get the current video frame after stretching.
  • the foreground and background may not be divided, but all image regions below the waistline position (or the target waistline position) in the video frame may be stretched as a whole, and details are not described herein.
  • the network device smoothes the slope of the waist line position portion in the current video frame after stretching to obtain a result frame, and then executes step 211.
  • the smoothing strategy can be determined according to actual application requirements. For example, a transition zone can be set at the waistline position, and then the slope in the transition zone can be adjusted to make the stretching junction smoother. And soft.
  • the network device outputs a result frame.
  • the network device may display the result frame on the terminal interface.
  • the video frame after the leg is stretched may be displayed, thereby realizing the function of long legs in real time when recording a video.
  • the user can flexibly adjust the stretching degree of the human leg in the current video frame by adjusting the "slider" 283 on the interface.
  • the user can also view the contrast effect before and after stretching through preset trigger keys, such as by pressing the video screen.
  • FIG. 2e this figure is a comparison diagram of the current video frame before and after stretching, where the left picture is the current video frame before the stretch, and the right picture is the current video frame after the stretch. It can be seen that after stretching the human body part below the waistline position, the leg length of the human body in the left picture is significantly larger than that of the human body in the right picture.
  • step 210 the step of collecting the current video frame (that is, step 201) may be returned to stretch the next video frame until the end of video collection, so that the The human body is stretched in real time, such as the function of long legs in the video.
  • this embodiment can detect the positions of the key points of the human body in the current video frame, obtain the point set of the human body in the current video frame, and then obtain the human body in the historical video frame.
  • Point set according to the point set of the human body in the historical video frame, smooth the point set of the human body in the current video frame, and according to the smoothed point set of the human body in the current video frame and the human body in the historical video frame.
  • the point set analyzes the jitter amplitude of the key points of the human body in the current video frame.
  • a preset fixed waistline value is used as the waistline position of the current video frame.
  • the waist line position of the current video frame is calculated according to the point set of the human body in the current video frame. Thereafter, the human body part below the waist line position in the current video frame is stretched and output, thereby realizing the human body in the current frame.
  • the Fixed waistline only flexible waistline is used for video frames with large amplitude of jitter. Therefore, real-time long legs of the human body in the video frame can be realized to improve the effect of long legs while ensuring the background in the video frame to a greater extent.
  • the natural smoothness of the deformation greatly improves the robustness and adaptability of the real-time long-leg function, which is conducive to improving the display effect of the video picture.
  • the solution can smooth the point set after obtaining the point set of the human body in the current video frame. Therefore, the point difference between the upper and lower frames due to the violent movement of the human body can be greatly reduced. Big problems can further improve the processing effect of video pictures.
  • An embodiment of the present application may also provide a video data processing apparatus.
  • the video data processing apparatus may be specifically integrated in a network device, and the network device may be a device such as a terminal or a server.
  • the video data processing apparatus may include a detection unit 301, an acquisition unit 302, an analysis unit 303, a determination unit 304, and a stretching unit 305, as follows:
  • the detecting unit 301 is configured to collect a current video frame and detect positions of key points of a human body in the current video frame to obtain a point set of a human body in the current video frame.
  • the detection unit 301 may be specifically configured to detect a position of a key point of a human body in the current video frame by using a pose estimation model to obtain a point set of the human body in the current video frame.
  • the pose estimation model can be established in advance according to the requirements of the actual application. It can be provided to the video data processing device after being established by other equipment, or it can be established by the video data processing device itself. That is, the video data processing device can also include The establishing unit can be specifically used to collect multiple image samples that have been labeled with key points of the human body, and then use these image samples to train the initial pose estimation initial model to obtain the pose estimation model. For details of the training method, refer to the foregoing method embodiments, and details are not described herein.
  • the key points of the human body refer to characteristic points that can identify key parts of the human body.
  • the specific points can be determined according to the needs of the actual application. For example, the specific key points of the human body can be shown in Table 1. For details, see the previous implementation. Examples are not repeated here.
  • the point set of the human body in the current video frame can also be saved.
  • the current video frame can be As a historical video frame of subsequent video frames
  • the saved point set can be directly read; that is, as shown in FIG. 3b, the video data processing apparatus may further include a saving unit 306, as follows:
  • the saving unit 306 is configured to save a point set of a human body in the current video frame.
  • the obtaining unit 302 is configured to obtain a point set of a human body in a historical video frame.
  • the historical video frame refers to a video frame in the video stream with the current video frame as a starting point and a preset time range in the past.
  • the obtaining unit 302 may be specifically configured to obtain a video frame in the past preset time range from the current video frame as a starting point to obtain a historical video frame, and then detect positions of key points of the human body in the historical video frame. The point set of the human body in the historical video frame is obtained.
  • the point set of the human body in the historical video frame has been saved before then, at this time, the point set of the human body in the historical video frame can be directly read, that is:
  • the obtaining unit 302 may be specifically configured to obtain the current video frame as a starting point and a video frame within a preset time range in the past to obtain a historical video frame, and determine whether a point set of a human body in the historical video frame is saved. If yes, Then the point set of the human body in the historical video frame is read; if not, the position of the key point of the human body in the historical video frame is detected to obtain the point set of the human body in the historical video frame.
  • the analysis unit 303 is configured to analyze a jitter amplitude of a key point of the human body in the current video frame according to the point set of the human body in the current video frame and a point set of the human body in the historical video frame.
  • the analysis unit 303 may include a point acquisition subunit 3031 and an analysis subunit 3032, as follows:
  • the point obtaining subunit 3031 is configured to obtain the position of the human crotch in the current video frame from the point set of the human body in the current video frame, and acquire the historical video frame from the point set of the human body in the historical video frame. The point of the human crotch.
  • the analysis sub-unit 3032 is configured to analyze the jitter amplitude of the key points of the human body in the current video frame according to the points of the human crotch in the current video frame and the points of the human crotch in the historical video frame.
  • the analysis subunit 3032 may be specifically configured to calculate the waist line of the human body in the current video frame according to the points of the human crotch in the current video frame; calculate the human body in the historical video frame according to the points of the human crotch in the historical video frame. Calculate the distance between the waist line of the human body in the current video frame and the waist line of the human body in the historical video frame to obtain the jitter amplitude of the key points of the human body in the current video frame.
  • the analysis subunit 3032 can be specifically used to calculate the average value of the "right ⁇ " and "left ⁇ ” points in the current video frame to obtain the waistline of the human body in the current video frame; and calculate the "right” in the historical video frame.
  • the determining unit 304 is configured to determine a waist line position of a human body in the current video frame according to a shaking amplitude.
  • the determining unit 304 may be specifically configured to determine that the current video frame is in a stable state according to the jitter amplitude, and use a preset waistline value as the waistline position of the current video frame; determine that the current video frame is in a non- In the steady state, the waist line position of the current video frame is calculated according to the point set of the human body in the current video frame.
  • the determining unit 304 may be specifically configured to use a preset waistline value as the waistline position of the current video frame when determining that the shake amplitude is less than a preset threshold; when determining that the shake amplitude is greater than or equal to the preset threshold, use the current
  • the set of points of the human body in the video frame calculates the waistline position of the current video frame.
  • the determining unit 304 may be specifically configured to determine the position of the human body in the current video frame when determining that the current video frame is in an unstable state according to the shake amplitude (for example, when determining that the shake amplitude is greater than or equal to a preset threshold).
  • the points of the human crotch in the current video frame are collectively obtained, the waist line of the human body in the current video frame is calculated according to the points of the human crotch in the current video frame, and the waist line position of the current video frame is determined according to the calculated waist line.
  • the preset threshold can be set according to the requirements of the actual application.
  • the waist line of the upper and lower frames can be set to be within a stable state of "10%", while exceeding "10%” is an unstable state, of course. It can also be set that when the waist line of consecutive X1 frames floats within a certain range such as "10%", it is determined as a stable state, and when the waist line of consecutive X2 frames floats outside a certain amplitude such as "10%”, then Is unstable, and so on.
  • the X1 and X2 may also be determined according to actual application requirements. For example, X1 may be 60 frames, X2 may be 100 frames, and so on, and details are not described herein.
  • the stretching unit 305 is configured to stretch a human body part below the waistline position in the current video frame, and output the stretched current video frame.
  • the stretching unit 305 may be specifically configured to stretch the portion below the waistline position in the current video frame according to the golden ratio (that is, the leg length is 0.618) to obtain a result frame, and output the result frame.
  • the golden ratio that is, the leg length is 0.618
  • the stretching unit 305 can be specifically used to stretch the human body part below the waist line position in the current video frame according to the golden ratio, smooth the slope of the waist line position part in the current video frame after stretching, obtain the result frame, and output The result frame.
  • the smoothing strategy may be determined according to actual application requirements. For example, a transition zone may be set at the waistline position, such as a 20% transition zone, and then the slope in the transition zone is adjusted downward. To make the stretch junction smoother and softer.
  • a waistline position can be selected as the target waistline position for stretching. For example, the highest position can be selected. As the target waistline position, and so on. which is:
  • the stretching unit 305 can be specifically used to select the highest waistline position from the multiple waistline positions of the current video frame as the target waistline position, and stretch the human body part below the target waistline position in the current video frame. And output.
  • the stretching unit 305 may be specifically configured to stretch the portion below the target waistline position in the current video frame according to the golden ratio, and smooth the slope of the target waistline position portion in the current video frame after stretching. Get the result frame and output the result frame.
  • the stretching unit 305 can return to trigger the detection unit 301 to perform the step of collecting the current video frame to stretch the next video frame until the end of the video collection.
  • the human body in the video can be stretched in real time. For example, perform the function of long legs on the human body in the video.
  • the video data processing device may further include a smoothing unit 307, as follows:
  • the smoothing unit 307 is configured to use a preset point smoothing algorithm to smooth the point set of the human body in the current video frame according to the point set of the human body in the historical video frame to obtain the smoothed body of the human body in the current video frame. Point set.
  • the analysis unit is specifically configured to analyze the jitter amplitude of the key points of the human body in the current video frame according to the smoothed point set of the human body in the current video frame and the human body point set in the historical video frame.
  • the above units may be implemented as independent entities, or may be arbitrarily combined, and implemented as the same or several entities.
  • the above units refer to the foregoing method embodiments, and details are not described herein again.
  • the video data processing device of this embodiment can detect the position of the key points of the human body in the current video frame by the detection unit 301 to obtain the point set of the human body in the current video frame. Then, the point set of the human body in the historical video frame is obtained by the obtaining unit 302, and the key point of the human body in the current video frame is analyzed by the analysis unit 303 according to the point set of the human body in the current video frame and the point set of the human body in the historical video frame.
  • the determining unit 304 determines the waistline position of the human body in the current video frame according to the shaking amplitude.
  • the determining unit 304 uses a preset fixed waistline value as the The waistline position of the current video frame. Conversely, in an unstable state, the determination unit 304 calculates the waistline position of the current video frame according to the point set of the human body in the current video frame, and so on.
  • the stretching unit 305 stretch and output the human body part below the waist line position in the current video frame, and then realize the real-time long-legged goal of the human body in the current frame ; Because this solution only stretches the part below the waistline, and for video frames with small jitter amplitude, a fixed waistline is used, and for video frames with large jitter amplitude, a flexible waistline is used. It can realize real-time long legs of the human body in the video frame and improve the effect of long legs. At the same time, it can ensure the natural smoothness of the background deformation in the video frame. Improve the display of video images.
  • the embodiment of the present application further provides a network device, which may specifically be a terminal or a server.
  • the network device may integrate any video data processing device provided by the embodiment of the present application.
  • FIG. 4 shows a schematic structural diagram of a network device involved in an embodiment of the present application. Specifically,
  • the network device may include components such as a processor 401 of one or more processing cores, a memory 402 of one or more computer-readable storage media, a power source 403, and an input unit 404.
  • a processor 401 of one or more processing cores a memory 402 of one or more computer-readable storage media
  • a power source 403 a power source for converting signals to electrical signals.
  • the processor 401 is the control center of the network device, and uses various interfaces and lines to connect various parts of the entire network device.
  • the processor 401 runs or executes software programs and / or modules stored in the memory 402 and calls the stored programs in the memory 402. Data, perform various functions of network equipment and process data, so as to monitor the overall network equipment.
  • the processor 401 may include one or more processing cores; the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and an application program, and the modem processor mainly Handle wireless communication. It can be understood that the foregoing modem processor may not be integrated into the processor 401.
  • the memory 402 may be used to store software programs and modules.
  • the processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402.
  • the memory 402 may mainly include a storage program area and a storage data area, where the storage program area may store an operating system, at least one application required by a function (such as a sound playback function, an image playback function, etc.), etc .; Use of data created by network devices, etc.
  • the memory 402 may include a high-speed random access memory, and may further include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may further include a memory controller to provide the processor 401 access to the memory 402.
  • the network device further includes a power source 403 for supplying power to various components.
  • the power source 403 can be logically connected to the processor 401 through the power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system.
  • the power source 403 may further include any component such as one or more DC or AC power sources, a recharging system, a power failure detection circuit, a power converter or inverter, and a power status indicator.
  • the network device may further include an input unit 404.
  • the input unit 404 may be used to receive inputted numeric or character information, and generate a keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
  • the network device may further include a display unit and the like, and details are not described herein again.
  • the processor 401 in the network device loads the executable files corresponding to one or more application processes into the memory 402 according to the following instructions, and the processor 401 runs the stored in An application program in the memory 402, thereby implementing any data processing method in the embodiments of the present application. :
  • an embodiment of the present application provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute steps in any one of the video data processing methods provided in the embodiments of the present application.
  • the storage medium may include a read-only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disk.
  • ROM read-only memory
  • RAM random access memory
  • magnetic disk or an optical disk.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Human Computer Interaction (AREA)
  • Health & Medical Sciences (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • General Health & Medical Sciences (AREA)
  • Image Analysis (AREA)
  • Television Signal Processing For Recording (AREA)

Abstract

本申请实施例公开了一种视频数据处理方法、装置和存储介质；本申请实施例在从视频流中采集到当前视频帧后，可以对当前视频帧中的人体关键点的位置进行检测，得到当前视频帧中人体的点位集，然后，获取历史视频帧中人体的点位集，并根据当前视频帧中人体的点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度，根据抖动幅度确定所述当前视频帧中人体的调整参考位置，此后，对当前视频帧中调整参考位置对应的人体部分进行调整并输出。

Description

视频数据处理方法、装置和存储介质
本申请要求于2018年7月4日提交中国专利局、申请号为2018107218570，发明名称为“一种视频数据处理方法、装置和存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及通信技术领域，具体涉及一种视频数据处理方法、装置和存储介质。
背景技术
随着网络技术的发展和移动终端的普及，视频拍摄已渐渐成为很多用户进行社交分享和沟通交流的一种方式，与此同时，也催生了许多用于美化视频画面的技术，比如对视频中的人脸进行美白、磨皮或瘦脸等。当然，除了人脸之外，也可以对视频中的人体进行美化，比如对腿部进行拉伸(简称长腿)，等等。
由于在视频中，人体姿势的变化幅度一般较大，因此，现有在对视频中人体进行调整时，通常需要针对每一帧图像进行离线人工调整，不仅处理效率较低，而且无法满足实时播放(比如视频聊天或直播)的需求。
技术内容
本申请实施例提供一种视频数据处理方法、装置和存储介质；可以增强视频数据处理的鲁棒性和适应性，改善视频数据处理效果，进而提高视频画面的显示效果。
本申请实施例提供一种视频数据处理方法，包括：
从视频流中采集当前视频帧，并对所述当前视频帧中的人体关键点的位置进行检测，得到当前视频帧中人体的点位集；
获取历史视频帧中人体的点位集，所述历史视频帧为在所述视频流中，以所述当前视频帧为起点，过去预设时间范围内的视频帧；
根据所述当前视频帧中人体的点位集和历史视频帧中人体的点位集，分析当前视频帧中人体关键点的抖动幅度；
根据抖动幅度确定所述当前视频帧中人体的调整参考位置；
对当前视频帧中调整参考位置对应的人体部分进行调整，并输出调整后的当前视频帧。
相应的，本申请实施例还提供一种视频数据处理装置，包括：
检测单元，用于从视频流中采集当前视频帧，并对所述当前视频帧中的人体关键点的位置进行检测，得到当前视频帧中人体的点位集；
获取单元，用于获取历史视频帧中人体的点位集，所述历史视频帧为在所述视频流中，以所述当前视频帧为起点，过去预设时间范围内的视频帧；
分析单元，用于根据所述当前视频帧中人体的点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度；
确定单元，用于根据抖动幅度确定所述当前视频帧中人体的调整参考位置；
拉伸单元，用于对当前视频帧中调整参考位置对应的人体部分进行调整，并输出调整后的当前视频帧。
本申请实施例还提供了一种电子设备，所述电子设备包括：
处理器；
与所述处理器相连接的存储器；所述存储器中存储有机器可读指令， 所述机器可读指令可以由处理器执行，以执行本申请实施例所提供的任一种视频数据处理方法中的步骤。
此外，本申请实施例还提供一种非易失性机器可读存储介质，所述存储介质存储有机器可读指令，所述机器可读指令可以由处理器执行，以执行本申请实施例所提供的任一种视频数据处理方法中的步骤。
本申请实施例在采集到当前视频帧后，可以对当前视频帧中的人体关键点的位置进行检测，得到当前视频帧中人体的点位集，然后，获取历史视频帧中人体的点位集，并根据当前视频帧中人体的点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度，根据抖动幅度确定当前视频帧中人体的调整参考位置，此后，对当前视频帧中调整参考位置对应的人体部分进行调整并输出，进而实现对当前帧中人体进行实时调整的目的；由于该方案只对调整参考位置对应的人体部分进行调整，而且可以根据抖动幅度灵活地确定调整参考位置，因此，可以在实现对视频帧中人体进行实时调整，改善调整效果的同时，较大程度保证视频帧中背景形变的自然平滑，大大提高了实时调整功能的鲁棒性和适应性，有利于提高视频画面的显示效果。
附图简要说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1a是本申请一些实施例提供的视频数据处理方法的场景示意图；
图1b是本申请一些实施例提供的视频数据处理方法的流程图；
图1c是本申请一些实施例提供的视频数据处理方法中人体关键点 的示意图；
图1d是本申请一些实施例中获取历史视频帧中人体的点位集的流程图；
图1e是本申请一些实施例中分析当前视频帧中人体关键点的抖动幅度的流程图；
图1f是本申请一些实施例中对当前视频帧中人体的点位集进行平滑处理的流程图；
图1g是本申请一些实施例中根据抖动幅度确定当前视频帧中人体的调整参考位置的流程图；
图2a是本申请一些实施例提供的视频数据处理方法的另一流程图；
图2b是本申请一些实施例提供的视频数据处理方法中人体关键点位置检测的示意图；
图2c是本申请一些实施例提供的方法中拉伸区域斜率变化示意图；
图2d是本申请一些实施例提供的方法中平滑后的拉伸区域斜率变化示意图；
图2e是本申请一些实施例提供的视频数据处理方法中拉伸前后的示例图；
图2f是本申请一些实施例提供的视频数据处理方法的用户界面示例图；
图3a是本申请一些实施例提供的视频数据处理装置的结构示意图；
图3b是本申请一些实施例提供的视频数据处理装置的另一结构示意图；
图4是本申请一些实施例提供的网络设备的结构示意图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面结合附图对本申请作进一步的详细阐述。
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
在一些对视频中人体进行调整的方案中，获取到视频数据后，对视频数据中的每一帧画面进行整体调整。比如，在“实时长腿”的方案中，在获取到视频数据后，对视频数据中的每一帧画面进行整体纵向拉伸。虽然对每一帧画面进行整体纵向拉伸可以在一定程度中起到“长腿”效果，但实际上也会导致画面中的其他事物产生较大形变，大大影响视频的显示效果；而且由于人体姿势的变化幅度一般较大(比如跳舞)，因此，会出现不同帧中的长腿效果不一致、以及背景抖动幅度较大的情况，也就是说，这些方案的鲁棒性和适应性较差，调整效果不佳，不利于提高视频画面的显示效果。
本申请一些实施例提供一种视频数据处理方法、装置和存储介质。
其中，该视频数据处理装置具体可以集成在网络设备中，该网络设备可以是终端或服务器等设备。例如，参见图1a，网络设备可以从视频流汇总采集当前视频帧，并对该当前视频帧中的人体关键点的位置进行检测，得到当前视频帧中人体的点位集；然后，获取该视频流的历史视频帧中人体的点位集，并根据该当前视频帧中人体的点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度并根据当前视频帧中人体的点位集和历史视频帧中人体的点位集分析当前视频 帧中人体关键点的抖动幅度，在根据抖动幅度确定当前视频帧位于稳定态时，采用预设固定值作为该当前视频帧的调整参考位置，反之，在非稳定态时，则根据当前视频帧中人体的点位集计算该当前视频帧的调整参考位置(即采用灵活的调整参考位置)，此后，对当前视频帧中调整参考位置对应的人体部分进行拉伸，并输出拉伸后的当前视频帧，比如，所述调整参考位置可以是腰线，可以根据黄金比例(即腿长为身长的0.618)对当前视频帧中腰线位置以下的部分进行拉伸，得到结果帧，然后，输出该结果帧，等等。
以下分别进行详细说明。
图1b为本申请一些实施例提供的视频数据处理方法的流程图。在图1b所示的实施例中，将从视频数据处理装置的角度进行描述，该视频数据处理装置具体可以集成在网络设备中，该网络设备可以是终端或服务器等设备，其中，该终端可以包括手机、平板电脑、笔记本电脑或个人计算机(PC，Personal Computer)等。
如图1b所示，该视频数据处理方法的具体流程可以如下：
101、从视频流中采集当前视频帧。
例如，具体可以通过本地摄像部件采集视频图像，得到当前视频帧；或者，也可以通过接收其他设备，如其他终端或摄像设备发送的视频图像，得到当前视频帧，等等。
102、对该当前视频帧中的人体关键点的位置进行检测，得到当前视频帧中人体的点位集；例如，具体可以如下：
采用姿势估计模型对该当前视频帧中的人体关键点的位置进行检测，得到当前视频帧中人体的点位集。
其中，该姿势估计模型可以根据实际应用的需求预先进行建立，可以由其他设备建立后提供给该视频数据处理装置，也可以由视频数据处 理装置自行进行建立。例如，具体可以采集多个已标注了人体关键点的图像样本，然后，采用这些图像样本对预设姿势估计初始模型进行训练，便可得到该姿势估计模型。其中，训练的方式可以如下：
从这些图像样本中确定当前需要进行训练的图像样本，然后，将该当前需要进行训练的图像样本导入该姿势估计初始模型中进行检测，得到该当前需要进行训练的图像样本的预测人体关键点，对该当前需要进行训练的图像样本的预测人体关键点和标注的人体关键点进行收敛，以对该姿势估计初始模型中的参数进行调整，便可以完成一次训练，此后，可以返回执行“从这些图像样本中确定当前需要进行训练的图像样本”的步骤，以采用另一个图像样本进行另一次训练，依次类推，经过多次训练后，便可得到训练好的姿势估计模型。
其中，人体关键点指的是可以标识出人体关键部位的特征点，其具体可以根据实际应用的需求而定。比如，如图1c所示，该人体关键点可以包括如表一所示的特征点：
表一：
Figure PCTCN2019091340-appb-000001
Figure PCTCN2019091340-appb-000002
需说明的是，由于人体的姿势的变化，该当前视频帧中人体的点位集中可能会丢失某些人体关键点，譬如，在当人体的右手放在背后时，可能检测不到“右手肘”、“右手腕”和“右手心”等人体关键点，则此时，可以沿用上一次的有效人体关键点，比如，若该当前视频帧的上一帧的“右手肘”、“右手腕”和“右手心”等人体关键点是有效的，则可以将该当前视频帧的上一帧的“右手肘”、“右手腕”和“右手心”等人体关键点作为该当前视频帧中人体的点位集中的“右手肘”、“右手腕”和“右手心”等人体关键点。
在得到当前视频帧中人体的点位集之后，还可以进一步包括：
102’，保存该当前视频帧中人体的点位集。
这样，若后续需要获取该当前视频帧中人体的点位集时(该当前视频帧可以作为后续视频帧的历史视频帧)，便可以直接读取该保存的点位集，详见步骤103。
103、获取历史视频帧中人体的点位集。
其中，历史视频帧指的是在该视频流中，以该当前视频帧为起点，过去预设时间范围内的视频帧，比如，若当前帧为某视频流的第三帧，则其历史视频帧可以为该视频流的第一帧和第二帧，等等。
获取历史视频帧中人体的点位集的方式可以有多种，比如，参见图1d，可以如下：
1031，获取以该当前视频帧为起点，过去预设时间范围内的视频帧，得到历史视频帧；
1032，对该历史视频帧中的人体关键点的位置进行检测，得到该历史视频帧中人体的点位集。
又比如，若在此之前，已保存了历史视频帧中人体的点位集，则此时，可以直接读取该历史视频帧中人体的点位集，即步骤103也可以包括：
1033，获取以该当前视频帧为起点，过去预设时间范围内的视频帧，得到历史视频帧；
1034，确定是否保存了该历史视频帧中人体的点位集，若是，则执行步骤1035，读取该保存的历史视频帧中人体的点位集；若否，则执行步骤1036，对该历史视频帧中的人体关键点的位置进行检测，得到该历史视频帧中人体的点位集。
104，根据该当前视频帧中人体的点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度。
在得到历史视频帧中人体的点位集之后，便可以根据该当前视频帧中人体的点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度，例如，参见图1e，具体可以如下：
1041，从该当前视频帧中人体的点位集中获取当前视频帧中人体胯部的点位。
比如，参见表一，具体可以从该当前视频帧中人体的点位集中获取“右胯”人体关键点和“左胯”人体关键点，得到当前视频帧中人体胯部的点位。
1042，从该历史视频帧中人体的点位集中获取历史视频帧中人体胯部的点位。
比如，参见表一，具体可以从该历史视频帧中人体的点位集中获取“右胯”人体关键点和“左胯”人体关键点，得到历史视频帧中人体胯部的 点位。
1043，根据当前视频帧中人体胯部的点位和历史视频帧中人体胯部的点位分析当前视频帧中人体关键点的抖动幅度。例如，具体可以如下：
根据当前视频帧中人体胯部的点位计算当前视频帧中人体的腰线，根据历史视频帧中人体胯部的点位计算历史视频帧中人体的腰线，计算当前视频帧中人体的腰线与历史视频帧中人体的腰线的距离，得到当前视频帧中人体关键点的抖动幅度。
比如，具体可以计算当前视频帧中“右胯”点(即图1c中8号点)和“左胯”点(即图1c中11号点)的平均值，得到当前视频帧中人体的腰线；以及计算历史视频帧中“右胯”点(即图1c中8号点)和“左胯”点(即图1c中11号点)的平均值，得到历史视频帧中人体的腰线；然后，计算该当前视频帧中人体的腰线与历史视频帧中人体的腰线的距离，便可得到当前视频帧中人体关键点的抖动幅度。
此后，可以确定该抖动幅度是否小于预设阈值，若是，则确定该当前视频帧位于稳定态，于是执行步骤105，若否，则确定该当前视频帧位于非稳定态，于是执行步骤106。
其中，该预设阈值可以根据实际应用的需求进行设置，比如，可以设置上下帧腰线浮动在“10％”之内，为稳定态，而超过“10％”，则为非稳定态，当然，还可以设置当连续X1帧腰线浮动在某个幅度如“10％”之内，则确定为稳定态，而当连续X2帧腰线浮动在某个幅度如“10％”之外，则为非稳定态，等等。其中，该X1和X2也可以根据实际应用的需求而定，比如，X1可以为60帧，X2可以为100帧，等等，在此不作赘述。
由于人体剧烈运动(比如跳舞)时，上下帧之间的点位可能差距较大，可能会造成点位剧烈抖动的问题，因此，为了避免该问题，改善处理效果，在分析人体关键点的抖动幅度之前，还可以对这些点位进行平 滑处理；即在步骤104之前，该视频数据处理方法还可以包括：
104’，采用预设点位平滑算法，根据该历史视频帧中人体的点位集对该当前视频帧中人体的点位集进行平滑处理，得到当前视频帧中人体的平滑后点位集。
则此时，步骤104具体可以为：根据该当前视频帧中人体的平滑后点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度。
比如，具体可以从该当前视频帧中人体的平滑后点位集中获取当前视频帧中人体胯部的点位，以及从该历史视频帧中人体的点位集中获取历史视频帧中人体胯部的点位，然后，根据当前视频帧中人体胯部的点位和历史视频帧中人体胯部的点位分析当前视频帧中人体关键点的抖动幅度，等等。
该点位平滑算法可以根据实际应用的需求进行设置，比如，具体可以从时间、空间和置信度等维度对该当前视频帧中人体的点位集进行平滑处理。即，参见图1f，步骤104’可以包括：
104’-A、根据该历史视频帧中人体的点位集，在时间维度上对该当前视频帧中人体的点位集进行平滑处理，得到时间平滑结果。
例如，具体可以从该历史视频帧中选择N帧(N为大于1的正整数)，然后，在时间维度上，将该当前视频帧中人体的点位集在该N帧之间进行平滑，比如，可以对该N帧的时间向量进行加权(即加权和)运算，如下：
Figure PCTCN2019091340-appb-000003
其中，
Figure PCTCN2019091340-appb-000004
为该当前视频帧(即第i帧)的点位集中某个人体关键点S的时间平滑结果，P j为该N帧历史视频帧里某一帧j中人体关键点S的时间向量，α j为P j的权重，离当前视频帧越近，该权重α j越大。
104’-B、根据该历史视频帧中人体的点位集，在空间维度上对该当前视频帧中人体的点位集进行平滑处理，得到空间平滑结果。
例如，具体可以从该历史视频帧中选择该当前视频帧的上一帧，然后，将该上一帧的点位集中人体关键点的空间向量加上偏移量，便可得到该当前视频帧中相应人体关键点的空间平滑结果；比如，对该上一帧的点位集中的右胯点加上偏移量，便可得到该当前视频帧中右胯点的空间平滑结果，以此类推，便可得到该当前视频帧中所有人体关键点(即点位集中所有点位)对应的空间平滑结果，用公式表示即为：
Figure PCTCN2019091340-appb-000005
其中，
Figure PCTCN2019091340-appb-000006
为该当前视频帧(即第i帧)的点位集中某个人体关键点S的空间平滑结果，P i-1为该当前视频帧的上一帧(即第i-1帧)的点位集中人体关键点S的空间向量，P i为该当前视频帧中人体的点位集中人体关键点S的空间向量；f为偏移函数(非线性)，其具有阻尼作用，通过该空间维度上的平滑处理，可以减少上下帧点位之间的差距。
104’-C、根据该历史视频帧中人体的点位集，在置信度维度上对该当前视频帧中人体的点位集进行平滑处理，得到置信度平滑结果。
例如，具体可以从该历史视频帧中选择M帧(M为大于1的正整数)，然后，在置信度维度上，将该当前视频帧中人体的点位集在该M帧之间进行平滑，比如，可以对该M帧的置信度进行加权(即加权和)运算，如下：
Figure PCTCN2019091340-appb-000007
其中，
Figure PCTCN2019091340-appb-000008
为该当前视频帧(即第i帧)的点位集中某个人体关键点S的置信度平滑结果，P k为该M帧历史视频帧里的某一帧k中人体关键点S的置信度向量，β k为P k的权重，其可以由该需要计算权重的帧的置信度score k除以该M帧历史视频帧的置信度之和∑score来得到，如 下：
β k＝score k/∑score。
其中，置信度可以在对人体关键点的位置进行检测时得到，详见步骤102。置信度越大，表示该人体关键点的所检测出来的位置越准确。
104’-D、对该时间平滑结果、空间平滑结果和置信度平滑结果进行加权(即加权和)运算，得到当前视频帧中人体的平滑后点位集，用公式表示可以如下：
Figure PCTCN2019091340-appb-000009
其中，P i为当前视频帧(即第i帧)的点位集中某个人体关键点S的总平滑结果，依次类推，计算出该当前视频帧点位集中的所有人体关键点的总平滑结果，即可得到当前视频帧中人体的平滑后点位集(该平滑后点位集包括了该当前视频帧点位集中的所有人体关键点的总平滑结果)；γ 1、γ 2和γ 3分别为
Figure PCTCN2019091340-appb-000010
和
Figure PCTCN2019091340-appb-000011
的权重，该权重具体可以根据实际应用的需求进行设置，在此不再赘述。
105、根据抖动幅度确定该当前视频帧中人体的调整参考位置。
在一些实施例中，所述调整参考位置可以是腰线位置，对应的，所述预设固定值可以是预设腰线值。例如，参见图1g，具体可以如下：
1051，根据抖动幅度确定该当前视频帧位于稳定态时，采用预设腰线值作为该当前视频帧的腰线位置。
其中，该预设腰线值可以根据实际应用的需求进行设置，比如，可以采用该当前帧的前若干个历史视频帧的平均腰线位置作为该腰线值，或者，也可以采用该当前帧的上一帧的腰线位置作为该腰线值，等等，在此不作赘述。
这是因为，人在舞蹈或轻微运动时，腰线会上下浮动，进而导致其背景拉伸区域的变化，如果每一帧都根据其实际腰线进行拉伸的话，背 景形变浮动会比较大且显得不自然。而鉴于人在舞动或轻微运动时，腰线不会变化太大，以及腰线的轻微变化对于长腿效果影响并不太明显，所以，可以在人体比较稳定，即当前视频帧位于稳定态时，采用一固定腰线位置，以作为候选拉伸的参考基准，来减少背景形变的浮动，使得显得更为自然，提高处理效果。
1052，根据抖动幅度确定该当前视频帧位于非稳定态时，根据当前视频帧中人体的点位集计算该当前视频帧的腰线位置。
由于该当前视频帧位于非稳定态，因此，表明该当前视频帧的腰线较历史视频帧浮动较大，如果依旧采用预设腰线值作为该当前视频帧的腰线位置的话，可以会导致后续的拉伸不准确，影响拉伸效果，因此，此时可以根据当前视频帧中人体的点位集计算该当前视频帧的腰线位置，具体可以如下：
从该当前视频帧中人体的点位集中获取当前视频帧中人体胯部的点位，根据当前视频帧中人体胯部的点位计算当前视频帧中人体的腰线，根据计算得到的腰线确定该当前视频帧的腰线位置。
比如，具体可以从该当前视频帧中人体的点位集中获取当前视频帧中人体的“右胯”点和“左胯”点，然后，计算该“右胯”点和“左胯”点的平均值，得到当前视频帧中人体的腰线，可以直接将该腰线的位置(如坐标)作为该当前视频帧的腰线位置，也可以按照预设策略对该腰线的位置进行微调后，作为该当前视频帧的腰线位置。
其中，该预设策略可以根据实际应用的需求而定，在此不作赘述。
106、对当前视频帧中调整参考位置对应的人体部分进行调整，并输出调整后的当前视频帧。
在一些实施例中，当所述调整参考位置为腰线位置时，步骤106可以为：对当前视频帧中的腰线位置以下的人体部分进行拉伸，并输出拉 伸后的当前视频帧。例如，具体可以如下：
根据黄金比例(即腿长为身长的0.618)对当前视频帧中腰线位置以下的人体部分进行拉伸，得到结果帧，输出该结果帧。
或者，也可以根据用户的喜好或设置对当前视频帧中腰线位置以下的人体部分进行拉伸，得到结果帧，输出该结果帧。
由于拉伸后，在拉伸分界线(即腰线位置)上有一明显的斜率跳变，使得背景显示较为不自然，影响其图像显示效果，因此，在进行拉伸之后，还可以对该部分的斜率进行平滑处理，以消除该“跳变”，即在步骤“根据黄金比例对当前视频帧中腰线位置以下的人体部分进行拉伸”之前，该视频数据处理方法还可以包括：
对拉伸后当前视频帧中腰线位置部分的斜率进行平滑处理，得到结果帧。
其中，该平滑处理的策略可以根据实际应用的需求而定，比如，可以在该腰线位置部分设置一过渡带，比如设置一20％的过渡带，然后，对该过渡带内的斜率进行下调，使其拉伸交界处更加平滑柔和。
若该当前视频帧中存在多个人体，则必然会存在多个腰线位置，那么在拉伸时，可以从中选一腰线位置作为目标腰线位置来进行拉伸，比如，可以选择位置最高的腰线位置来作为目标腰线位置，等等。即步骤“对当前视频帧中腰线位置以下的人体部分进行拉伸，并输出拉伸后的当前视频帧”可以包括：
从当前视频帧的多个腰线位置中选择位置最高的腰线位置作为目标腰线位置，对当前视频帧中目标腰线位置以下的人体部分进行拉伸，并输出拉伸后的当前视频帧。
比如，具体可以根据黄金比例对当前视频帧中目标腰线位置以下的人体部分进行拉伸，并输出拉伸后的当前视频帧。
可以对拉伸后当前视频帧中目标腰线位置部分的斜率进行平滑处理，然后才进行输出。即步骤“对当前视频帧中目标腰线位置以下的人体部分进行拉伸，并输出拉伸后的当前视频帧”可以包括；
根据黄金比例对当前视频帧中目标腰线位置以下的人体部分进行拉伸，对拉伸后当前视频帧中目标腰线位置部分的斜率进行平滑处理，得到结果帧，输出结果帧。
其中，当当前视频帧中存在多个人体时，其拉伸的程度可以根据实际应用的需求而定，比如，可以取该多个人体拉伸程度的均值来进行拉伸，譬如，可以根据该当前视频帧中每个人体的腿长和身长的黄金比例，确定每个人体的最佳拉伸程度，然后，计算该当前视频帧中所有人体的拉伸程度的均值，根据该均值对当前视频帧中目标腰线位置以下的部分进行拉伸，等等。
需说明的是，在本申请实施例中，对腰线位置(或目标腰线位置)以下的人体部分进行拉伸可以包括只对视频帧中腰线位置(或目标腰线位置)以下的人体图像区域进行拉伸，也可以包括对视频帧中腰线位置(或目标腰线位置)以下的所有图像区域进行整体拉伸。其中，在只对人体图像区域进行拉伸时，可以根据检测到的人体关键点从视频帧中提取出人体图像区域，作为前景部分，并将提取后剩余的其他部分作为背景部分，然后，对前景部分中腰线位置以下的图像区域进行拉伸，再将拉伸后的前景部分与背景部分进行叠加，即可得到拉伸后当前视频帧，在此不作赘述。
在执行步骤106之后，可以返回执行采集当前视频帧的步骤(即步骤101)，以对下一视频帧进行调整，直至视频采集结束，这样，便可实现对视频中的人体进行实时调整，比如对视频中的人体进行调整的功能。
由上可知，本实施例在采集到当前视频帧后，可以对当前视频帧中 的人体关键点的位置进行检测，得到当前视频帧中人体的点位集，然后，获取历史视频帧中人体的点位集，并根据当前视频帧中人体的点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度，根据抖动幅度确定该当前视频帧中人体的调整参考位置，比如，在根据抖动幅度确定当前视频帧位于稳定态时，采用预设固定值作为该当前视频帧的调整参考位置，反之，在非稳定态时，则根据当前视频帧中人体的点位集计算该当前视频帧的调整参考位置，等等，此后，便可以对当前视频帧中调整参考位置对应的人体部分进行调整并输出，进而实现对当前帧中人体进行实时调整的目的；由于该方案只对调整参考位置对应的人体部分进行调整，而且对于抖动幅度不大的视频帧，采用的是固定调整参考位置，对于抖动幅度较大的视频帧，才采用灵活的调整参考位置，因此，可以在实现对视频帧中人体进行实时调整，改善调整效果的同时，较大程度保证视频帧中背景形变的自然平滑，大大提高了实时调整功能的鲁棒性和适应性，有利于提高视频画面的显示效果。
根据前面实施例所描述的方法，以下将以该视频数据处理装置具体集成在网络设备中为例进行说明。
如图2a所示，一种视频数据处理方法，具体流程可以如下：
201、网络设备从视频流中采集当前视频帧。
例如，网络设备可以通过本地摄像部件采集视频图像，得到当前视频帧；或者，也可以通过接收其他设备，如其他终端或摄像设备发送的视频图像，得到当前视频帧，等等。
比如，如图2f所示，当用户需要对视频进行实时长腿时，可以点击界面中的“实时长腿”触发键281来触发该长腿功能。
还可以通过其他接口如“触发键”、“输入框”或“滑杠”设置“拉伸”程度，比如，参见图2f，若用户将“滑杠”283调整至“65％”处，则意味着可以 将当前视频帧中人体的腿部的长度在原来的基础上，增加65％。
该用户界面中还可以设置其他的接口，用于调用预设的其他功能函数，以实现该功能函数对应的功能。比如设置一“选择框”282，当用户选择该“选择框”282，则可以触发调用预设功能函数，使得用户在当前视频帧所设置的“拉伸”程度，默认适用于其他的视频帧(当然，若用户需要对某一帧中的人体腿部的拉伸程度进行重新设置，也可以通过当前视频帧中的用于设置“拉伸”程度的接口来对该默认项进行调整)；而当用户未选择该“选择框”282时，则用户可以灵活对视频帧中的人体腿部的拉伸程度进行逐帧设置，比如，参见图2f，同样可以通过用户界面中的“滑杠”283来进行调整，在此不作赘述。
202、网络设备采用姿势估计模型对该当前视频帧中的人体关键点的位置进行检测，得到当前视频帧中人体的点位集。
其中，该姿势估计模型可以根据实际应用的需求预先进行建立，例如，具体可以采集多个已标注了人体关键点的图像样本，然后，采用这些图像样本对预设姿势估计初始模型进行训练，便可得到该姿势估计模型，具体可参见前面的实施例，在此不作赘述。
其中，人体关键点指的是可以标识出人体关键部位的特征点，其具体可以根据实际应用的需求而定。比如，具体可以包括如图1c和表一所示的22个人体关键点：鼻尖、脖子(华盖穴)、右肩、右手肘、右手腕、左肩、左手肘、左手腕、右胯、右膝盖、右脚踝、左胯、左膝盖、左脚踝、右眼、左眼、左眼、右耳、左耳、右手心、左手心、右脚尖和左脚尖等。
例如，参见图2b，网络设备在采集到当前视频帧之后，可以将该当前视频帧导入至姿势估计模型中进行检测，姿势估计模型对该当前视频帧进行人体关键点位置检测之后，可以输出检测到的多个人体关键点的 位置(简称点位)，将该多个人体关键点的位置添加至同一集合中，便可得到该当前视频帧中人体的点位集。
需说明的是，由于人体的姿势的变化，在对该当前视频帧中的人体关键点的位置进行检测时，可能会存在若干个人体关键点无法检测到，譬如，在当人体的左手放在背后时，可能检测不到“左手肘”、“左手腕”和“左手心”等人体关键点，则此时，可以沿用上一次的有效人体关键点，比如，若该当前视频帧的上上一帧的“左手肘”、“左手腕”和“左手心”等人体关键点是有效的，则在可以将该当前视频帧的上上一帧的“左手肘”、“左手腕”和“左手心”等人体关键点作为该当前视频帧中人体的点位集中的“左手肘”、“左手腕”和“左手心”等人体关键点，以此类推，等等。
在得到当前视频帧中人体的点位集之后，网络设备还可以保存该当前视频帧中人体的点位集，这样，若后续需要获取该当前视频帧中人体的点位集时(该当前视频帧可以作为后续视频帧的历史视频帧)，便可以直接读取该保存的点位集，详见步骤203。
203、网络设备获取该视频流的历史视频帧中人体的点位集。
例如，网络设备可以获取以该当前视频帧为起点，过去预设时间范围内的视频帧，得到历史视频帧，然后，对该历史视频帧中的人体关键点的位置进行检测，得到该历史视频帧中人体的点位集。
又比如，若在此之前，已保存了历史视频帧中人体的点位集，则此时，网络设备也可以直接读取该历史视频帧中人体的点位集，即步骤“网络设备获取历史视频帧中人体的点位集”也可以包括：
网络设备从该视频流中，获取以该当前视频帧为起点，过去预设时间范围内的视频帧，得到历史视频帧，确定是否保存了该历史视频帧中人体的点位集，若是，则读取该保存的历史视频帧中人体的点位集；若否，则对该历史视频帧中的人体关键点的位置进行检测，得到该历史视 频帧中人体的点位集。
其中，该预设时间范围可以根据实际应用的需求而定，比如，以当前视频帧具体为第10帧，其该预设时间范围为5帧为例，则此时，网络设备可以获取第5帧、第6帧、第7帧、第8帧和第9帧的视频帧作为历史视频帧，然后，确定是否存在这些历史视频帧中人体的点位集，如果存在，则直接读取相应的点位集，如果不存在，则对该历史视频帧中的人体关键点的位置进行检测，比如，如果第9帧没有保存相应的点位集，则此时需要对第9帧中的人体关键点的位置进行检测，以得到第9帧的点位集，以此类推，等等。
其中，对历史视频帧中的人体关键点的位置进行检测的方式则与对当前视频帧中的人体关键点的检测方式类似，即具体可以采用姿势估计模型对该历史视频帧中的人体关键点的位置进行检测，其中，该姿势估计模型可参见步骤202中的描述，在此不作赘述。
204、网络设备采用预设点位平滑算法，根据该历史视频帧中人体的点位集对该当前视频帧中人体的点位集进行平滑处理，得到当前视频帧中人体的平滑后点位集。
该点位平滑算法可以根据实际应用的需求进行设置，比如，具体可以从时间、空间和置信度等维度对该当前视频帧中人体的点位集进行平滑处理，具体可以如下：
(1)时间维度；
网络设备根据该历史视频帧中人体的点位集，在时间维度上对该当前视频帧中人体的点位集进行平滑处理，得到时间平滑结果。
(2)空间维度；
网络设备根据该历史视频帧中人体的点位集，在空间维度上对该当前视频帧中人体的点位集进行平滑处理，得到空间平滑结果。
(3)置信度维度；
网络设备根据该历史视频帧中人体的点位集，在置信度维度上对该当前视频帧中人体的点位集进行平滑处理，得到置信度平滑结果。
具体计算时间平滑结果、空间平滑结果和置信度平滑结果的过程可以参见步骤104’-A至104’-C，在此不再赘述。在得到时间平滑结果、空间平滑结果和置信度平滑结果之后，便可以对该时间平滑结果、空间平滑结果和置信度平滑结果进行加权(即加权和)运算，得到当前视频帧中人体的平滑后点位集，参见步骤104’-D，在此不再赘述。
205、网络设备根据该当前视频帧中人体的平滑后点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度。例如，参见表一，可以如下：
网络设备从该当前视频帧中人体的点位集中获取“右胯”人体关键点和“左胯”人体关键点，得到当前视频帧中人体胯部的点位，以及从该历史视频帧中人体的点位集中获取“右胯”人体关键点和“左胯”人体关键点，得到历史视频帧中人体胯部的点位，然后，根据当前视频帧中人体胯部的点位计算当前视频帧中人体的腰线，根据历史视频帧中人体胯部的点位计算历史视频帧中人体的腰线，计算当前视频帧中人体的腰线与历史视频帧中人体的腰线的距离，得到当前视频帧中人体关键点的抖动幅度。
比如，具体可以计算当前视频帧中“右胯”点和“左胯”点的平均值，得到当前视频帧中人体的腰线；以及计算历史视频帧中“右胯”点和“左胯”点的平均值，得到历史视频帧中人体的腰线；然后，计算该当前视频帧中人体的腰线与历史视频帧中人体的腰线的距离，便可得到当前视频帧中人体关键点的抖动幅度。
206、网络设备确定该抖动幅度是否小于预设阈值，若是，则确定该当前视频帧位于稳定态，于是执行步骤207，若否，则确定该当前视频 帧位于非稳定态，于是执行步骤208。
其中，该预设阈值可以根据实际应用的需求进行设置，比如，以该预设阈值为10％为例，则此时，网络设备可以该抖动幅度(即上下帧腰线浮动)是否小于10％，若是，则定该当前视频帧位于稳定态，于是执行步骤207，若否，比如，其上下帧浮动为20％，则可以确定该当前视频帧位于非稳定态，于是执行步骤208。
207、网络设备根据抖动幅度确定该当前视频帧位于稳定态时，采用预设腰线值作为该当前视频帧的腰线位置。
其中，该预设腰线值可以根据实际应用的需求进行设置，比如，可以采用该当前帧的前若干个历史视频帧的平均腰线位置作为该腰线值，或者，也可以采用该当前帧的上一帧的腰线位置作为该腰线值，等等，在此不作赘述。
208、网络设备根据抖动幅度确定该当前视频帧位于非稳定态时，根据当前视频帧中人体的点位集计算该当前视频帧的腰线位置，例如，具体可以如下：
从该当前视频帧中人体的点位集中获取当前视频帧中人体胯部的点位，根据当前视频帧中人体胯部的点位计算当前视频帧中人体的腰线，根据计算得到的腰线确定该当前视频帧的腰线位置。
比如，具体可以从该当前视频帧中人体的点位集中获取当前视频帧中人体的“右胯”点和“左胯”点，然后，计算该“右胯”点和“左胯”点的平均值，得到当前视频帧中人体的腰线，可以直接将该腰线的位置(如坐标)作为该当前视频帧的腰线位置，也可以按照预设策略对该腰线的位置进行微调后，作为该当前视频帧的腰线位置。
其中，该预设策略可以根据实际应用的需求而定，在此不作赘述。
209、网络设备对当前视频帧中腰线位置以下的人体部分进行拉伸。
其中，在拉伸时，可以按照黄金比例(即腿长为身长的0.618)来进行拉伸，或者，也可以根据用户的喜好或设置进行拉伸，比如，可以将腰线位置以下的部分拉伸为原来的1.2倍或1.5倍，等等。
若该当前视频帧中存在多个人体，则必然会存在多个腰线位置，那么在拉伸时，网络设备可以从中选一腰线位置作为目标腰线位置来进行拉伸，比如，可以从当前视频帧的多个腰线位置中选择位置最高的腰线位置作为目标腰线位置，然后，对当前视频帧中目标腰线位置以下的人体部分进行拉伸。具体的拉伸程度可以根据实际应用的需求而定，比如，可以取该多个人体拉伸程度的均值来进行拉伸，如下：
根据该当前视频帧中每个人体的腿长和身长的黄金比例，确定每个人体的最佳拉伸程度，然后，计算该当前视频帧中所有人体的拉伸程度的均值，根据该均值对当前视频帧中目标腰线位置以下的人体部分进行拉伸，等等。
由于拉伸后，在拉伸分界线(即腰线位置)上有一明显的斜率跳变，使得背景显示较为不自然，影响其图像显示效果；比如，若腰线位置为“y＝0.5”，那么，在拉伸时，将会从“y＝0.5”这条线开始拉伸，假设拉伸程度为20％，则该当前视频帧中的像素点的斜率变化可以如图2c所示，从图2c可以看到，在y＝0.5有一明显的斜率跳变，因此，为了消除该“跳变”，在进行拉伸之后，还可以对该部分的斜率进行平滑处理，即该视频数据处理方法还可以执行步骤210。
需说明的是，在本申请实施例中，对腰线位置(或目标腰线位置)以下的人体部分进行拉伸方式可以有多种，例如，可以如下：
网络设备对视频帧中腰线位置(或目标腰线位置)以下的人体图像区域进行拉伸。
比如，具体可以根据检测到的人体关键点从视频帧中提取出人体图 像区域，作为前景部分，并将提取后剩余的其他部分作为背景部分，然后，对前景部分中腰线位置以下的图像区域进行拉伸，再将拉伸后的前景部分与背景部分进行叠加，即可得到拉伸后当前视频帧。
为了提高简化操作，提高处理效率，也可以不对前景和背景进行划分，而是对视频帧中腰线位置(或目标腰线位置)以下的所有图像区域进行整体拉伸，在此不作赘述。
210、网络设备对拉伸后当前视频帧中腰线位置部分的斜率进行平滑处理，得到结果帧，然后执行步骤211。
其中，该平滑处理的策略可以根据实际应用的需求而定，比如，可以在该腰线位置部分设置一过渡带然后，对该过渡带内的斜率进行下调，使其拉伸交界处更加平滑、以及柔和。
例如，还是以图2c中的例子为例，由于在y＝0.5有一明显的斜率跳变，因此，可以在y＝0.5的位置设置一过渡带，比如，设置一20％的过渡带，然后，将该过渡带内的斜率从1.0平滑至0.8，由图2d可以看到，平滑处理后的拉伸交界处相对于图2c中的拉伸交界处而言，显得更加平滑柔和。
当然，该过渡带的具体设置、以及斜率下调的幅度可以根据实际应用的需求进行灵活设置，上述的例子仅仅为示例，并不作为限定。
211、网络设备输出结果帧。
例如，网络设备可以在终端界面上显示该结果帧，比如，参见图2f中的右图，可以显示该拉伸了腿部之后的视频帧，进而实现在录制视频时，实时长腿的功能。
如图2f所示，用户可以通过调整界面上的“滑杠”283，来灵活调整当前视频帧中人体腿部的拉伸程度。
在一些实施例中，用户还可以通过预设触发键，如通过按压视频画 面来查看拉伸前后的对比效果。比如，如图2e所示，该图为拉伸前和拉伸后的当前视频帧的对比图，其中，左图为拉伸前的当前视频帧，右图为拉伸后的当前视频帧，可见，在对腰线位置以下的人体部分进行拉伸之后，左图中的人体的腿长明显大于右图中人体的腿长。
需说明的是，在执行步骤210之后，可以返回执行采集当前视频帧的步骤(即步骤201)，以对下一视频帧进行拉伸，直至视频采集结束，这样，便可实现对视频中的人体进行实时拉伸，比如对视频中的人体进行长腿的功能。
由上可知，本实施例在采集到当前视频帧后，可以对当前视频帧中的人体关键点的位置进行检测，得到当前视频帧中人体的点位集，然后，获取历史视频帧中人体的点位集，根据该历史视频帧中人体的点位集对该当前视频帧中人体的点位集进行平滑处理，并根据当前视频帧中人体的平滑后点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度，在根据抖动幅度确定当前视频帧位于稳定态时，采用预设固定腰线值作为该当前视频帧的腰线位置，反之，在非稳定态时，则根据当前视频帧中人体的点位集计算该当前视频帧的腰线位置，此后，对当前视频帧中腰线位置以下的人体部分进行拉伸并输出，进而实现对当前帧中人体进行实时长腿的目的；由于该方案只对腰线以下的部分进行拉伸，而且对于抖动幅度不大的视频帧，采用的是固定腰线，对于抖动幅度较大的视频帧，才采用灵活的腰线，因此，可以在实现对视频帧中人体进行实时长腿，改善长腿效果的同时，较大程度保证视频帧中背景形变的自然平滑，大大提高了实时长腿功能的鲁棒性和适应性，有利于提高视频画面的显示效果。
此外，由于该方案在得到当前视频帧中人体的点位集之后，还可以对该点位集进行平滑处理，因此，可以大大降低由于人体剧烈运动所导 致的上下帧之间的点位差距较大的问题，可以进一步改善视频画面的处理效果。
本申请实施例还可以提供一种视频数据处理装置，该视频数据处理装置具体可以集成在网络设备中，该网络设备可以是终端或服务器等设备。
例如，如图3a所示，该视频数据处理装置可以包括检测单元301、获取单元302、分析单元303、确定单元304和拉伸单元305，如下：
检测单元301，用于采集当前视频帧，并对该当前视频帧中的人体关键点的位置进行检测，得到当前视频帧中人体的点位集。
例如，该检测单元301，具体可以用于采用姿势估计模型对该当前视频帧中的人体关键点的位置进行检测，得到当前视频帧中人体的点位集。
其中，该姿势估计模型可以根据实际应用的需求预先进行建立，可以由其他设备建立后提供给该视频数据处理装置，也可以由视频数据处理装置自行进行建立，即该视频数据处理装置还可以包括建立单元，具体可以用于采集多个已标注了人体关键点的图像样本，然后，采用这些图像样本对预设姿势估计初始模型进行训练，便可得到该姿势估计模型。其中，训练的方式具体可参见前面的方法实施例，在此不作赘述。
其中，人体关键点指的是可以标识出人体关键部位的特征点，其具体可以根据实际应用的需求而定，比如，具体可包括表一所示的22个人体关键点，详见前面的实施例，在此不作赘述。
需说明的是，由于人体的姿势的变化，该当前视频帧中人体的点位集中可能会丢失某些人体关键点，若某些关键点发生了丢失(即关键点失效)，则此时，可以沿用上一次的有效人体关键点。
在得到当前视频帧中人体的点位集之后，还可以保存该当前视频帧 中人体的点位集，这样，若后续需要获取该当前视频帧中人体的点位集时(该当前视频帧可以作为后续视频帧的历史视频帧)，便可以直接读取该保存的点位集；即如图3b所示，该视频数据处理装置还可以包括保存单元306，如下：
该保存单元306，用于保存该当前视频帧中人体的点位集。
获取单元302，用于获取历史视频帧中人体的点位集。
其中，历史视频帧指的是在该视频流中，以该当前视频帧为起点，过去预设时间范围内的视频帧。
获取历史视频帧中人体的点位集的方式可以有多种，例如：
该获取单元302，具体可以用于获取以该当前视频帧为起点，过去预设时间范围内的视频帧，得到历史视频帧，然后，对该历史视频帧中的人体关键点的位置进行检测，得到该历史视频帧中人体的点位集。
又比如，若在此之前，已保存了历史视频帧中人体的点位集，则此时，可以直接读取该历史视频帧中人体的点位集，即：
该获取单元302，具体可以用于获取以该当前视频帧为起点，过去预设时间范围内的视频帧，得到历史视频帧，确定是否保存了该历史视频帧中人体的点位集，若是，则读取该历史视频帧中人体的点位集；若否，则对该历史视频帧中的人体关键点的位置进行检测，得到该历史视频帧中人体的点位集。
分析单元303，用于根据该当前视频帧中人体的点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度。
例如，该分析单元303可以包括点位获取子单元3031和分析子单元3032，如下：
该点位获取子单元3031，用于从该当前视频帧中人体的点位集中获取当前视频帧中人体胯部的点位，以及从该历史视频帧中人体的点位集 中获取历史视频帧中人体胯部的点位。
该分析子单元3032，用于根据当前视频帧中人体胯部的点位和历史视频帧中人体胯部的点位分析当前视频帧中人体关键点的抖动幅度。
例如，该分析子单元3032，具体可以用于根据当前视频帧中人体胯部的点位计算当前视频帧中人体的腰线；根据历史视频帧中人体胯部的点位计算历史视频帧中人体的腰线；计算当前视频帧中人体的腰线与历史视频帧中人体的腰线的距离，得到当前视频帧中人体关键点的抖动幅度。
比如，该分析子单元3032，具体可以用于计算当前视频帧中“右胯”点和“左胯”点的平均值，得到当前视频帧中人体的腰线；以及计算历史视频帧中“右胯”点和“左胯”点的平均值，得到历史视频帧中人体的腰线；然后，计算该当前视频帧中人体的腰线与历史视频帧中人体的腰线的距离，便可得到当前视频帧中人体关键点的抖动幅度。
确定单元304，用于根据抖动幅度确定所述当前视频帧中人体的腰线位置。
例如，该确定单元304，具体可以用于根据抖动幅度确定该当前视频帧位于稳定态时，采用预设腰线值作为该当前视频帧的腰线位置；根据抖动幅度确定该当前视频帧位于非稳定态时，根据当前视频帧中人体的点位集计算该当前视频帧的腰线位置。
譬如，该确定单元304，具体可以用于在确定抖动幅度小于预设阈值时，采用预设腰线值作为该当前视频帧的腰线位置；在确定抖动幅度大于等于预设阈值时，根据当前视频帧中人体的点位集计算该当前视频帧的腰线位置。
比如，该确定单元304，具体可以用于在根据抖动幅度确定所述当前视频帧位于非稳定态时(比如在确定抖动幅度大于等于预设阈值时)， 从该当前视频帧中人体的点位集中获取当前视频帧中人体胯部的点位，根据当前视频帧中人体胯部的点位计算当前视频帧中人体的腰线，根据计算得到的腰线确定该当前视频帧的腰线位置。
其中，该预设阈值可以根据实际应用的需求进行设置，比如，可以设置上下帧腰线浮动在“10％”之内，为稳定态，而超过“10％”，则为非稳定态，当然，还可以设置当连续X1帧腰线浮动在某个幅度如“10％”之内，则确定为稳定态，而当连续X2帧腰线浮动在某个幅度如“10％”之外，则为非稳定态，等等。其中，该X1和X2也可以根据实际应用的需求而定，比如，X1可以为60帧，X2可以为100帧，等等，在此不作赘述。
拉伸单元305，用于对当前视频帧中腰线位置以下的人体部分进行拉伸，并输出拉伸后的当前视频帧。
例如，该拉伸单元305，具体可以用于根据黄金比例(即腿长为身长的0.618)对当前视频帧中腰线位置以下的部分进行拉伸，得到结果帧，输出该结果帧。
由于拉伸后，在拉伸分界线(即腰线位置)上有一明显的斜率跳变，使得背景显示较为不自然，影响其图像显示效果，因此，在进行拉伸之后，还可以对该部分的斜率进行平滑处理，以消除该“跳变”，即：
该拉伸单元305，具体可以用于根据黄金比例对当前视频帧中腰线位置以下的人体部分进行拉伸，对拉伸后当前视频帧中腰线位置部分的斜率进行平滑处理，得到结果帧，输出该结果帧。
其中，该平滑处理的策略可以根据实际应用的需求而定，比如，可以在该腰线位置部分设置一过渡带，比如设置一20％的过渡带，然后，对该过渡带内的斜率进行下调，使其拉伸交界处更加平滑和柔和。
若该当前视频帧中存在多个人体，则必然会存在多个腰线位置，那么在拉伸时，可以从中选一腰线位置作为目标腰线位置来进行拉伸，比 如，可以选择位置最高的腰线位置来作为目标腰线位置，等等。即：
该拉伸单元305，具体可以用于从当前视频帧的多个腰线位置中选择位置最高的腰线位置作为目标腰线位置，对当前视频帧中目标腰线位置以下的人体部分进行拉伸并输出。
例如，该拉伸单元305，具体可以用于根据黄金比例对当前视频帧中目标腰线位置以下的部分进行拉伸，对拉伸后当前视频帧中目标腰线位置部分的斜率进行平滑处理，得到结果帧，输出该结果帧。
此后，拉伸单元305可以返回触发检测单元301执行采集当前视频帧的步骤，以对下一视频帧进行拉伸，直至视频采集结束，这样，便可实现对视频中的人体进行实时拉伸，比如对视频中的人体进行长腿的功能。
由于人体剧烈运动(比如跳舞)时，上下帧之间的点位可能差距较大，可能会造成点位剧烈抖动的问题，因此，为了避免该问题，改善处理效果，在分析人体关键点的抖动幅度之前，还可以对这些点位进行平滑处理；即如图3b所示，该视频数据处理装置还可以包括平滑单元307，如下：
该平滑单元307，用于采用预设点位平滑算法，根据该历史视频帧中人体的点位集对该当前视频帧中人体的点位集进行平滑处理，得到当前视频帧中人体的平滑后点位集。
则此时，该分析单元，具体用于根据该当前视频帧中人体的平滑后点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度。
例如，该平滑单307元，具体可以用于：根据该历史视频帧中人体的点位集，在时间维度上对该当前视频帧中人体的点位集进行平滑处理，得到时间平滑结果；根据该历史视频帧中人体的点位集，在空间维度上 对该当前视频帧中人体的点位集进行平滑处理，得到空间平滑结果；根据该历史视频帧中人体的点位集，在置信度维度上对该当前视频帧中人体的点位集进行平滑处理，得到置信度平滑结果；对该时间平滑结果、空间平滑结果和置信度平滑结果进行加权运算，得到当前视频帧中人体的平滑后点位集，具体可参见前面的方法实施例，在此不作赘述。
具体实施时，以上各个单元可以作为独立的实体来实现，也可以进行任意组合，作为同一或若干个实体来实现，以上各个单元的具体实施可参见前面的方法实施例，在此不再赘述。
由上可知，本实施例的视频数据处理装置在采集到当前视频帧后，可以由检测单元301对当前视频帧中的人体关键点的位置进行检测，得到当前视频帧中人体的点位集，然后，由获取单元302获取历史视频帧中人体的点位集，并由分析单元303根据当前视频帧中人体的点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度，由确定单元304根据抖动幅度确定当前视频帧中人体的腰线位置，比如，在根据抖动幅度确定当前视频帧位于稳定态时，由确定单元304采用预设固定腰线值作为该当前视频帧的腰线位置，反之，在非稳定态时，则由确定单元304根据当前视频帧中人体的点位集计算该当前视频帧的腰线位置，等等，此后，由拉伸单元305对当前视频帧中腰线位置以下的人体部分进行拉伸并输出，进而实现对当前帧中人体进行实时长腿的目的；由于该方案只对腰线以下的部分进行拉伸，而且对于抖动幅度不大的视频帧，采用的是固定腰线，对于抖动幅度较大的视频帧，才采用灵活的腰线，因此，可以在实现对视频帧中人体进行实时长腿，改善长腿效果的同时，较大程度保证视频帧中背景形变的自然平滑，大大提高了实时长腿功能的鲁棒性和适应性，有利于提高视频画面的显示效果。
本申请实施例还提供一种网络设备，具体可以为终端，也可以为服 务器，该网络设备可以集成本申请实施例所提供的任一种视频数据处理装置。
例如，如图4所示，其示出了本申请实施例所涉及的网络设备的结构示意图，具体来讲：
该网络设备可以包括一个或者一个以上处理核心的处理器401、一个或一个以上计算机可读存储介质的存储器402、电源403和输入单元404等部件。本领域技术人员可以理解，图4中示出的网络设备结构并不构成对网络设备的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。其中：
处理器401是该网络设备的控制中心，利用各种接口和线路连接整个网络设备的各个部分，通过运行或执行存储在存储器402内的软件程序和/或模块，以及调用存储在存储器402内的数据，执行网络设备的各种功能和处理数据，从而对网络设备进行整体监控。处理器401可包括一个或多个处理核心；处理器401可集成应用处理器和调制解调处理器，其中，应用处理器主要处理操作系统、用户界面和应用程序等，调制解调处理器主要处理无线通信。可以理解的是，上述调制解调处理器也可以不集成到处理器401中。
存储器402可用于存储软件程序以及模块，处理器401通过运行存储在存储器402的软件程序以及模块，从而执行各种功能应用以及数据处理。存储器402可主要包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需的应用程序(比如声音播放功能、图像播放功能等)等；存储数据区可存储根据网络设备的使用所创建的数据等。此外，存储器402可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他易失性固态存储器件。相应地，存储器402还可以包括存储器控制器， 以提供处理器401对存储器402的访问。
网络设备还包括给各个部件供电的电源403，电源403可以通过电源管理系统与处理器401逻辑相连，从而通过电源管理系统实现管理充电、放电、以及功耗管理等功能。电源403还可以包括一个或一个以上的直流或交流电源、再充电系统、电源故障检测电路、电源转换器或者逆变器、电源状态指示器等任意组件。
该网络设备还可包括输入单元404，该输入单元404可用于接收输入的数字或字符信息，以及产生与用户设置以及功能控制有关的键盘、鼠标、操作杆、光学或者轨迹球信号输入。
尽管未示出，网络设备还可以包括显示单元等，在此不再赘述。具体在本实施例中，网络设备中的处理器401会按照如下的指令，将一个或一个以上的应用程序的进程对应的可执行文件加载到存储器402中，并由处理器401来运行存储在存储器402中的应用程序，从而实现本申请实施例中任一数据处理方法。：
本领域普通技术人员可以理解，上述实施例的各种方法中的全部或部分步骤可以通过指令来完成，或通过指令控制相关的硬件来完成，该指令可以存储于一计算机可读存储介质中，并由处理器进行加载和执行。
为此，本申请实施例提供一种存储介质，其中存储有多条指令，该指令能够被处理器进行加载，以执行本申请实施例所提供的任一种视频数据处理方法中的步骤。
其中，该存储介质可以包括：只读存储器(ROM，Read Only Memory)、随机存取记忆体(RAM，Random Access Memory)、磁盘或光盘等。
以上对本申请实施例所提供的一种视频数据处理方法、装置和存储介质进行了详细介绍，本文中应用了具体个例对本申请的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本申请的方法及其 核心思想；同时，对于本领域的技术人员，依据本申请的思想，在具体实施方式及应用范围上均会有改变之处，综上所述，本说明书内容不应理解为对本申请的限制。
以上所述仅为本申请的较佳实施例而已，并不用以限制本申请，凡在本申请的原则之内所作的任何修改、等同替换和改进等，均应包含在本申请的保护范围之内。

Claims (18)

  1. 一种视频数据处理方法，由电子设备执行，包括：
    从视频流中采集当前视频帧，并对所述当前视频帧中的人体关键点的位置进行检测，得到当前视频帧中人体的点位集；
    获取历史视频帧中人体的点位集，所述历史视频帧为在所述视频流中，以所述当前视频帧为起点，过去预设时间范围内的视频帧；
    根据所述当前视频帧中人体的点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度；
    根据抖动幅度确定所述当前视频帧中人体的调整参考位置；
    对当前视频帧中调整参考对应的人体部分进行调整，并输出调整后的当前视频帧。
  2. 根据权利要求1所述的方法，所述根据抖动幅度确定所述当前视频帧中人体的调整参考位置，包括：
    根据抖动幅度确定所述当前视频帧位于稳定态时，采用预设固定值确定所述当前视频帧中人体的调整参考位置；
    根据抖动幅度确定所述当前视频帧位于非稳定态时，根据当前视频帧中人体的点位集计算所述当前视频帧中人体的调整参考位置。
  3. 根据权利要求2所述的方法，进一步包括：
    确定抖动幅度是否小于预设阈值；
    若是，则确定所述当前视频帧位于稳定态；
    若否，则确定所述当前视频帧位于非稳定态。
  4. 根据权利要求2所述的方法，所述调整参考位置为腰线位置；所述根据当前视频帧中人体的点位集计算所述当前视频帧的调整参考位置，包括：
    从所述当前视频帧中人体的点位集中获取当前视频帧中人体胯部的 点位；
    根据当前视频帧中人体胯部的点位计算当前视频帧中人体的腰线；
    根据计算得到的腰线确定所述当前视频帧的腰线位置。
  5. 根据权利要求1所述的方法，所述对所述当前视频帧中的人体关键点的位置进行检测，得到当前视频帧中人体的点位集，包括：
    采用姿势估计模型对所述当前视频帧中的人体关键点的位置进行检测，得到当前视频帧中人体的点位集。
  6. 根据权利要求1所述的方法，所述得到当前视频帧中人体的点位集之后，还包括：
    保存所述当前视频帧中人体的点位集。
  7. 根据权利要求6所述的方法，所述获取历史视频帧中人体的点位集，包括：
    获取以所述当前视频帧为起点，过去预设时间范围内的视频帧，得到历史视频帧；
    确定是否保存了所述历史视频帧中人体的点位集；
    若是，则读取所述历史视频帧中人体的点位集；
    若否，则对所述历史视频帧中的人体关键点的位置进行检测，得到所述历史视频帧中人体的点位集。
  8. 根据权利要求1至7任一项所述的方法，所述根据所述当前视频帧中人体的点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度，包括：
    从所述当前视频帧中人体的点位集中获取当前视频帧中人体胯部的点位；
    从所述历史视频帧中人体的点位集中获取历史视频帧中人体胯部的点位；
    根据当前视频帧中人体胯部的点位和历史视频帧中人体胯部的点位分析当前视频帧中人体关键点的抖动幅度。
  9. 根据权利要求8所述的方法，所述根据当前视频帧中人体胯部的点位和历史视频帧中人体胯部的点位分析当前视频帧中各个人体关键点的抖动幅度，包括：
    根据当前视频帧中人体胯部的点位计算当前视频帧中人体的腰线；
    根据历史视频帧中人体胯部的点位计算历史视频帧中人体的腰线；
    计算当前视频帧中人体的腰线与历史视频帧中人体的腰线的距离，得到当前视频帧中人体关键点的抖动幅度。
  10. 根据权利要求1至7任一项所述的方法，所述根据所述当前视频帧中人体的点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度之前，还包括：
    采用预设点位平滑算法，根据所述历史视频帧中人体的点位集对所述当前视频帧中人体的点位集进行平滑处理，得到当前视频帧中人体的平滑后点位集；
    所述根据所述当前视频帧中人体的点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度，包括：根据所述当前视频帧中人体的平滑后点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度。
  11. 根据权利要求10所述的方法，所述采用预设点位平滑算法，根据所述历史视频帧中人体的点位集对所述当前视频帧中人体的点位集进行平滑处理，得到当前视频帧中人体的平滑后点位集，包括：
    根据所述历史视频帧中人体的点位集，在时间维度上对所述当前视频帧中人体的点位集进行平滑处理，得到时间平滑结果；
    根据所述历史视频帧中人体的点位集，在空间维度上对所述当前视 频帧中人体的点位集进行平滑处理，得到空间平滑结果；
    根据所述历史视频帧中人体的点位集，在置信度维度上对所述当前视频帧中人体的点位集进行平滑处理，得到置信度平滑结果；
    对所述时间平滑结果、空间平滑结果和置信度平滑结果进行加权运算，得到当前视频帧中人体的平滑后点位集。
  12. 根据权利要求1至7任一项所述的方法，所述调整参考位置为腰线位置；所述当前帧存在多个腰线位置，所述对当前视频帧中调整参考位置对应的人体部分进行调整，并输出调整后的当前视频帧，包括：
    从当前视频帧的多个腰线位置中选择位置最高的腰线位置作为目标腰线位置；
    对当前视频帧中目标腰线位置以下的人体部分进行拉伸，并输出拉伸后的当前视频帧。
  13. 根据权利要求12所述的方法，所述对当前视频帧中目标腰线位置以下的人体部分进行拉伸，并输出拉伸后的当前视频帧包括：
    根据黄金比例对当前视频帧中目标腰线位置以下的人体部分进行拉伸；
    对拉伸后当前视频帧中目标腰线位置部分的斜率进行平滑处理，得到结果帧；
    输出所述结果帧。
  14. 一种视频数据处理装置，包括：
    检测单元，用于从视频流中采集当前视频帧，并对所述当前视频帧中的人体关键点的位置进行检测，得到当前视频帧中人体的点位集；
    获取单元，用于获取历史视频帧中人体的点位集，所述历史视频帧为在所述视频流中，以所述当前视频帧为起点，过去预设时间范围内的视频帧；
    分析单元，用于根据所述当前视频帧中人体的点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度；
    确定单元，用于根据抖动幅度确定所述当前视频帧中人体的调整参考位置；
    拉伸单元，用于对当前视频帧中调整参考位置对应的人体部分进行调整，并输出调整后的当前视频帧。
  15. 根据权利要求14所述的装置，其中，
    所述确定单元，进一步用于根据抖动幅度确定所述当前视频帧位于稳定态时，采用预设固定值确定所述当前视频帧中人体的调整参考位置；根据抖动幅度确定所述当前视频帧位于非稳定态时，根据当前视频帧中人体的点位集计算所述当前视频帧中人体的调整参考位置。
  16. 根据权利要求14或15所述的装置，还包括平滑单元；
    所述平滑单元，用于采用预设点位平滑算法，根据所述历史视频帧中人体的点位集对所述当前视频帧中人体的点位集进行平滑处理，得到当前视频帧中人体的平滑后点位集；
    所述分析单元，具体用于根据所述当前视频帧中人体的平滑后点位集和历史视频帧中人体的点位集分析当前视频帧中人体关键点的抖动幅度。
  17. 一种电子设备，包括：
    处理器；
    与所述处理器相连接的存储器；所述存储器中存储有机器可读指令，所述机器可读指令可以由处理器执行以完成上述权利要求1至13中任一项所述的视频数据处理方法。
  18. 一种非易失性机器可读存储介质，所述存储介质存储有机器可读指令，所述机器可读指令可由处理器执行，以执行权利要求1至13 任一项所述的视频数据处理方法中的步骤。
PCT/CN2019/091340 2018-07-04 2019-06-14 视频数据处理方法、装置和存储介质 Ceased WO2020007183A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/993,061 US11461876B2 (en) 2018-07-04 2020-08-13 Video data processing method and apparatus, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810721857.0A CN108898118B (zh) 2018-07-04 2018-07-04 一种视频数据处理方法、装置和存储介质
CN201810721857.0 2018-07-04

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/993,061 Continuation US11461876B2 (en) 2018-07-04 2020-08-13 Video data processing method and apparatus, and storage medium

Publications (1)

Publication Number Publication Date
WO2020007183A1 true WO2020007183A1 (zh) 2020-01-09

Family

ID=64348176

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/091340 Ceased WO2020007183A1 (zh) 2018-07-04 2019-06-14 视频数据处理方法、装置和存储介质

Country Status (4)

Country Link
US (1) US11461876B2 (zh)
CN (1) CN108898118B (zh)
TW (1) TWI702822B (zh)
WO (1) WO2020007183A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112183309A (zh) * 2020-09-25 2021-01-05 咪咕文化科技有限公司 人脸关键点处理方法、系统、终端及存储介质

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108898118B (zh) * 2018-07-04 2023-04-18 腾讯科技（深圳）有限公司 一种视频数据处理方法、装置和存储介质
CN109583391B (zh) * 2018-12-04 2021-07-16 北京字节跳动网络技术有限公司 关键点检测方法、装置、设备及可读介质
CN109670474B (zh) * 2018-12-28 2023-07-25 广东工业大学 一种基于视频的人体姿态估计方法、装置及设备
CN110047118B (zh) * 2019-04-08 2023-06-27 腾讯科技（深圳）有限公司 视频生成方法、装置、计算机设备及存储介质
CN111985268B (zh) * 2019-05-21 2024-08-06 北京搜狗科技发展有限公司 一种人脸驱动动画的方法和装置
CN110170167B (zh) * 2019-05-28 2023-02-28 上海米哈游网络科技股份有限公司 一种画面显示方法、装置、设备及介质
CN112019868A (zh) * 2019-05-31 2020-12-01 广州虎牙信息科技有限公司 人像分割方法、装置及电子设备
CN110288554B (zh) * 2019-06-29 2022-09-16 北京字节跳动网络技术有限公司 视频美化方法、装置及电子设备
CN111028346B (zh) * 2019-12-23 2023-10-10 北京奇艺世纪科技有限公司 一种视频对象的重建方法和装置
CN111935491B (zh) 2020-06-28 2023-04-07 百度在线网络技术（北京）有限公司 直播的特效处理方法、装置以及服务器
US11961333B2 (en) * 2020-09-03 2024-04-16 Board Of Trustees Of Michigan State University Disentangled representations for gait recognition
CN113873175B (zh) * 2021-09-15 2024-03-15 广州繁星互娱信息科技有限公司 视频播放方法、装置和存储介质及电子设备
CN114494534B (zh) * 2022-01-25 2022-09-27 成都工业学院 基于动作点捕捉分析的帧动画自适应显示方法及系统
CN114821717B (zh) * 2022-04-20 2024-03-12 北京百度网讯科技有限公司 目标对象融合方法、装置、电子设备及存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150185828A1 (en) * 2013-12-27 2015-07-02 Semiconductor Manufacturing International (Beijing) Corporation Wearable intelligent systems and interaction methods thereof
CN105516589A (zh) * 2015-12-07 2016-04-20 凌云光技术集团有限责任公司 基于人脸识别的智能曝光方法及系统
CN107335192A (zh) * 2017-05-26 2017-11-10 深圳奥比中光科技有限公司 运动辅助训练方法、装置及存储装置
CN107704829A (zh) * 2017-10-09 2018-02-16 武汉斗鱼网络科技有限公司 一种人脸关键点追踪方法及应用和装置
CN107734207A (zh) * 2017-09-28 2018-02-23 北京奇虎科技有限公司 视频对象变换处理方法、装置及计算设备
CN107967693A (zh) * 2017-12-01 2018-04-27 北京奇虎科技有限公司 视频关键点处理方法、装置、计算设备及计算机存储介质
CN108898118A (zh) * 2018-07-04 2018-11-27 腾讯科技（深圳）有限公司 一种视频数据处理方法、装置和存储介质

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9165199B2 (en) * 2007-12-21 2015-10-20 Honda Motor Co., Ltd. Controlled human pose estimation from depth image streams
US8520975B2 (en) * 2009-10-30 2013-08-27 Adobe Systems Incorporated Methods and apparatus for chatter reduction in video object segmentation using optical flow assisted gaussholding
JP5582171B2 (ja) * 2012-06-15 2014-09-03 フリュー株式会社 画像処理装置および方法、並びにプログラム
CN104156912B (zh) * 2014-08-18 2018-11-06 厦门美图之家科技有限公司 一种人像增高的图像处理的方法
CN106034203B (zh) * 2015-03-11 2020-07-28 维科技术有限公司 拍摄终端的图像处理方法及其装置
US9582860B2 (en) * 2015-07-24 2017-02-28 Google Inc. Temporally smooth video enhancement
JP6685094B2 (ja) * 2015-07-30 2020-04-22 加賀デバイス株式会社 画像処理装置、画像処理方法及びコンピュータプログラム
WO2017129827A1 (en) * 2016-01-29 2017-08-03 MAX-PLANCK-Gesellschaft zur Förderung der Wissenschaften e.V. Crowdshaping realistic 3d avatars with words
KR20180017978A (ko) * 2016-08-12 2018-02-21 천봉우 중환자용 바지
CN107016357B (zh) * 2017-03-23 2020-06-16 北京工业大学 一种基于时间域卷积神经网络的视频行人检测方法

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150185828A1 (en) * 2013-12-27 2015-07-02 Semiconductor Manufacturing International (Beijing) Corporation Wearable intelligent systems and interaction methods thereof
CN105516589A (zh) * 2015-12-07 2016-04-20 凌云光技术集团有限责任公司 基于人脸识别的智能曝光方法及系统
CN107335192A (zh) * 2017-05-26 2017-11-10 深圳奥比中光科技有限公司 运动辅助训练方法、装置及存储装置
CN107734207A (zh) * 2017-09-28 2018-02-23 北京奇虎科技有限公司 视频对象变换处理方法、装置及计算设备
CN107704829A (zh) * 2017-10-09 2018-02-16 武汉斗鱼网络科技有限公司 一种人脸关键点追踪方法及应用和装置
CN107967693A (zh) * 2017-12-01 2018-04-27 北京奇虎科技有限公司 视频关键点处理方法、装置、计算设备及计算机存储介质
CN108898118A (zh) * 2018-07-04 2018-11-27 腾讯科技（深圳）有限公司 一种视频数据处理方法、装置和存储介质

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112183309A (zh) * 2020-09-25 2021-01-05 咪咕文化科技有限公司 人脸关键点处理方法、系统、终端及存储介质
CN112183309B (zh) * 2020-09-25 2024-05-28 咪咕文化科技有限公司 人脸关键点处理方法、系统、终端及存储介质

Also Published As

Publication number Publication date
TW202007143A (zh) 2020-02-01
TWI702822B (zh) 2020-08-21
US11461876B2 (en) 2022-10-04
CN108898118A (zh) 2018-11-27
US20200372613A1 (en) 2020-11-26
CN108898118B (zh) 2023-04-18

Similar Documents

Publication Publication Date Title
WO2020007183A1 (zh) 视频数据处理方法、装置和存储介质
US12070673B2 (en) Intelligent voice playing method and device
US6360003B1 (en) Image processing apparatus
WO2020063009A1 (zh) 图像处理方法、装置、存储介质及电子设备
CN109194879A (zh) 拍照方法、装置、存储介质及移动终端
CN112163479B (zh) 动作检测方法、装置、计算机设备和计算机可读存储介质
WO2021004285A1 (zh) 人眼闭合程度的确定方法、眼睛控制方法、装置、设备和存储介质
WO2021042364A1 (zh) 拍摄图像的方法和装置
CN111757175A (zh) 视频处理方法及装置
CN110163045A (zh) 一种手势动作的识别方法、装置以及设备
CN107835359A (zh) 一种移动终端的拍照触发方法、移动终端及存储设备
CN112988010B (zh) 图像处理方法、装置、电子设备和存储介质
CN112232258A (zh) 一种信息处理方法、装置及计算机可读存储介质
WO2020108656A1 (zh) 一种图片选择方法及相关设备
CN110163046B (zh) 人体姿态识别方法、装置、服务器及存储介质
CN111953907A (zh) 构图方法、装置及电子设备
CN112887792A (zh) 视频处理方法、装置、电子设备及存储介质
CN115471778A (zh) 转场视频生成方法、装置、电子设备及存储介质
CN111182207B (zh) 图像拍摄方法、装置、存储介质及电子设备
CN114779925A (zh) 一种基于单目标的视线交互方法及装置
CN115376207A (zh) 健身互动方法、装置、电子设备及计算机可读存储介质
CN114140560B (zh) 动画生成方法、装置、设备和存储介质
CN119992398A (zh) 一种多媒体内容生成方法、电子设备及存储介质
CN115639904A (zh) 一种基于人体姿态识别的对象控制方法和电子设备
CN110996149A (zh) 一种信息处理方法、装置和系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19830539

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19830539

Country of ref document: EP

Kind code of ref document: A1