WO2022143078A1

WO2022143078A1 - Video automatic motion compensation method, apparatus, and device, and storage medium

Info

Publication number: WO2022143078A1
Application number: PCT/CN2021/136774
Authority: WO
Inventors: 郑蒙帅; 田超
Original assignee: 深圳创维－Rgb电子有限公司
Priority date: 2020-12-28
Filing date: 2021-12-09
Publication date: 2022-07-07
Also published as: CN112788337A

Abstract

Disclosed are a video automatic motion compensation method, apparatus, and device, and a storage medium. The method comprises: when the frame rate of a video played by a video playing device is greater than a first set threshold, determining, on the basis of the inter-frame correlation between two adjacent frames in each base group of the video, whether a scene transformation occurs between the two adjacent frames; when a scene transformation occurs between the two adjacent frames, acquiring an inter-frame motion speed between the adjacent frames; and performing motion compensation on the video on the basis of the inter-frame motion speed and the inter-frame correlation.

Description

Video automatic motion compensation method, device, device and storage medium

priority information

This application claims the priority of the Chinese patent application filed on December 28, 2020 with application number 202011588510.7, the entire contents of which are incorporated herein by reference.

technical field

The present application relates to the field of video motion compensation, and in particular, to a video automatic motion compensation method, system, device and storage medium.

Background technique

With the development of the television industry, televisions are becoming more and more intelligent and humanized, which is embodied in, for example, that the image modes of televisions are becoming more and more abundant. Now TVs are often set with a variety of image modes, including vivid, standard, sports, cinema, etc. In these modes, some customized parameters will be designed to meet user needs. For example: sharpness, contrast, color, noise reduction, motion compensation, etc. However, these tunable parameters can also be divided into different levels. Such as: strong, medium, low and so on. Taking motion compensation as an example, motion compensation estimates the trajectory of object motion through chips and algorithms, and finally compensates for the picture that is not in the video source itself, so as to achieve a smoother picture. In sports mode, most of them are based on motion pictures, which naturally require strong motion compensation. But in standard mode playback source type is messy.

However, the motion compensation gear in the menu of the existing TV needle cannot be automatically adapted in some pictures. In the fixed mode, only one set of motion compensation parameters is fixed, resulting in too strong or too weak motion compensation parameters in some special pictures, so the picture is broken, jittered, delayed and other problems, affecting the user experience.

The above content is only used to assist the understanding of the technical solutions of the present application, and does not mean that the above content is the prior art.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a video automatic motion compensation method, system, device, and storage medium, which aim to solve the problem that the existing motion compensation method cannot be automatically adapted in some pictures, which affects user experience.

The embodiment of the present application provides a video automatic motion compensation method, which is used in a video playback device, and the method includes:

Obtain the video played by the video playback device;

detecting the frame rate of the video;

When the frame rate is greater than the first set threshold, based on the inter-frame correlation between two adjacent frames in each group of the video, it is judged whether there is a scene change in the two adjacent frames; the The base group consists of the video frame images contained per second;

When a scene change occurs in the two adjacent frames, obtaining the inter-frame motion speed between the adjacent frames in which the scene change occurs in the group; and

The video is motion compensated based on the inter-frame motion speed and the inter-frame correlation.

In some embodiments, the method further includes: when the frame rate is less than or equal to the first set threshold, not performing motion compensation on the two adjacent frames.

In some embodiments, determining whether a scene change occurs in the two adjacent frames based on the inter-frame correlation between two adjacent frames in each group of the video includes:

When the inter-frame correlation of the next frame of video frame image in the two adjacent frames in the group is less than or equal to the second set threshold, then it is judged that the two adjacent frames have scene change;

When the inter-frame correlation of the next video frame image in the two adjacent frames in the group is greater than the second set threshold, it is determined that there is no scene change in the two adjacent frames.

In some embodiments, performing motion compensation on the video based on the inter-frame motion speed and the inter-frame correlation includes:

Sorting the inter-frame motion speed and the inter-frame correlation respectively;

Determine the position of the frame to be inserted based on the sorted inter-frame correlation and the inter-frame motion speed;

obtaining an interpolated frame at the location of the scene change;

Based on the block motion vector between the corresponding macroblocks of two adjacent frames in the group and the interpolation frame, the motion compensation frame interpolation is performed at the position of the frame to be inserted.

In some embodiments, the sorting of the inter-frame motion speed and the inter-frame correlation, respectively, includes:

Obtaining the inter-frame correlation between two adjacent frames in the group in turn, and sorting the obtained inter-frame correlation;

Obtaining the block motion vector between the corresponding macroblocks of two adjacent frames in the group and the inter-block correlation between the corresponding macroblocks between the frames in turn;

The inter-frame motion speed is obtained according to the block motion vector and the inter-block correlation, and the obtained inter-frame motion speed is sorted.

In some embodiments, the determining the position of the frame to be inserted based on the sorted inter-frame correlation and the inter-frame motion speed includes:

Determine the motion speed of the video frame image based on the sorted inter-frame correlation and inter-frame motion speed;

Determine the accuracy of the required inter-frame motion speed according to the similarity between the adjacent inter-frame motion speeds; obtain the motion speed sorting of the video frame images and the accuracy sorting of the inter-frame motion speeds;

The position of the frame to be inserted is determined according to the motion speed sorting and the accuracy sorting.

In some embodiments, the step of performing motion-compensated frame interpolation at the position of the frame to be inserted based on the block motion vector between corresponding macroblocks of two adjacent frames in the group and the interpolation frame further includes: :

Calculate the number of motion-compensated interpolated frames; determine the motion-compensated matching gear based on the number of motion-compensated interpolated frames.

In some embodiments, the motion-compensated matching gear includes a first gear and a second gear; or the motion-compensated matching gear includes a first gear, a second gear, and a third gear.

In some embodiments, the similarity between the adjacent inter-frame motion speeds is calculated by the following formula:

i represents any frame in the group, v _i , v _i+1 represent the respective motion speeds of two adjacent frames, M represents the total number of frames in a group, S _i,i+1 represent the adjacent frames Similarity between motion speeds.

In some embodiments, the inter-frame correlation is the correlation between two adjacent frames in the group, and is calculated by the following formula:

t=1,2,...,M-1;

γ _t = |ρ _t |

Among them, f _t and f _t+1 respectively represent two adjacent frames in a group; E[f _t ] and D[f _t ] represent the mean and variance of frame pixel values, respectively; ρ _t represents the adjacent frame in the group The degree of correlation between two frames, the two adjacent frames include positive or negative numbers; γ _t represents the absolute value of ρ _t ; M represents the total number of frames in a group.

In some embodiments, frame interpolation is performed using bidirectional overlapping block motion compensation.

In some embodiments, the block motion vector between corresponding macroblocks of two adjacent frames in the group and the inter-block correlation between corresponding macroblocks between frames are sequentially obtained, and the block motion vector The inter-frame motion speed is obtained in combination with the inter-block correlation degree, and the obtained inter-frame motion speed is sorted, including:

Motion estimation is performed on two adjacent frames f _t and f _t+1 by using the diamond search method;

Perform macroblock division on two adjacent frames f _t and f _t+1 , and calculate the absolute value of the relative coefficient between the corresponding macroblocks as the inter-block correlation;

Based on the block motion vector and the inter-block correlation between the corresponding macroblocks, adopt the tail-pinching method to remove the block motion vector of the macroblock whose inter-block correlation is greater than 0.9;

Calculate the average value of the block motion vectors of the remaining macroblocks in the adjacent two frames, as the motion speed vt between the adjacent two frames f _t and f _t+1 , t=1,2,...,M-1; where f _t and f _t+1 respectively represent two adjacent frames in a group; M represents the total number of frames in a group;

Repeat the above three steps for M frames in the base group in turn to obtain M-1 inter-frame motion speeds;

Sort the obtained motion speed between frames in descending order.

The present application also proposes a video automatic motion compensation device, the device comprising:

a first acquisition module, used to acquire the video played by the video playback device;

a detection module for detecting the frame rate of the video;

A judgment module, configured to judge whether the two adjacent frames appear based on the inter-frame correlation between the adjacent two frames in each group of the video when the frame rate is greater than the first set threshold Scene change; the group is composed of video frame images contained per second;

a second acquiring module, configured to acquire the inter-frame motion speed between the adjacent frames in which the scene transition occurs in the group when the scene transition occurs in the two adjacent frames; and

A motion compensation module, configured to perform motion compensation on the video based on the inter-frame motion speed and the inter-frame correlation.

The present application also proposes a video automatic motion compensation device, the device includes a processor, a memory electrically connected to the processor, and a video automatic motion compensation program stored on the memory and executable on the processor ; the steps of implementing the video automatic motion compensation method when the video automatic motion compensation program is executed by the processor.

The present application also proposes a computer-readable storage medium, where one or more programs are stored in the computer-readable storage medium, and the one or more programs can be executed by one or more processors to realize the video Steps in an automatic motion compensation method.

In this embodiment of the present application, the video played by the video playback device is acquired; the frame rate of the video is then detected, and if the frame rate is greater than the first set threshold, the frame rate of each group of the video is The inter-frame correlation between two adjacent frames, to determine whether scene changes occur in the two adjacent frames; the base group is composed of video frame images included per second; if scene changes occur in the two adjacent frames, Then, the inter-frame motion speed between adjacent frames in which scene change occurs in the group is acquired, and motion compensation is performed on the video based on the inter-frame motion speed and the inter-frame correlation. In this embodiment, the speed of the motion of the object is judged based on the combination of the inter-frame motion speed and the inter-frame correlation, and the motion compensation gear is automatically adapted according to the number of frames inserted through motion compensation. The present application matches the corresponding motion compensation matching gear according to whether the picture is in a motion state and the degree of the motion state. When the TV is used to watch sports content, the appropriate motion compensation gear is automatically matched to improve the viewer's experience.

Description of drawings

1 is a hardware framework diagram of an embodiment of an automatic video motion compensation device of the present application;

2 is a flowchart of an embodiment of a video automatic motion compensation method of the present application;

FIG. 3 is a flowchart of an embodiment of step 300 in FIG. 2;

FIG. 4 is a flowchart of an embodiment of step 340 in FIG. 3;

FIG. 5 is a flowchart of another embodiment of step 300 in FIG. 2;

6 is a flowchart of another embodiment of the automatic video motion compensation method of the present application;

FIG. 7 is a schematic block diagram of an embodiment of the video automatic motion compensation apparatus of the present application;

Fig. 8a and Fig. 8b respectively represent the schematic diagrams of the large diamond and the small diamond in the diamond search method in this application;

FIG. 9 shows a schematic diagram of bidirectional overlapping block motion compensation in the present application.

Detailed ways

For better understanding of the above technical solutions, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thoroughly understood, and will fully convey the scope of the present disclosure to those skilled in the art.

Therefore, it is currently necessary to solve the problem that the existing motion compensation method cannot automatically adapt to some pictures, which affects the user experience. In view of this, the present application proposes a video automatic motion compensation method, system, device and storage medium.

Referring to FIG. 1 , an automatic video motion compensation device is described below. The automatic video motion compensation device may include: a processor 1001 , such as a CPU, a memory 1005 , a user interface 1003 , a network interface 1004 , and a communication bus 1002 . Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include an input unit such as a keyboard, and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a wireless interface (eg, a WI-FI interface). The memory 1005 may be high-speed RAM memory, or may be non-volatile memory, such as disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001 .

Those skilled in the art can understand that the structure of the automatic video motion compensation device shown in FIG. 1 does not constitute a limitation on the automatic video motion compensation device, and may include more or less components than those shown in the figure, or combine some components, or Different component arrangements.

As shown in FIG. 1 , the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module and a video automatic motion compensation program. The operating system is a program that manages and controls the hardware and software resources of the automatic video motion compensation device, and supports the operation of the automatic video motion compensation program and other software or programs.

In the video automatic motion compensation device shown in FIG. 1, the network interface 1004 is mainly used for the background server, and performs data communication with the background server; the processor 1001 can be used to call the video automatic motion compensation program stored in the memory 1005, and execute the above The steps of the video automatic motion compensation method.

In one embodiment, the processor 1001 may be configured to call the video automatic motion compensation program stored in the memory 1005, and execute the method including:

Obtain the video played by the video playback device;

detecting the frame rate of the video;

It also includes: when the frame rate is less than or equal to the first set threshold, not performing motion compensation on the two adjacent frames.

Described based on the inter-frame correlation between two adjacent frames in each group of the video, judging whether there is a scene change in the two adjacent frames, including:

obtaining an interpolated frame at the location of the scene change;

The determining the position of the frame to be inserted based on the sorted inter-frame correlation and the inter-frame motion speed includes:

The step of performing motion-compensated frame interpolation at the position to be inserted based on the block motion vector between the corresponding macroblocks of two adjacent frames in the group and the interpolation frame further includes:

The motion-compensated matching gear includes a first gear and a second gear; or the motion-compensated matching gear includes a first gear, a second gear, and a third gear.

In one embodiment, the processor 1001 may be configured to call the video automatic motion compensation program stored in the memory 1005, and the execution of the method includes: wherein, the similarity between the adjacent inter-frame motion speeds is determined by: Calculated by the following formula:

In one embodiment, the processor 1001 may be configured to call the video automatic motion compensation program stored in the memory 1005, and executing the method includes: wherein, the inter-frame correlation is between two adjacent frames in the group. The degree of correlation is calculated by the following formula:

t=1,2,...,M-1;

γ _t = |ρ _t |

In one embodiment, the processor 1001 may be configured to call the video automatic motion compensation program stored in the memory 1005, and the execution of the method includes: wherein, frame interpolation is performed using bidirectional overlapping block motion compensation.

In one embodiment, the processor 1001 may be configured to call the video automatic motion compensation program stored in the memory 1005, and executing the method includes: wherein the sequentially obtaining macroblocks corresponding to two adjacent frames in the group The inter-block correlation between the block motion vector and the inter-frame corresponding macroblock, the block motion vector and the inter-block correlation are combined to obtain the inter-frame motion speed, and the obtained inter-frame motion speed Sort by including:

Sort the obtained motion speed between frames in descending order.

The video automatic motion compensation device of the present application obtains the video played by the video playback device; then detects the frame rate of the video, and when the frame rate is greater than the first set threshold The inter-frame correlation between two adjacent frames in the group, to determine whether scene changes occur in the two adjacent frames; the base group is composed of video frame images contained in every second; when the two adjacent frames appear scene changes is obtained, the inter-frame motion speed between adjacent frames where scene changes occur in the group are acquired, and motion compensation is performed on the video based on the inter-frame motion speed and the inter-frame correlation. In this embodiment, the speed of the motion of the object is judged based on the combination of the inter-frame motion speed and the inter-frame correlation, and the motion compensation gear is automatically adapted according to the number of frames inserted through motion compensation. The present application matches the corresponding motion compensation matching gear according to whether the picture is in a motion state and the degree of the motion state. When the TV is used to watch sports content, the appropriate motion compensation gear is automatically matched to improve the viewer's experience.

Please refer to FIG. 2 , based on the hardware architecture of the above-mentioned automatic video motion compensation equipment, the first embodiment of the automatic video motion compensation method of the present application is proposed below. The automatic video motion compensation method is used in a video playback device, and the automatic video motion compensation method includes the following steps:

S100. Acquire the video played by the video playing device.

Specifically, the playback device includes various devices having the function of playing video, such as a playback device, a TV, a computer, a smart phone, a tablet computer, etc. This embodiment mainly takes a TV as an example for illustration.

S200. Detect the frame rate of the video.

S300, when the frame rate is greater than the first set threshold, then based on the inter-frame correlation between two adjacent frames in each group of the video, determine whether there is a scene change in the two adjacent frames; The population consists of video frame images contained per second.

Specifically, the frame rate of the video can be obtained from the configuration file of the video, where the frame rate refers to the number of video frame images displayed by the TV per second. The base group consists of video frame images contained per second. For example, the frame rate of the video is 24FPS, which means that the displayed video frame image is 24 frames per second. Then 24 frames of video frame images form a group.

If it is determined that the frame rate is greater than the first set threshold, in order to improve the fluency of the displayed video, motion compensation should be performed on the video. The first set threshold here can be set according to the actual situation. For example, the first set threshold can be set to 24 frames/second. If it is detected that the frame rate of the video played by the playback device is higher than 24 frames/second, it is further based on The inter-frame correlation between two adjacent frames in each group of the video is used to determine whether a scene change occurs in the two adjacent frames. It should be understood that the specific numerical values of the first set threshold listed here are only for those skilled in the art to better understand the technical solutions of the embodiments of the present application, and should not be construed as limitations on the first set thresholds of this embodiment.

It should be noted that the inter-frame correlation is used to describe the degree of correlation between two adjacent frames in a group, that is, the inter-frame correlation determines whether a scene change occurs in this group. For example, the video frame images of the previous frame and the next frame are different, it is considered that a scene change has occurred, and the previous frame is similar to the next frame, and the scene is considered to be in no scene change. The inter-frame correlation is shown in Equation 1:

Formula 1:

In the formula, f _t and f _t+1 represent two adjacent frames in a group, E[f _t ] and D[f _t ] represent the mean and variance of frame pixel values, respectively; ρ _t represents the adjacent frame in the group The degree of correlation between two frames, including positive or negative numbers; γ _t represents the absolute value of ρ _t ; M represents the total number of frames in a group.

It should be understood that f _t and f t+1 appearing below have the same meaning as f _t and f _t ₊₁ in this paragraph, and both represent two adjacent frames in a group.

Specifically, in some embodiments, step S300 specifically includes:

S310. When the inter-frame correlation of the next video frame image in the two adjacent frames in the group is less than or equal to the second set threshold, determine that the two adjacent frames have scene change.

S320. When the inter-frame correlation of the video frame image of the next frame in the two adjacent frames in the group is greater than the second set threshold, determine that there is no scene change in the two adjacent frames.

The smaller the correlation between frames, the greater the difference between the previous frame and the next frame, which further indicates that the picture in the video is in a rapidly changing motion state, and a scene change occurs at this time. It is necessary to perform motion compensation on the video to enrich and compensate the pictures that are not in the video source itself, so as to achieve the purpose of smoother pictures. If the inter-frame correlation is relatively large, it means that no scene change has occurred.

It can be understood that the value of the inter-frame correlation degree γ _t varies in the range of 0 to 1, and γ _t = 0-0.3 indicates that the correlation degree of two adjacent frames is low, that is, a scene change may have occurred; the inter-frame correlation degree γ _t = 0.3-0.9 means that the degree of correlation between two adjacent frames is normal, that is, it is possible that the motion speed between the two frames is relatively large; the degree of inter-frame correlation γ _t = 0.9-1.0 means that the degree of correlation between two adjacent frames is extremely high, that is, it is possible that the two frames are closely related. The movement speed is relatively small or no movement. Therefore, the second set threshold can be set to T=0.3. It should also be understood that the specific values of the second set threshold listed here are only for those skilled in the art to better understand the technical solutions of the embodiments of the present application, and should not be understood as Restriction on the second set threshold of this embodiment.

S400. When a scene change occurs in the two adjacent frames, obtain the inter-frame motion speed between the adjacent frames in the group where the scene change occurs.

S500. Perform motion compensation on the video based on the inter-frame motion speed and the inter-frame correlation.

When a scene change occurs, in order to further know the change speed of the TV playing picture, the inter-frame motion speed between adjacent frames in which the scene change occurs is further obtained. When the rate of change between frames is faster, it proves that the picture is in a state of motion, to prevent frame loss of the picture, and it is necessary to insert frames for motion compensation. The video is motion compensated based on the inter-frame motion speed and the inter-frame correlation.

Specifically, in some embodiments, referring to FIG. 3 , step S500 further includes:

S510. Sort the inter-frame motion speed and the inter-frame correlation degree respectively.

Step S510 specifically includes:

S511. Acquire the inter-frame correlations between two adjacent frames in the group in sequence, and sort the inter-frame correlations.

Specifically, the inter-frame correlations within the group obtained in step S200 are sorted.

S512. Obtain block motion vectors between corresponding macroblocks of two adjacent frames in the group and the inter-block correlation between corresponding macroblocks between frames in sequence.

S513: Acquire the inter-frame motion speed according to the block motion vector and the inter-block correlation degree, and sort the acquired inter-frame motion speed.

In some embodiments, specifically, step S513 includes the following subdivision steps:

S5131. Perform motion estimation on two adjacent frames f _t and f _t+1 , and adopt a diamond search method with better performance in block matching search. The search pattern of the diamond search method is neither too large nor too small. The large diamond (as shown in Figure 8a) has a large search step, which can be used for rough positioning, so that the search process will not fall into local optimum, and the small diamond (as shown in Figure 8b) ) is used for accurate positioning to obtain more precise matching points. The block matching algorithm obtains the block motion vector between corresponding macroblocks between two adjacent frames.

S5132. Perform macroblock division on two adjacent frames f _t and f _t+1 , and calculate the absolute value of the relative coefficient between the corresponding macroblocks as the inter-block correlation, the calculation formula and meaning of which are the same as the inter-frame correlation.

S5133. Based on the block motion vector and the inter-block correlation between the corresponding macroblocks, adopt a tail pinch method to remove the block motion vector of the macroblock whose inter-block correlation is greater than 0.9.

S5134. Calculate the mean value of the block motion vectors of the remaining macroblocks in the adjacent two frames, as the motion speed vt between the adjacent two frames f _t and f _t+1 , t=1, 2, . . . , M-1; wherein f _t and f _t+1 respectively represent two adjacent frames in a group; M represents the total number of frames in a group.

S5135: Repeat the above three steps for M frames in the base group in sequence, and finally obtain M-1 inter-frame motion speeds.

S5136. Sort the obtained motion speed between frames in descending order.

S520. Determine the position of the frame to be inserted based on the sorted inter-frame correlation and the inter-frame motion speed.

In some embodiments, specifically, referring to FIG. 4 , step S520 includes the following subdivision steps:

S521. Determine the motion speed of the video frame image based on the sorted inter-frame correlation and the inter-frame motion speed.

S522. Determine the accuracy of the obtained inter-frame motion speed according to the similarity between adjacent inter-frame motion speeds.

S523: Acquire the motion speed ranking of the video frame images and the accuracy ranking of the motion speed between frames.

S524. Determine the position of the frame to be inserted according to the motion speed sorting and the accuracy sorting.

In order to improve the accuracy of the positions of the frames to be inserted, and to determine the positions of all the frames to be inserted, the determination of the positions of the frames to be inserted can be realized in two steps. First, combine the sorted inter-frame correlation and inter-frame motion speed, and select two frames with fast inter-frame speed and small inter-frame correlation as the frame insertion position; If the requirements are met, go to the next step directly, otherwise continue to select the frame insertion position: calculate the similarity S between the motion speeds between two adjacent frames, as shown in the following formula 2. From the two inter-frame motion speeds v _i , v _i+1 with larger similarity S _i,i+ 1, select the two frames corresponding to the larger speed to perform frame insertion until the number of frame insertion positions reaches the requirement.

Formula 2:

_i represents any frame in the group, vi and vi ₊₁ represent the respective motion speeds of two adjacent frames, M represents the total number of frames in a group, and S _i,i+1 represent the adjacent frames Similarity between motion speeds.

S530. Obtain an interpolated frame at the position of the scene transition.

When the correlation between ft and ft ₊₁ _{is γ t} _≤T (ie, the second set threshold), it is considered that scene detection occurs between the two frames, and the interpolated frame is obtained by copying the previous frame.

S540. Based on the block motion vector between the corresponding macroblocks of two adjacent frames in the group and the interpolation frame, perform motion compensation frame interpolation at the position of the frame to be inserted.

The position of the frame to be inserted obtained in the above-mentioned step S330 is inserted, and in order to effectively solve the block effect and the hole problem caused by the block matching motion estimation, the bidirectional overlapping block motion compensation commonly used in the double frame rate improvement is used to insert the frame, such as As shown in FIG. 9, FIG. 9 shows a schematic diagram of bidirectional overlapping block motion compensation, in which the solid line represents the N×N original matching, B represents the current block, and N1-N8 represent the surrounding blocks of the current block; the dotted line block is the above nine The blocks are (N+2w)×(N+2w) extended blocks formed by extending the center by w widths in the upward, downward, left, and right directions respectively.

According to the relationship between the original frame rate M and the improved frame rate N, frames can be inserted in two cases, as follows:

a. When M<N<2*M, for example, M=15, N=24, that is, the video of 15 frames/second is increased to 24 frames/second, and 9 frame insertion positions need to be found at this time. Follow the above steps to find them in turn, and then insert a frame at the position of the frame insertion by using the frame insertion method commonly used in double boosting.

b. When 2*M<N, for example, M=24, N=60, that is, the video of 24 frames/second is increased to 60 frames/second, and 36 frames need to be inserted at this time. In this case, still follow the above steps to find 12 (N-2*M) interpolated frame positions, and then insert two interpolated frames at the found interpolated frame positions by the common interpolated frame method in triple boosting, and An interpolated frame is inserted between other adjacent frames with a double boosting method.

Referring to FIG. 5 , after step 540, it further includes:

S550. Calculate the number of motion compensation interpolation frames; determine the motion compensation matching gear based on the number of motion compensation interpolation frames.

Specifically, the motion compensation matching gear includes a second gear consisting of a first gear and a second gear, or a third gear consisting of a first gear, a second gear and a third gear. Specifically, the first gear may be a high gear, corresponding to a larger number of interpolated frames; the second gear may be a low gear, corresponding to a smaller number of inserted frames. The high level indicates that more interpolated frames are required to perform motion compensation on the video picture than the low level. In order to enrich the matching gears, the matching gears can also be set to the first gear, the second gear and the third gear. At this time, the first gear may be a high gear, the second gear may be a middle gear, and the second gear may be a low gear. As shown in Table 1 below:

number of frames

5

10

20


匹配档位match gear	低Low	中middle	高high

Table 1

Of course, the present application can also use the traditional frame interpolation method to determine the frame interpolation position according to the weight parameter of each frame in the group, and obtain the interpolation frame by linearly summing two adjacent frames.

The present application obtains the video played by the video playback device; and then detects the frame rate of the video, when the frame rate is greater than the first set threshold, then based on the two adjacent frames in each group of the video The inter-frame correlation between the two adjacent frames is used to determine whether there is a scene change in the two adjacent frames; the group is composed of video frame images contained in every second; when the two adjacent frames have scene changes, the The inter-frame motion speed between adjacent frames where scene changes occur in the group, and motion compensation is performed on the video based on the inter-frame motion speed and the inter-frame correlation. In this embodiment, the speed of the motion of the object is judged based on the combination of the inter-frame motion speed and the inter-frame correlation, and the motion compensation gear is automatically adapted according to the number of frames inserted through motion compensation. The present application matches the corresponding motion compensation matching gear according to whether the picture is in a motion state and the degree of the motion state. When the TV is used to watch sports content, the appropriate motion compensation gear is automatically matched to improve the viewer's experience.

Based on the same inventive concept, please refer to FIG. 6 , the present application also proposes a second embodiment, which is based on the first embodiment.

Embodiment 2

The video automatic motion compensation method of this embodiment includes:

S100, acquiring the video played by the video playback device;

S200, detecting the frame rate of the video;

S300, when the frame rate is greater than the first set threshold, then based on the inter-frame correlation between two adjacent frames in each group of the video, determine whether there is a scene change in the two adjacent frames; The group consists of video frame images contained per second;

S330. When the frame rate is less than or equal to the first set threshold, do not perform motion compensation on the two adjacent frames.

When the frame rate is less than or equal to the first set threshold, for example, less than 24 frames per second, at this time, due to the long interval between adjacent frames, it is easy to add video frames that did not appear in the original video if frame interpolation is performed. Causes the picture to be broken, further aggravates the possibility of screen jitter, delay and other problems, affecting the user experience. Therefore, when the video frame rate is less than the first set threshold, motion compensation is not performed. This step can avoid frame insertion when the video is at a low frame rate, prevent problems such as screen breakage, jitter, and delay, and improve user experience.

The specific implementation principles and manners of the above steps S100 to S300 are the same as those of the steps S100 to S300 in the embodiment corresponding to FIG. 2 . For details, please refer to the descriptions of the steps S100 to S300 .

Please refer to FIG. 7 , the present application also proposes a video automatic motion compensation device, including: an acquisition module 101, a detection module 102, a frame insertion module 103, a second acquisition module 104 and a motion compensation module 105, wherein:

The first acquisition module 101 is used to acquire the video played by the video playback device;

The detection module 102 is used to detect the frame rate of the video;

The judging module 103 is configured to judge whether the two adjacent frames appear based on the inter-frame correlation between the adjacent two frames in each group of the video when the frame rate is greater than the first set threshold Scene change; the group is composed of video frame images contained per second;

The second acquiring module 104 is configured to acquire the inter-frame motion speed between the adjacent frames in which the scene transition occurs in the group when the scene transition occurs in the two adjacent frames; and

The motion compensation module 105 is configured to perform motion compensation on the video based on the inter-frame motion speed and the inter-frame correlation.

Wherein, for the steps implemented by each functional module of the video automatic motion compensation device, reference may be made to the various embodiments of the video automatic motion compensation method of the present application, which will not be repeated here.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

It should be noted that, in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not preclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. do not denote any order. These words can be interpreted as names.

While the preferred embodiments of the present application have been described, additional changes and modifications to these embodiments may occur to those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be construed to include the preferred embodiment and all changes and modifications that fall within the scope of this application.

Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

A video automatic motion compensation method, applied to a video playback device, includes:

Obtain the video played by the video playback device;

detecting the frame rate of the video;

When the frame rate is greater than the first set threshold, based on the inter-frame correlation between two adjacent frames in each group of the video, it is judged whether there is a scene change in the two adjacent frames; the The base group consists of the video frame images contained per second;

When a scene change occurs in the two adjacent frames, obtaining the inter-frame motion speed between the adjacent frames in which the scene change occurs in the group; and

The video is motion compensated based on the inter-frame motion speed and the inter-frame correlation.
The method according to claim 1, further comprising: when the frame rate is less than or equal to the first set threshold, not performing motion compensation on the two adjacent frames.
The method according to claim 1, wherein the determining whether a scene change occurs in the two adjacent frames based on the inter-frame correlation between two adjacent frames in each group of the video comprises:

When the inter-frame correlation of the next frame of video frame image in the two adjacent frames in the group is less than or equal to the second set threshold, then it is judged that the two adjacent frames have scene change;

When the inter-frame correlation of the next video frame image in the two adjacent frames in the group is greater than the second set threshold, it is determined that there is no scene change in the two adjacent frames.
The method according to claim 1, wherein the performing motion compensation on the video based on the inter-frame motion speed and the inter-frame correlation comprises:

Sorting the inter-frame motion speed and the inter-frame correlation respectively;

Determine the position of the frame to be inserted based on the sorted inter-frame correlation and the inter-frame motion speed;

obtaining an interpolated frame at the location of the scene change;

Based on the block motion vector between the corresponding macroblocks of two adjacent frames in the group and the interpolation frame, the motion compensation frame interpolation is performed at the position of the frame to be inserted.
The method according to claim 4, wherein the sorting of the inter-frame motion speed and the inter-frame correlation respectively comprises:

Obtaining the inter-frame correlation between two adjacent frames in the group in turn;

Sort the obtained inter-frame correlation;

Obtaining the block motion vector between the corresponding macroblocks of two adjacent frames in the group and the inter-block correlation between the corresponding macroblocks between the frames in turn;

The inter-frame motion speed is obtained according to the block motion vector and the inter-block correlation, and the obtained inter-frame motion speed is sorted.
The method according to claim 4, wherein the determining the position of the frame to be inserted based on the sorted inter-frame correlation and the inter-frame motion speed comprises:

Determine the motion speed of the video frame image based on the sorted inter-frame correlation and inter-frame motion speed;

Determine the accuracy of the requested inter-frame motion speed according to the similarity between adjacent inter-frame motion speeds;

Obtain the motion speed sorting of video frame images and the accuracy sorting of motion speed between frames;

The position of the frame to be inserted is determined according to the motion speed sorting and the accuracy sorting.
The method according to claim 4, wherein the motion-compensated frame interpolation is performed at the position of the frame to be inserted based on the block motion vector between corresponding macroblocks of two adjacent frames in the group and the interpolation frame The steps also include:

Calculate the number of motion-compensated frames;

A motion-compensated matching gear is determined based on the number of motion-compensated interpolation frames.
8. The method of claim 7, wherein the motion-compensated matching gear includes a first gear and a second gear; or the motion-compensated matching gear includes a first gear, a second gear, and a third gear The three-level gear is composed of gears.
The method according to claim 6, wherein the similarity between the adjacent inter-frame motion speeds is calculated by the following formula:

i represents any frame in the group, v i , v i+1 represent the respective motion speeds of two adjacent frames, M represents the total number of frames in a group, S i,i+1 represent the adjacent frames Similarity between motion speeds.
The method according to claim 1, wherein the inter-frame correlation is the correlation between two adjacent frames in the group, and is calculated by the following formula:

t=1,2,...,M-1;

γ t = |ρ t |

Among them, f t and f t+1 respectively represent two adjacent frames in a group; E[f t ] and D[f t ] represent the mean and variance of frame pixel values, respectively; ρ t represents the adjacent frame in the group The degree of correlation between two frames, the two adjacent frames include positive or negative numbers; γ t represents the absolute value of ρ t ; M represents the total number of frames in a group.
5. The method of claim 4, wherein frame interpolation is performed using bidirectional overlapping block motion compensation.
The method according to claim 5, wherein the step of sequentially obtaining the block motion vector between corresponding macroblocks of two adjacent frames in the group and the inter-block correlation between corresponding macroblocks between frames, according to the Obtain the inter-frame motion speed from the block motion vector and the inter-block correlation degree, and sort the acquired inter-frame motion speed, including:

Motion estimation is performed on two adjacent frames f t and f t+1 by using the diamond search method;

Perform macroblock division on two adjacent frames f t and f t+1 , and calculate the absolute value of the relative coefficient between the corresponding macroblocks as the inter-block correlation;

Based on the block motion vector and the inter-block correlation between the corresponding macroblocks, adopt the tail-pinching method to remove the block motion vector of the macroblock whose inter-block correlation is greater than 0.9;

Calculate the average value of the block motion vectors of the remaining macroblocks in the adjacent two frames, as the motion speed vt between the adjacent two frames f t and f t+1 , t=1,2,...,M-1; where f t and f t+1 respectively represent two adjacent frames in a group; M represents the total number of frames in a group;

Repeat the above three steps for M frames in the base group in turn to obtain M-1 inter-frame motion speeds;

Sort the obtained inter-frame motion speeds in descending order.
A video automatic motion compensation device, comprising:

a first acquisition module, configured to acquire the video played by the video playback device;

a detection module for detecting the frame rate of the video;

A judgment module, configured to judge whether the two adjacent frames appear based on the inter-frame correlation between the adjacent two frames in each group of the video when the frame rate is greater than the first set threshold scene change; the group consists of video frame images contained per second;

a second acquiring module, configured to acquire the inter-frame motion speed between the adjacent frames in which the scene transition occurs in the group when the scene transition occurs in the two adjacent frames; and

A motion compensation module, configured to perform motion compensation on the video based on the inter-frame motion speed and the inter-frame correlation.
A video automatic motion compensation device, comprising a processor, a memory electrically connected to the processor, and a video automatic motion compensation program stored on the memory and running on the processor; the video automatic motion The compensation program, when executed by the processor, implements the steps of the video automatic motion compensation method as claimed in any one of claims 1 to 12.
A computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to realize the invention as claimed in claims 1 to 12 The steps in any one of the video automatic motion compensation methods.