CN116193279A

CN116193279A - Video processing method, device, computer equipment and storage medium

Info

Publication number: CN116193279A
Application number: CN202211705293.4A
Authority: CN
Inventors: 曲超
Original assignee: Insta360 Innovation Technology Co Ltd
Current assignee: Insta360 Innovation Technology Co Ltd
Priority date: 2022-12-29
Filing date: 2022-12-29
Publication date: 2023-05-30

Abstract

The application relates to a video processing method, a video processing device, computer equipment and a storage medium. The method comprises the following steps: determining a first motion mask between adjacent video frames in a plurality of video frames of a video; determining a first motion area and a second motion area of adjacent video frames according to the first motion mask, and calculating optical flow between the first motion area and the second motion area; determining pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the first motion area and the optical flow; the candidate points are determined based on the selected pixel points of the optical flow when the optical flow is greater than a first threshold value; and selecting the pixel point with the largest pixel value from the pixel points corresponding to the candidate points and the pixel points corresponding to the positions in the video frames as a target pixel point, and according to the position corresponding to the target pixel point and the pixel value of the target video frame. The method has high processing speed and can realize better long exposure time delay photographic effect.

Description

Video processing method, device, computer equipment and storage medium

Technical Field

The present invention relates to the field of image processing, and in particular, to a video processing method, apparatus, computer device, and storage medium.

Background

Along with the improvement of the living standard of people, the requirements of people on video quality are higher and higher, and the video can be expected to show various cool and dazzling special effects, for example, the long-exposure time-delay shooting effect can be realized through the post-treatment of the tail special effect of the car lamp on the normally shot car flow video. In order to achieve the effect of long exposure time delay photography, at present, a mode is generally adopted that an original video is firstly subjected to frame insertion, then a frame stack is used for tailing, but a large amount of time is consumed for video frame insertion, and meanwhile, the memory space occupied by storing video frames after frame insertion before frame stacking is also increased sharply, so that the video processing efficiency is low and the effect is poor.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a video processing method, apparatus, computer device, and computer-readable storage medium that are fast in processing speed and capable of achieving a good long-exposure time-lapse photography effect.

The application provides a video processing method. The method comprises the following steps:

determining a first motion mask between adjacent video frames in a plurality of video frames of the video;

determining a first motion region and a second motion region of the adjacent video frames according to the first motion mask, and calculating optical flow between the first motion region and the second motion region;

Determining pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the first motion area and the optical flow; the candidate points are determined based on the selected pixel points of the optical flow when the optical flow is larger than a first threshold value;

and selecting the pixel point with the largest pixel value from the pixel points corresponding to the candidate points and the pixel points corresponding to the positions in the video frames as a target pixel point, and according to the position corresponding to the target pixel point and the pixel value of the target video frame.

In one embodiment, before determining the first motion mask between adjacent video frames in the plurality of video frames of the video, the method further comprises:

performing reduction processing on adjacent video frames in a plurality of video frames of the video to obtain reduced adjacent video frames;

carrying out gray scale processing on the reduced adjacent video frames to obtain adjacent gray scale images;

the determining a first motion mask between adjacent video frames in a plurality of video frames of the video comprises:

in a plurality of video frames of the video, performing difference processing on gray values corresponding to pixel points in adjacent gray images to obtain gray difference values; obtaining a difference gray scale map based on each gray scale difference;

And moving the sliding frame in the difference gray level diagram according to a target step length, counting the number of pixels with gray level values larger than a preset gray level value in the sliding frame in the process of moving the sliding frame each time, assigning the pixels in the sliding frame to be a first gray level value when the counted number of the pixels is larger than the preset number, and assigning the pixels in the sliding frame to be a second gray level value when the counted number of the pixels is smaller than or equal to the preset number until the assignment of the pixels in each region in the difference gray level diagram is completed, thereby obtaining a first motion mask.

In one embodiment, the calculating the optical flow between the first and second regions of motion comprises:

determining a target moving object according to the first moving area and the second moving area;

determining a pixel point of the target moving object in the first moving area, and determining a pixel point of the target moving object in the second moving area;

a plurality of optical flows are determined based on the pixels of the target moving object in the first motion region and the pixels in the second motion region.

In one embodiment, before determining the pixel point corresponding to each candidate point according to the pixel point corresponding to the optical flow in the first motion area and the optical flow, the method further includes:

when the optical flow is greater than a first threshold, taking the first threshold as an interpolation frame number;

equally dividing the optical flow according to the number of the inserted frames to obtain an equal value;

and determining candidate points based on the equivalent value and the pixel points corresponding to the optical flow in the first motion area.

In one embodiment, the method further comprises:

when the optical flow is greater than or equal to a second threshold value and is not greater than the first threshold value, determining a candidate point according to the optical flow and the pixel points corresponding to the optical flow in the first motion area;

determining a pixel point corresponding to the candidate point in the second motion area;

and selecting the pixel point with the largest pixel value from the pixel points corresponding to the candidate points and the pixel points at the corresponding positions in the video frames as a target pixel point, and according to the position and the pixel value corresponding to the target pixel point, selecting the pixel point with the largest pixel value as the target pixel point, and selecting the pixel point with the largest pixel value as the target pixel point according to the position and the pixel value corresponding to the target pixel point.

In one embodiment, after obtaining the target video frame according to the position and the pixel value corresponding to the target pixel point, the method further includes:

Obtaining the tailing quantity;

and selecting each adjacent video frame with the tailing number from the video frames, updating the pixel point at the target position in the target video frame, and obtaining the updated target video frame when the updating is completed.

when the tailing number is larger than the acceleration multiple, determining a frame number difference value according to the tailing number and the acceleration multiple;

determining a first pixel point of a target video frame in a plurality of video frames according to the frame number difference value;

acquiring a plurality of new video frames with the frame number equal to the acceleration multiple, and determining a second pixel point in the new video frames;

selecting the pixel point with the largest pixel value from the determined first pixel point and the determined second pixel point as a replacement pixel point;

and updating the pixel point at the target position in the next target video frame according to the replaced pixel point.

The application provides a video processing device. The device comprises:

a first determining module, configured to determine a first motion mask between adjacent video frames among a plurality of video frames of the video;

A determining and calculating module, configured to determine a first motion region and a second motion region of the adjacent video frame according to the first motion mask, and calculate an optical flow between the first motion region and the second motion region;

the second determining module is used for determining the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the first motion area and the optical flow; the candidate points are determined based on the selected pixel points of the optical flow when the optical flow is larger than a first threshold value;

the selecting and updating module is used for selecting the pixel point with the largest pixel value from the pixel points corresponding to the candidate points and the pixel points corresponding to the positions in the video frames as a target pixel point, and selecting the target video frame according to the position corresponding to the target pixel point and the pixel value.

In one embodiment, the first determining module is further configured to perform reduction processing on an adjacent video frame in the plurality of video frames of the video to obtain a reduced adjacent video frame; carrying out gray scale processing on the reduced adjacent video frames to obtain adjacent gray scale images; in a plurality of video frames of the video, performing difference processing on gray values corresponding to pixel points in adjacent gray images to obtain gray difference values; obtaining a difference gray scale map based on each gray scale difference; and moving the sliding frame in the difference gray level diagram according to a target step length, counting the number of pixels with gray level values larger than a preset gray level value in the sliding frame in the process of moving the sliding frame each time, assigning the pixels in the sliding frame to be a first gray level value when the counted number of the pixels is larger than the preset number, and assigning the pixels in the sliding frame to be a second gray level value when the counted number of the pixels is smaller than or equal to the preset number until the assignment of the pixels in each region in the difference gray level diagram is completed, thereby obtaining a first motion mask.

In one embodiment, the determining and calculating module is further configured to determine a target moving object according to the first moving region and the second moving region; determining a pixel point of the target moving object in the first moving area, and determining a pixel point of the target moving object in the second moving area; a plurality of optical flows are determined based on the pixels of the target moving object in the first motion region and the pixels in the second motion region.

In one embodiment, the second determining module is further configured to take the first threshold as the number of interpolated frames when the optical flow is greater than the first threshold; equally dividing the optical flow according to the number of the inserted frames to obtain an equal value; and determining candidate points based on the equivalent value and the pixel points corresponding to the optical flow in the first motion area.

In one embodiment, the second determining module is further configured to determine, when the optical flow is greater than or equal to a second threshold and not greater than the first threshold, a candidate point according to the optical flow and a pixel point corresponding to the optical flow in the first motion area; determining a pixel point corresponding to the candidate point in the second motion area; and selecting the pixel point with the largest pixel value from the pixel points corresponding to the candidate points and the pixel points at the corresponding positions in the video frames as a target pixel point, and according to the position and the pixel value corresponding to the target pixel point, selecting the pixel point with the largest pixel value as the target pixel point, and selecting the pixel point with the largest pixel value as the target pixel point according to the position and the pixel value corresponding to the target pixel point.

In one embodiment, the selecting and updating module is further configured to obtain a tailing number; and selecting each adjacent video frame with the tailing number from the video frames, updating the pixel point at the target position in the target video frame, and obtaining the updated target video frame when the updating is completed.

In one embodiment, the selecting and updating module is further configured to determine a frame number difference value according to the tailing number and the acceleration multiple when the tailing number is greater than the acceleration multiple; determining a first pixel point of a target video frame in a plurality of video frames according to the frame number difference value; acquiring a plurality of new video frames with the frame number equal to the acceleration multiple, and determining a second pixel point in the new video frames; selecting the pixel point with the largest pixel value from the determined first pixel point and the determined second pixel point as a replacement pixel point; and updating the pixel point at the target position in the next target video frame according to the replaced pixel point.

The application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the steps of the above method when the processor executes the computer program.

The present application also provides a computer-readable storage medium. The computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the above method.

The present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of the above method.

The video processing method, apparatus, computer device and storage medium described above, by determining a first motion mask between adjacent video frames among a plurality of video frames of a video; determining a first motion area and a second motion area of adjacent video frames according to the first motion mask, and calculating optical flow between the first motion area and the second motion area; determining pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the first motion area and the optical flow; the candidate points are determined based on the selected pixel points of the optical flow when the optical flow is greater than a first threshold value; and selecting the pixel point with the largest pixel value from the pixel points corresponding to the candidate points and the pixel points corresponding to the positions in the video frames as a target pixel point, and according to the position corresponding to the target pixel point and the pixel value of the target video frame. In addition, the target pixel point with the largest pixel value is selected to update the target video frame by comparing the pixel points, so that the long exposure time-delay shooting effect can be realized without additionally inserting frames into the video, the memory space is greatly saved, the video processing efficiency is effectively improved, and the video processing effect is optimized.

The application also provides a video processing method, which comprises the following steps:

grouping a plurality of video frames of the video to obtain at least two video frame groups;

acquiring a first video frame and a second video frame from each video frame group;

determining a second motion mask between the first video frame and the second video frame of each group in turn;

determining an optical flow between the first video frame and the second video frame based on the second motion mask;

taking the optical flow between the first video frame and the second video frame of each group as the optical flow corresponding to each video frame group;

sequentially determining pixel points corresponding to candidate points according to the pixel points corresponding to the optical flow and the optical flow in the third motion area of the adjacent video frames in each video frame group; the candidate points are determined based on the pixel values selected by the optical flow when the optical flow is greater than a first threshold value; the third motion region and the fourth motion region are motion regions of different ones of the adjacent video frames;

and selecting the pixel point with the largest pixel value from the determined pixel points as a target pixel point, and according to the position corresponding to the target pixel point and the pixel value, obtaining a target video frame.

The application also provides a video processing device. The device comprises:

the grouping module is used for grouping a plurality of video frames of the video to obtain at least two video frame groups;

the acquisition module is used for acquiring a first video frame and a second video frame from each video frame group;

a first determining module for sequentially determining a second motion mask between the first video frame and the second video frame of each group;

a second determination module for determining an optical flow between the first video frame and the second video frame based on the second motion mask;

the processing module is used for taking the optical flow between the first video frame and the second video frame of each group as the optical flow corresponding to each video frame group;

the third determining module is used for determining the pixel points corresponding to the candidate points according to the pixel points corresponding to the optical flow and the optical flow in the third motion area of the adjacent video frames in each video frame group in sequence; the candidate points are determined based on the pixel values selected by the optical flow when the optical flow is greater than a first threshold value; the third motion region and the fourth motion region are motion regions of different ones of the adjacent video frames;

And the updating module is used for selecting the pixel point with the largest pixel value from the determined pixel points as a target pixel point, and selecting the target video frame according to the position and the pixel value corresponding to the target pixel point.

The video processing method, the video processing device, the computer equipment and the storage medium are used for obtaining at least two video frame groups by grouping a plurality of video frames of the video; acquiring a first video frame and a second video frame from each video frame group; determining a second motion mask between the first video frame and the second video frame of each group in turn; determining an optical flow between the first video frame and the second video frame based on the second motion mask; taking the optical flow between the first video frame and the second video frame of each group as the optical flow corresponding to each video frame group; the method has the advantages that the plurality of video frames share one optical flow, the calculated amount of the optical flow is effectively saved, and the pixel points corresponding to the candidate points are determined sequentially according to the pixel points corresponding to the optical flow in the third motion area of the adjacent video frames in each video frame group and the optical flow; when the optical flow is larger than the first threshold value, the candidate points are determined based on the selected pixel values of the optical flow; the third motion region and the fourth motion region are motion regions of different video frames in adjacent video frames; and selecting the pixel point with the largest pixel value from the determined pixel points as a target pixel point, and according to the position corresponding to the target pixel point and the pixel value, obtaining a target video frame. The method has the advantages that the long exposure time-delay shooting effect can be realized without additionally inserting frames into the video, the memory space is greatly saved, and further, the video processing efficiency is further improved and the video processing effect is optimized by sharing the optical flow with a plurality of video frames.

Drawings

FIG. 1 is an application environment diagram of a video processing method in one embodiment;

FIG. 2 is a flow chart of a video processing method in one embodiment;

FIG. 3a is a corresponding one of the gray scale images in adjacent video frames in one embodiment;

FIG. 3b is another gray scale view of a corresponding one of adjacent video frames in one embodiment;

FIG. 3c is a schematic diagram of a motion mask in one embodiment;

FIG. 4 is a flowchart illustrating an optical flow determination step in another embodiment;

FIG. 5a is a schematic diagram of a target video frame in one embodiment;

FIG. 5b is a schematic diagram of an updated target video frame in one embodiment;

FIG. 6a is a schematic diagram illustrating the effect of the maximum stacking algorithm in one embodiment;

FIG. 6b is a schematic diagram illustrating the effect of the video processing method according to the present application in one embodiment;

FIG. 7a is a schematic diagram showing the effect of the maximum stacking algorithm in another embodiment;

FIG. 7b is a schematic diagram illustrating the effect of the video processing method according to another embodiment;

FIG. 8 is a flow chart of a video processing method according to another embodiment;

FIG. 9 is a block diagram showing a configuration of a video processing apparatus in one embodiment;

FIG. 10 is a block diagram showing the structure of a video processing apparatus according to another embodiment;

FIG. 11 is an internal block diagram of a computer device in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.

The video processing method provided by the embodiment of the application can be applied to an application environment shown in fig. 1. Wherein the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104 or may be located on a cloud or other network server. The present application may be executed by the terminal 102 or the server 104, and this embodiment is described by taking the execution of the terminal 102 as an example.

The terminal 102 determines a first motion mask between adjacent video frames among a plurality of video frames of the video; the terminal 102 determines a first motion area and a second motion area of adjacent video frames according to the first motion mask, and calculates an optical flow between the first motion area and the second motion area; the terminal 102 determines the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the first motion area and the optical flow; the candidate points are determined based on the selected pixel points of the optical flow when the optical flow is greater than a first threshold value; the terminal 102 selects a pixel point with the largest pixel value as a target pixel point from the pixel points corresponding to each candidate point and the pixel points corresponding to the positions in a plurality of video frames, and the terminal selects the pixel point with the largest pixel value as the target video frame according to the position and the pixel value corresponding to the target pixel point

The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things devices, and portable wearable devices, where the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, and the like. The portable wearable device may be a smart watch, smart bracelet, headset, or the like. The server 104 may be implemented as a stand-alone server or as a server cluster of multiple servers.

In one embodiment, as shown in fig. 2, a video processing method is provided, and the method is applied to the terminal 102 in fig. 1 for illustration, and includes the following steps:

s202, determining a first motion mask between adjacent video frames in a plurality of video frames of a video.

The adjacent video frames may refer to two adjacent video frames, the plurality of video frames may refer to video frames arranged in a certain order in the same video, and the adjacent video frames may be adjacent video frames in the plurality of video frames arranged in a certain order. For example, the plurality of video frames may be arranged in the order in which the video frames are played, and then the adjacent video frames may be a video frame played before and a video frame played after. The first motion mask may be used for images reflecting motion between adjacent video frames.

Specifically, the terminal may perform reduction processing on adjacent video frames in the plurality of video frames of the video to obtain reduced adjacent video frames; gray processing is carried out on the reduced adjacent video frames to obtain adjacent gray images, and gray values corresponding to pixel points in the adjacent gray images are subjected to difference processing in a plurality of video frames of the video to obtain gray difference values; obtaining a difference gray scale map based on each gray scale difference; and moving the sliding frame in the difference gray level diagram according to the target step length, counting the number of pixels with gray level values larger than a preset gray level value in the sliding frame in the process of moving the sliding frame each time, assigning the pixels in the sliding frame to be a first gray level value when the counted number of the pixels is larger than the preset number, assigning the pixels in the sliding frame to be a second gray level value when the counted number of the pixels is smaller than or equal to the preset number, and obtaining a first motion mask until the assignment of the pixels in each region in the difference gray level diagram is completed. FIG. 3a is a corresponding one of the gray scale images in adjacent video frames in one embodiment; FIG. 3b is another gray scale view of a corresponding one of adjacent video frames in one embodiment; as shown, the adjacent gray scale patterns may be those shown in FIGS. 3a and 3b, and FIG. 3c is a schematic diagram of a motion mask in one embodiment; a first motion mask as shown in fig. 3c can be obtained from the adjacent gray scale pattern.

The gray level difference may refer to a difference obtained by subtracting gray level values of corresponding pixels. The difference gray map may refer to a gray map generated based on gray differences. The target step size may refer to a step size for each movement of the sliding frame in the difference gray scale map. The sliding box may be used to define the range of pixels currently being determined in the difference gray scale map. The preset gray value may refer to a preset gray value, and the preset gray value may be used to determine the gray value of the pixel point. The preset number may refer to a preset number, which may be used to determine the number of pixel points. The first gray value and the second gray value may refer to gray values obtained by binarizing gray values of pixels in the difference gray map, for example, the first gray value may be 1 and the second gray value may be 0.

In one embodiment, the terminal may further perform size adjustment processing on adjacent video frames in the plurality of video frames according to a preset proportion, to obtain a size-adjusted adjacent video frame, and perform gray scale processing on the size-adjusted adjacent video frame, to obtain an adjacent gray scale map.

The preset ratio may refer to a preset ratio, and the preset ratio may be used to perform a reduction process on the video frame, for example, the preset ratio may be 0.75, that is, the preset ratio represents that the video frame is reduced to 0.75 times of the original video frame.

In one embodiment, the difference processing is performed on the gray values corresponding to the pixels in the determined adjacent gray images in the plurality of gray images, so that each gray difference value is obtained, and the terminal can perform the difference processing on the gray values corresponding to the pixels in the determined adjacent gray images in the plurality of gray images, so as to obtain the gray values after the difference processing, and perform the absolute value processing on the gray values after the difference processing, so as to obtain each gray difference value.

For example, the adjacent gray-scale images are gray-scale image a and gray-scale image B, the gray-scale values of the pixels in the gray-scale image a are {33, 76, 89, 188, 125}, the gray-scale values of the pixels in the gray-scale image B are {12, 54, 66, 198, 215}, and the gray-scale values of the pixels in the gray-scale image a and the gray-scale image B are correspondingly subtracted {33-12, 76-54, 89-66, 188-198, 125-215}, thereby obtaining the gray-scale values after the difference: {21, 22, 23, -10, -90}, and performing absolute value processing on each of the difference gray values to obtain each gray difference {21, 22, 23, 10, 90}.

S204, determining a first motion area and a second motion area of the adjacent video frames according to the first motion mask, and calculating optical flow between the first motion area and the second motion area.

The first motion region may refer to a region corresponding to a first gray value in a first motion mask in a previous video frame of the adjacent video frames. The second motion region may refer to a region in a subsequent one of the adjacent video frames corresponding to the first gray value in the first motion mask. Optical flow may be used to reflect the motion of moving objects between adjacent video frames. For example, the moving object is a, and the coordinate positions corresponding to a certain window center position in a on the adjacent video frames i, j are (1, 3), (5, 6), and then the optical flow of the certain window center position is (5-1, 6-3) = (4, 3) or (1-5, 3-6) = (-4, -3).

Specifically, the terminal may first determine a first gray scale region in the first motion mask, and determine a corresponding first motion region and a corresponding second motion region in the adjacent video frame according to the first gray scale region. And calculating a plurality of optical flows corresponding to the moving objects in the first and second motion areas. It should be noted that there may be one or more moving objects in the first moving region and the second moving region, where one moving object may correspond to a plurality of pixels, and one optical flow corresponds to each pair of corresponding pixels in the first moving region and the second moving region.

The first gray scale region may refer to a region where the gray scale value of the pixel point is the first gray scale value. A moving object may refer to an object whose corresponding positions in the first and second movement regions change, for example, there is a vehicle C whose center is (3, 4) in the first movement region and whose center is (4, 6) in the second movement region, because (3, 4) is different from (4, 6), that is, the corresponding position of the vehicle C changes, and the vehicle C is a moving object.

S206, determining the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the first motion area and the optical flow; the candidate points are determined based on the selected pixels of the optical flow when the optical flow is greater than the first threshold.

The candidate points may refer to points determined based on the optical flow and the corresponding pixel points. For example, the candidate points may be points determined based on the pixel points corresponding to the optical flows and the equal score values of the optical flows, and the equal score values may refer to the equal score values. It should be noted that, here, the present application is not limited to the value of the optical flow, and the proportion of the optical flow in the calculation candidate points may be adjusted correspondingly according to the specific situation. The first threshold may refer to a threshold for determining the light flow. The pixel point corresponding to the candidate point may refer to a pixel point corresponding to the candidate point in the second motion region.

For example, the position of the pixel point corresponding to the optical flow in the first motion area is (a, b), the optical flow is (m, n), the number of interpolation frames is d, d is an integer, and the candidate point H _i The calculation formula of (c) can be:

where i=d. Assuming that the optical flow is (6, 9), the number of the inserted frames is 2, and the position of the pixel point corresponding to the optical flow in the first motion area is (1, 2), judging that two candidate points exist, and carrying the two candidate points into the formula to obtain the candidate points

Candidate Point->

Wherein the number of the interpolated frames may be used to determine the candidate points corresponding to the optical flow and to determine the number of the candidate points, e.g., the optical flow is (6, 9), and when the number of the interpolated frames is 2, the number of the candidate points may be 2. The equivalent values of the optical flow may be (2, 3) and (4, 6).

In one embodiment, when the optical flow is greater than the first threshold, the first threshold is taken as the number of interpolated frames; equally dividing the optical flow according to the number of the inserted frames to obtain a processing result; and determining a candidate point based on the processing result and the pixel point corresponding to the optical flow in the first motion area. The processing result may refer to an equal value of the optical flow, that is, a result of performing equal processing on the optical flow.

For example, the optical flow is (m, n), the number of interpolated frames is d, d is an integer, and the calculation formula of the processing result may be:

where d=i. For example, the number of inserted frames is 2, the optical flow is (6, 9), namely, the optical flow is expressed, 2 candidate points are obtained by equally dividing in (6, 9), and the calculation formula of the processing result is substituted to obtain: processing the resultAre (2, 3) and (4, 6).

In one embodiment, determining the candidate points includes determining the candidate points based on the pixel points corresponding to the optical flow and the equal-value of the optical flow when the optical flow is greater than a first threshold, or determining the candidate points based on the pixel points corresponding to the optical flow and the equal-value of the optical flow when the maximum value of the optical flow is greater than the first threshold, or determining the optical flow product value between the optical flow and the optical flow coefficient first, and determining the candidate points based on the pixel points corresponding to the optical flow and the equal-value of the optical flow when the maximum value of the optical flow product value is greater than the first threshold.

Where the optical flow coefficient may refer to a coefficient for amplifying or reducing the optical flow. For example, the optical flow coefficient may correspond to a preset ratio, and when the preset ratio is 0.25, the optical flow coefficient may be 4. The maximum value of the optical flow may refer to a larger value of the optical flows, for example, (3, 4) then the maximum value of the optical flow is 4. The optical flow product value may refer to the product between optical flow and optical flow coefficients.

In one embodiment, when the optical flow is greater than or equal to the second threshold value and not greater than the first threshold value, determining a candidate point according to the optical flow and the pixel points corresponding to the optical flow in the first motion area; determining pixel points corresponding to the candidate points in the second motion area; and selecting the pixel point with the largest pixel value from the pixel points corresponding to the candidate points and the pixel points corresponding to the positions in the video frames as a target pixel point, and according to the position corresponding to the target pixel point and the pixel value of the target video frame.

The second threshold may be used to determine a threshold of optical flow, the second threshold being less than or equal to the first threshold, the first threshold and the second threshold being different thresholds. The pixel value may refer to a value corresponding to a pixel, and the pixel value may refer to a gray value or a sum of values of three channels of RGB (red, green and blue), for example, RGB of the pixel is (80, 90, 180), where the pixel value of the pixel is: 80+90+180=350. The target pixel point may refer to a point used to replace the pixel point at the target location within the target video frame. The target video frame may refer to a video frame that needs to be updated in pixel points, and the target video frame may be any video frame of a plurality of video frames.

In one embodiment, when the optical flow is greater than or equal to the second threshold and is not greater than the first threshold, determining a pixel corresponding to a candidate point in the second motion area according to the pixel corresponding to the optical flow in the first motion area and the optical flow, selecting a pixel with the largest pixel value from the pixel corresponding to the candidate point and the pixels corresponding to the positions in the plurality of video frames as a target pixel, and obtaining the target video frame according to the position and the pixel value corresponding to the target pixel. For example, if the coordinates of the pixel corresponding to the optical flow in the first motion region are (1, 2) and the optical flow is (3, 4), the candidate point is (1+3, 2+4) = (4, 6), that is, the coordinates of the pixel corresponding to the candidate point in the second motion region are (4, 6).

In one embodiment, when the optical flow is smaller than the second threshold, the current round is skipped to update the pixel at the target position in the target video frame according to the optical flow, and a new optical flow is selected to update the pixel at the next target in the target video frame.

The target position may refer to a position of a pixel point updated in the target video frame, where each time the updated pixel point is different, and the target position is different, for example, the target position of the pixel point updated corresponding to the current optical flow may be (1, 2), and the target position of the pixel point updated corresponding to the next optical flow may be (1, 3).

S208, selecting the pixel point with the largest pixel value from the pixel points corresponding to the candidate points and the pixel points corresponding to the positions in the video frames as a target pixel point, and according to the position and the pixel value corresponding to the target pixel point, selecting the pixel point with the largest pixel value as the target pixel point.

Specifically, the terminal may select, from among the pixel points corresponding to each candidate point and the pixel points corresponding to the candidate points in the plurality of video frames, the pixel point with the largest pixel value as the target pixel point, determine the target position according to the position of the target pixel point in the target video frame after determining the target video frame in the plurality of video frames, and update the pixel value in the target position according to the pixel value of the target pixel point, thereby obtaining the updated target video frame.

In one embodiment, when the target video frame is a first video frame (a first video frame) of the plurality of video frames and the adjacent video frames are a first Zhang Shipin frame and a second video frame, a pixel point with the largest pixel value is selected as the target pixel point from the pixel points corresponding to the optical flow in the first motion area and the pixel points corresponding to the candidate points in the second motion area, and the pixel point at the target position in the first video frame is updated according to the target pixel point.

In one embodiment, when the target video frame is not the first video frame of the plurality of video frames and is not the last video frame of the plurality of video frames, selecting a pixel with the largest pixel value from the determined pixel points and the pixel points of the target positions in the target video frames of the plurality of video frames as the target pixel point, and updating the pixel point at the target position in the target video frame according to the target pixel point. For example, FIG. 5a is a schematic diagram of a target video frame in one embodiment; FIG. 5b is a schematic diagram of an updated target video frame in one embodiment; as shown, when all the pixels in the motion area in the target video frame as shown in fig. 5a are replaced, a schematic diagram with good tailing effect as shown in fig. 5b can be obtained.

In one embodiment, when the target video frame is a last Zhang Shipin frame (last video frame) of the plurality of video frames and the adjacent video frames are the last video frame and the last but one video frame, the pixel point with the largest pixel value is selected as the target pixel point from the pixel points corresponding to the optical flow in the first motion area and the pixel points corresponding to the candidate points in the second motion area, and the pixel point at the target position in the last video frame is updated according to the target pixel point.

In one embodiment, after updating the pixel at the target location within the target video frame based on all optical flows between the first motion region and the second motion region in the adjacent video frames, a next adjacent video frame is selected from the plurality of video frames to update the pixel at the target location within the target video frame.

In one embodiment, after S208, the terminal may acquire the smear amount; and selecting a tailing number of adjacent video frames from the plurality of video frames, updating the pixel points at the target positions in the target video frames, and obtaining the updated target video frames when the updating is completed.

Wherein the hangover number may refer to the number of video frames used to obtain the target video frame, e.g., a hangover number of 4 frames, indicating each neighboring video frame of the 4 video frames to be used: the method comprises the steps of updating a pixel point at a target position in a target video frame (a first frame or a fourth frame) for 4 times, and then playing the target video frame.

In one embodiment, after S208, the terminal may further determine a frame number difference value according to the smear number and the acceleration multiple when the smear number is greater than the acceleration multiple; determining a first pixel point of a target video frame in a plurality of video frames according to the frame number difference value; acquiring a plurality of new video frames with the frame number equal to the acceleration multiple, and determining a second pixel point in the plurality of new video frames; selecting a pixel point with the maximum pixel value from the determined first pixel point and the determined second pixel point as a replacement pixel point; and updating the pixel point at the target position in the next target video frame according to the replaced pixel point.

The acceleration multiple may refer to a playing speed when playing the target video frame. The frame number difference may refer to a difference between frame numbers. The first pixel point may refer to a pixel point in the target video frame. The number of frames may refer to the number of video frames. The second pixel point may refer to a pixel point in the new video. The replacement pixel point may refer to a point for a pixel point at a target location within a target video frame.

For example, the smear length (smear number) is 32 frames, and the acceleration multiple is 16 times. Firstly, synthesizing the 1 st frame to the 32 nd frame into one frame (target video frame) and then playing the frame; since the acceleration multiple is half of the hangover length, and the frame number difference is 16, the next frame of the output video (the next target video frame) needs to be played by synthesizing the 17 th frame to the 48 th frame into the 1 st frame. Therefore, we can save the target pixel point with the largest pixel value in the 17 th frame to the 32 th frame, namely save the target pixel point of the target video frame in the 17 th frame to the 32 nd frame, then only determine the pixel point with the largest pixel value in the 33 rd frame to the 48 th frame so as to obtain the target pixel point, compare the pixel value of the target pixel point saved in the past, select the target pixel point with larger pixel value as the replacement pixel point, and update the pixel point at the target position in the next target video frame according to the replacement pixel point, thereby avoiding the need of recalculating the target pixel point in the 17 th frame to the 32 nd frame.

The video processing method comprises the steps of determining a first motion mask between adjacent video frames in a plurality of video frames of a video; determining a first motion area and a second motion area of adjacent video frames according to the first motion mask, and calculating optical flow between the first motion area and the second motion area; the motion mask is used for determining the motion areas of the moving objects in the adjacent video frames so as to determine the optical flow between the motion areas, and compared with the traditional mode of directly calculating the optical flow between the complete video frames, the calculation amount is effectively reduced. Moreover, according to the pixel points corresponding to the optical flows in the first motion area and the optical flows, determining the pixel points corresponding to each candidate point; the candidate points are determined based on the selected pixel points of the optical flow when the optical flow is greater than a first threshold value; and selecting the pixel point with the largest pixel value from the pixel points corresponding to the candidate points and the pixel points corresponding to the positions in the video frames as a target pixel point, and according to the position corresponding to the target pixel point and the pixel value of the target video frame. In addition, the target pixel point with the largest pixel value is selected to update the target video frame by comparing the pixel points, so that the long exposure time-delay shooting effect can be realized without additionally inserting frames into the video, the memory space is greatly saved, the video processing efficiency is effectively improved, and the video processing effect is optimized.

In one embodiment, as shown in FIG. 4, the determining optical flow step includes:

s402, determining a target moving object according to the first moving area and the second moving area.

The target moving object may refer to the same object that appears in both the first moving area and the second moving area, and the position of the object in the first moving area is different from the position of the object in the second moving area, that is, the target moving object.

Specifically, the terminal may determine a first motion region in a previous video frame in the adjacent video frames according to the first motion mask and identify a motion object in the first motion region, determine a second motion region in a next video frame in the adjacent video frames according to the first motion mask and identify a motion object in the second motion region, and determine a target motion object from the motion object in the first motion region and the motion object in the second motion region.

In one embodiment, determining the first motion region in the previous video frame of the adjacent video frames according to the first motion mask includes the terminal determining the first gray scale region according to the first motion mask, and then determining the first motion region corresponding to the first gray scale region in the previous video frame of the adjacent video frames.

In one embodiment, determining the second motion region in the next video frame of the adjacent video frames based on the first motion mask includes the terminal determining the first gray scale region based on the first motion mask and determining the second motion region corresponding to the first gray scale region in the next video frame of the adjacent video frames.

S404, determining the pixel point of the target moving object in the first moving area and determining the pixel point of the target moving object in the second moving area.

Specifically, the terminal may determine a first target area in which the target moving object is located in the first moving area, and use a pixel point in the first target area as a pixel point of the target moving object; and determining a second target area where the target moving object is located in the second moving area, and taking the pixel point in the second target area as the pixel point of the target moving object.

The first target area may refer to an area where the target moving object is located in the first moving area. The second target region may refer to a region in which the target moving object is located in the second moving region.

S406, determining a plurality of optical flows based on the pixel points of the target moving object in the first moving region and the pixel points in the second moving region.

Specifically, the terminal may determine a correspondence between a pixel point of the target moving object in the first moving area and a pixel point of the target moving object in the second moving area, determine a plurality of corresponding pixel points of the target moving object between the first moving area and the second moving area according to the correspondence, and determine a plurality of optical flows according to the plurality of corresponding pixel points.

The correspondence may refer to a correspondence between a pixel point of the target moving object in the first moving area and a pixel point in the second moving area, for example, the target moving object is a pedestrian D, a center position of a left shoulder of the pedestrian D, a corresponding pixel point in the first moving area is (2, 3), and a corresponding pixel point in the second moving area is (3, 4), where (2, 3) and (3, 4) are position coordinates of the pixel points, the correspondence is (2, 3) in the first moving area, and the correspondence is corresponding to (3, 4) in the second moving area, and then an optical flow of the center position of the left shoulder of the pedestrian D may be (2-3, 3-4) = (-1, -1), or (3-2, 4-3) = (1, 1).

In this embodiment, by determining the target moving object according to the first moving region and the second moving region, determining the pixel point of the target moving object in the first moving region, and determining the pixel point of the target moving object in the second moving region, based on the pixel point of the target moving object in the first moving region and the pixel point in the second moving region, determining a plurality of optical flows, it is possible to obtain the optical flow between the first moving region and the second moving region, so that the pixel point at the target position in the target video frame is updated subsequently.

As an example, the embodiments are as follows:

step 1, inputting n frames (the smear length, namely the smear number) of the original video;

step 2, from the nth frame to the 2 nd frame, taking the nth frame and the n-1 st frame, preprocessing the gray level image reduced to 0.25 times (only one preprocessing operation is needed when the frames are continuously inserted), and calculating a motion value (motion) between the two reduced images, namely counting gray level difference values in a certain area to obtain a motion mask.

Step 3, calculating the optical flow in the contracted graph mask by using an optical flow algorithm (parameter setting bias calculation micro displacement) based on the motion mask; (the optical flow algorithm may be any optical flow algorithm here, and only the optical flow of the region with the gray value of 1 in the motion mask needs to be calculated

Step 4, directly on the original image of the nth frame, updating the pixel value of the original image of the nth frame according to the corresponding optical flow for the pixel points mapped to the inside of the motion mask:

step 4.1. After multiplying the optical flow by the corresponding coefficient (optical flow coefficient), equally dividing the optical flow into y candidate points (equivalent to inserting y frames) if the optical flow is larger than x pixels; if the pixel size is smaller than z pixels, skipping, and performing no processing; suggesting that x=y, z=1, and can be arbitrarily selected according to the requirements of performance and effect;

step 4.2. Comparing the current pixel value (the pixel value of the pixel point at the target position in the target video frame) with the pixel value of the candidate point, and taking the largest replacement current pixel point (the pixel point at the target position in the target video frame).

In the white area (gray value of 1) of the motion mask (motion_mask), calculation is performed according to the optical flow, such as: according to the optical flow, in the 32 th frame, the object located at the position a appears at the position a+u of the 31 st frame (this u is the optical flow, and a and u are two-dimensional positions such as a= (1, 2)), so that all points on the line "a to a+u" of the 31 st frame pass through the position a of the 32 nd frame in the time when the 31 st frame changes to the 32 nd frame in the real world.

If 3 frames are inserted between every two frames, the point at the three positions of a+0.25u, a+0.5u and a+0.75u in the 31 st frame should be taken, and compared with the point at the 32 th frame position a, the point with the largest pixel value is taken and assigned to the 32 th frame position a. This is done later on for all points in the "motion_mask".

The next round should take the 31 st frame and the 30 th frame, after completing the previous operation (assuming that the optical flow of this time is v), the following steps are: in the 30 th frame, the points at the three positions of "a+0.25v, a+0.5v and a+0.75v", the point at the 31 st frame position a and the point at the 32 nd frame position a (the point at the 32 nd frame position a after the previous round of assignment) are compared, and the maximum reassigned to the 32 nd frame position a is obtained.

Step 5, after all the frames from the nth frame to the 1 st frame are replaced, obtaining frames after n frame inserting processing, and obtaining a synthesized 1 frame result; FIG. 6a is a schematic diagram illustrating the effect of the maximum stacking algorithm in one embodiment; FIG. 6b is a schematic diagram illustrating the effect of the video processing method according to the present application in one embodiment; FIG. 7a is a schematic diagram showing the effect of the maximum stacking algorithm in another embodiment; fig. 7b is a schematic diagram illustrating an effect of the video processing method according to another embodiment. As is evident from a comparison of fig. 6a and 6b, and fig. 7a and 7b, the consistency effect of the composite frame of the present application is better than that of a general maximum stacking algorithm.

Step 6, outputting the synthesized frames, and when the tail length is greater than the acceleration multiple, storing n video frames subjected to frame inserting processing so as to be reused.

In one embodiment, as shown in fig. 8, another video processing method is provided, and the method is applied to the terminal in fig. 1 for illustration, and includes the following steps:

s802, grouping a plurality of video frames of a video to obtain at least two video frame groups;

the adjacent video frames may refer to two adjacent video frames, the plurality of video frames may refer to video frames arranged in a certain order in the same video, and the adjacent video frames may be adjacent video frames in the plurality of video frames arranged in a certain order. For example, the plurality of video frames may be arranged in the order in which the video frames are played, and then the adjacent video frames may be a video frame played before and a video frame played after. A group of video frames may refer to a group comprising at least two video frames.

Specifically, the terminal may group the plurality of video frames according to the number of packets to obtain at least two video frame groups. Wherein the number of packets may refer to the number of groups of video frames, for example, the number of packets may be 4.

In one embodiment, the terminal may determine the total number of the plurality of video frames, determine the number of video frames in each video frame group according to the total number and the number of packets, and sequentially select the video frames with the number of video frames from the plurality of video frames to obtain at least two video frame groups.

Wherein the total number may refer to the number of the plurality of video frames. The number of video frames may refer to the number of video frames.

For example, there are 32 video frames in total, the number of packets is 4, then the number of video frames in each video frame group can be determined to be 8 according to the total number of 32 and the number of packets 4, and then the video frames 1 to 8,9 to 16, 17 to 24, and 25 to 32 are sequentially selected for the 32 video frames, so that 4 video frame groups are obtained.

S804, acquiring a first video frame and a second video frame from each video frame group;

wherein a first video frame may refer to one video frame within a group of video frames that is used to determine optical flow between adjacent video frames. The second video frame may refer to another video frame within the group of video frames for determining optical flow between adjacent video frames.

Specifically, the terminal may first obtain the target frame number, and obtain the first video frame and the second video frame from each video frame group according to the target frame number.

The target frame number may refer to a sequence number of the first video frame and the second video frame in the video frame group, for example, each video frame group contains 8 video frames, the sequence number in each video frame group is 1 to 8, and the target frame numbers may be 4 and 6, which indicates that the first video frame and the second video frame are the fourth frame and the sixth frame in the video frame group, respectively.

S806, determining a second motion mask between the first video frame and the second video frame of each group in turn.

Wherein the second motion mask is operable to reflect an image of a motion situation between the first video frame and the second video frame.

Specifically, the terminal may perform gray scale processing on the first video frame and the second video frame or perform reduction processing on the first video frame and the second video frame to obtain reduced first video frame and second video frame, perform gray scale processing on the reduced first video frame and the reduced second video frame to obtain two corresponding gray scale images, and perform difference processing on gray scale values corresponding to pixels in the determined two gray scale images to obtain each gray scale difference value; obtaining a difference gray scale map based on each gray scale difference; and moving the sliding frame in the difference gray level diagram according to the target step length, counting the number of pixels with gray level values larger than a preset gray level value in the sliding frame in the process of moving the sliding frame each time, assigning the pixels in the sliding frame to be a first gray level value when the counted number of the pixels is larger than the preset number, assigning the pixels in the sliding frame to be a second gray level value when the counted number of the pixels is smaller than or equal to the preset number, and obtaining a second motion mask until the assignment of the pixels in each region in the difference gray level diagram is completed.

In one embodiment, performing gray scale processing on the first video frame and the second video frame to obtain two corresponding gray scale images includes that the terminal can perform size adjustment processing on the first video frame and the second video frame according to a preset proportion to obtain two corresponding video frames after size adjustment, and performing gray scale processing on the two corresponding video frames after size adjustment to obtain two corresponding gray scale images.

In one embodiment, performing difference processing on the gray values corresponding to the pixel points in the determined two gray images to obtain each gray difference value includes that the terminal can perform difference processing on the gray values corresponding to the pixel points in the determined two gray images to obtain the gray values after difference processing, and performing absolute value taking processing on the gray values after difference processing to obtain each gray difference value.

For example, the gray scale values of the pixels in the gray scale map a are {33, 76, 89, 188, 125}, the gray scale values of the pixels in the gray scale map B are {12, 54, 66, 198, 215}, and the gray scale values of the pixels in the gray scale map a and the gray scale map B are correspondingly subtracted {33-12, 76-54, 89-66, 188-198, 125-215}, thereby obtaining the gray scale values after the difference: {21, 22, 23, -10, -90}, and performing absolute value processing on each of the difference gray values to obtain each gray difference {21, 22, 23, 10, 90}.

S808, determining an optical flow between the first video frame and the second video frame based on the second motion mask.

Wherein the optical flow may be used to reflect the motion of the moving object between the first video frame and the second video frame. For example, the moving object is a, and the coordinate positions corresponding to a certain window center position in a on the adjacent video frames i, j are (1, 3), (5, 6), and then the optical flow of the certain window center position is (5-1, 6-3) = (4, 3) or (1-5, 3-6) = (-4, -3).

Specifically, the terminal may first determine a first gray scale region in the second motion mask, and determine a first target motion region and a second target motion region corresponding to the first video frame and the second video frame according to the first gray scale region, respectively. And calculating a plurality of optical flows corresponding to the moving objects in the first target moving region and the second target moving region. It should be noted that there may be one or more moving objects in the first target moving region and the second target moving region, where one moving object may correspond to a plurality of pixels, and one optical flow corresponds to each pair of corresponding pixels in the first target moving region and the second target moving region.

The first gray scale region may refer to a region where the gray scale value of the pixel point is the first gray scale value. The first target motion region may refer to a region in the first video frame corresponding to the first gray value in the second motion mask. The second target motion region may refer to a region in the second video frame corresponding to the first gray value in the second motion mask. The moving object may refer to an object whose corresponding positions in the first target moving region and the second target moving region are changed, for example, there is a vehicle C whose center is (3, 4) in the first target moving region and whose center is (4, 6) in the second target moving region because (3, 4) is different from (4, 6), that is, the corresponding position of the vehicle C is changed, and the vehicle C is a moving object.

In one embodiment, the terminal may determine a first target motion area corresponding to the first video frame according to the second motion mask, and determine a target motion object according to the first target motion area and the second target motion area corresponding to the second video frame; determining pixel points of the target moving object in a first target moving area and determining pixel points of the target moving object in a second target moving area; a plurality of optical flows are determined based on the pixels of the target moving object in the first target moving region and the pixels in the second target moving region.

The target moving object may refer to the same object that appears in both the first target moving area and the second target moving area, and the position of the object in the first target moving area is different from the position of the object in the second target moving area, that is, the target moving object.

Specifically, determining the target moving object according to the first target moving area and the second target moving area includes that the terminal can determine the first target moving area in the first video frame according to the second moving mask, identify the moving object in the first target moving area, determine the second target moving area in the second video frame according to the second moving mask, identify the moving object in the second target moving area, and determine the target moving object from the moving object in the first target moving area and the moving object in the second target moving area.

In one embodiment, determining the first target motion region corresponding to the first video frame according to the second motion mask includes determining a first gray scale region by the terminal according to the second motion mask, and determining the first target motion region corresponding to the first gray scale region of the first video frame.

In one embodiment, determining the second target motion region in the second video frame according to the second motion mask includes the terminal determining the first gray scale region according to the second motion mask, and determining a second motion region of the second video frame corresponding to the first gray scale region.

In one embodiment, determining the pixel point of the target moving object in the first target moving area, and determining the pixel point of the target moving object in the second target moving area includes determining a third target area where the target moving object in the first target moving area is located by the terminal, and taking the pixel point in the third target area as the pixel point of the target moving object; and determining a fourth target area where the target moving object is located in the second target moving area, and taking the pixel point in the fourth target area as the pixel point of the target moving object.

The third target area may refer to an area where the target moving object is located in the first target moving area. The fourth target area may refer to an area in which the target moving object is located in the second target moving area.

In one embodiment, determining the plurality of optical flows includes the terminal determining a correspondence between the pixel of the target moving object in the first target moving area and the pixel of the target moving object in the second target moving area based on the pixel of the target moving object in the first target moving area and the pixel of the target moving object in the second target moving area, determining a plurality of corresponding pixels of the target moving object between the first target moving area and the second target moving area according to the correspondence, and determining a plurality of optical flows according to the plurality of corresponding pixels.

The correspondence may refer to a correspondence between a pixel point of the target moving object in the first target moving area and a pixel point of the target moving object in the second target moving area, for example, the target moving object is a pedestrian D, a center position of a left shoulder of the pedestrian D, a corresponding pixel point in the first target moving area is (2, 3), a corresponding pixel point in the second target moving area is (3, 4), and a corresponding pixel point in the second target moving area is (3, 4), where (2, 3) and (3, 4) are position coordinates of the pixel points, the correspondence is that (2, 3) in the first target moving area corresponds to (3, 4) in the second target moving area, and then an optical flow of the center position of the left shoulder of the pedestrian D may be (2-3, 3-4) = (-1, -1), or (3-2, 4-3) = (1, 1).

S810, the optical flow between the first video frame and the second video frame of each group is regarded as the optical flow corresponding to each video frame group.

S812, determining pixel points corresponding to the candidate points according to the pixel points corresponding to the optical flows in the third motion areas of the adjacent video frames in each video frame group and the optical flows in sequence; when the optical flow is larger than the first threshold value, the candidate points are determined based on the selected pixel values of the optical flow; the third motion region and the fourth motion region are motion regions of different ones of the adjacent video frames.

The third motion region may refer to a region corresponding to the first gray value in the second motion mask in a previous video frame of the adjacent video frames. The fourth motion region may refer to a region in a subsequent one of the adjacent video frames corresponding to the first gray value in the second motion mask. The candidate points may refer to points determined based on the pixel points corresponding to the optical flows and the optical flows, for example, the candidate points may be points determined based on the pixel points corresponding to the optical flows and the equal score values of the optical flows, and the equal score values may refer to the equal score values. It should be noted that, here, the present application is not limited to the value of the optical flow, and the proportion of the optical flow in the calculation candidate points may be adjusted correspondingly according to the specific situation. The first threshold may refer to a threshold for determining the light flow. The pixel point corresponding to the candidate point may refer to a pixel point corresponding to the candidate point in the second motion region.

For example, the position of the pixel corresponding to the optical flow in the third motion region is (a, b), the optical flow is (m, n), and the number of interpolation framesD is an integer, and the candidate point H _i The calculation formula of (c) can be:

where i=d. Assuming that the optical flow is (6, 9), the number of the inserted frames is 2, and the position of the pixel point corresponding to the optical flow in the third motion area is (1, 2), judging that two candidate points exist, and carrying the two candidate points into the formula to obtain the candidate points

Candidate Point->

In one embodiment, when the optical flow is greater than the first threshold, the first threshold is taken as the number of interpolated frames; equally dividing the optical flow according to the number of the inserted frames to obtain a processing result; and determining a candidate point based on the processing result and the pixel point corresponding to the optical flow in the third motion area. The processing result may refer to an equal value of the optical flow, that is, a result of performing equal processing on the optical flow.

Where d=i. For example, the number of inserted frames is 2, the optical flow is (6, 9), namely, the optical flow is expressed, 2 candidate points are obtained by equally dividing in (6, 9), and the calculation formula of the processing result is substituted to obtain: the treatment results were (2, 3) and (4, 6).

In one embodiment, when the optical flow is greater than or equal to the second threshold and not greater than the first threshold, determining a candidate point according to the optical flow and the pixel points corresponding to the optical flow in the third motion area; determining a pixel point corresponding to the candidate point in the fourth motion region; and selecting the pixel point with the largest pixel value from the pixel points corresponding to the candidate points and the pixel points corresponding to the positions in the video frames as a target pixel point, and according to the position corresponding to the target pixel point and the pixel value of the target video frame.

S814, selecting the pixel with the largest pixel value from the determined pixel points as a target pixel point, and according to the position corresponding to the target pixel point and the pixel value, obtaining a target video frame.

Specifically, in each video frame group, selecting a pixel with the largest pixel value from the determined pixel points as a target pixel point, updating the pixel with the largest pixel value at the target position in the target video frames of the video frames according to the target pixel point, storing the pixel with the largest pixel value, then, in the next video frame group, selecting the pixel with the largest pixel value as the target pixel point in the determined pixel points, updating the pixel with the largest pixel value at the target position in the target video frames of the video frames according to the target pixel point, correspondingly comparing the pixel with the largest pixel value stored in the two video frame groups to obtain the pixel with the larger pixel value as the target pixel point, updating the pixel with the largest pixel value at the target position in the target video frames of the video frames according to the target pixel point, finally, obtaining the pixel with the largest pixel value as the target pixel point, and updating the pixel with the target pixel value at the target position in the target video frames of the video frames according to the target pixel point.

In one embodiment, when the target video frame is a first video frame (a first video frame) of the plurality of video frames and the adjacent video frames are the first Zhang Shipin frame and the second video frame, a pixel point with the largest pixel value is selected as the target pixel point from the pixel points corresponding to the optical flow in the third motion area and the pixel points corresponding to the candidate points in the fourth motion area, and the pixel point at the target position in the first video frame is updated according to the target pixel point.

In one embodiment, when the target video frame is a last Zhang Shipin frame (last video frame) of the plurality of video frames and the adjacent video frames are the last video frame and the last but one video frame, the pixel point with the largest pixel value is selected as the target pixel point from the pixel points corresponding to the optical flow in the third motion area and the pixel points corresponding to the candidate points in the fourth motion area, and the pixel point at the target position in the last video frame is updated according to the target pixel point.

In one embodiment, after updating the pixel at the target location within the target video frame based on all optical flows between the third and fourth motion regions in the adjacent video frames, a next adjacent video frame is selected from the plurality of video frames to update the pixel at the target location within the target video frame.

According to the video processing method, a plurality of video frames of a video are grouped to obtain at least two video frame groups; acquiring a first video frame and a second video frame from each video frame group; determining a second motion mask between the first video frame and the second video frame of each group in turn; determining an optical flow between the first video frame and the second video frame based on the second motion mask; taking the optical flow between the first video frame and the second video frame of each group as the optical flow corresponding to each video frame group; the method has the advantages that the plurality of video frames share one optical flow, the calculated amount of the optical flow is effectively saved, and the pixel points corresponding to the candidate points are determined sequentially according to the pixel points corresponding to the optical flow in the third motion area of the adjacent video frames in each video frame group and the optical flow; when the optical flow is larger than the first threshold value, the candidate points are determined based on the selected pixel values of the optical flow; the third motion region and the fourth motion region are motion regions of different video frames in adjacent video frames; and selecting the pixel point with the largest pixel value from the determined pixel points as a target pixel point, and according to the position corresponding to the target pixel point and the pixel value, obtaining a target video frame. The method has the advantages that the long exposure time-delay shooting effect can be realized without additionally inserting frames into the video, the memory space is greatly saved, and further, the video processing efficiency is further improved and the video processing effect is optimized by sharing the optical flow with a plurality of video frames.

As one example, the present example is as follows:

(1) The acceleration idea is as follows: multiple frame sequences (multiple video frames) may share the same optical flow for interpolation.

The common optical flow can be calculated by selecting the first and last frames of the sequence, but the illumination of the first and last frames is greatly changed, which is not beneficial to the calculation of the optical flow and the motion value, and the large displacement is also not beneficial to the optical flow calculation.

The two frames in the middle of the sequence can be selected to calculate the optical flow, the smaller the interval between the two selected frames is, the higher the calculation accuracy of the optical flow and the motion value is, but the sharing capability of the first frame and the last frame is weakened, and tailing incoherence can be caused.

(2) Acceleration scheme and possible loss of effect:

every 8 frames are 1 group of inserted frames, a small amount of gaps appear in the tailing small probability, and more gaps appear when the motion is faster; every 4 frames is 1 group to make interpolation frame, when the motion is faster, a small amount of gaps appear.

(3) Specific acceleration algorithm

Step 1, changing the input of all N frames into the input of N ' frames each time, wherein N is an integer multiple of N ', and N=32 and N ' =8 are taken now;

taking each group of input 8 frames as an example, firstly taking the ith frame and the jth frame (i=3, j=6 for example) in the 1-8 frames, and calculating a motion value and an optical flow according to the steps of the core algorithm;

step 3, for each temporarily stored pixel point, taking the pixel point with the maximum pixel value between the original pixel and the current 8 frames, traversing the pixel points of the motion area according to the previous calculated optical flow and the core algorithm steps of the steps S202-S208, performing optical flow frame insertion to obtain y points, taking the pixel point with the maximum pixel value, and assigning values to the result;

Step 4, after the input of 4 groups is completed, taking out the result, and emptying and temporarily storing.

It should be understood that, although the steps in the flowcharts related to the above embodiments are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides a video processing device for implementing the video processing method. The implementation of the solution provided by the apparatus is similar to the implementation described in the above method, so the specific limitation of one or more embodiments of the video processing apparatus provided below may refer to the limitation of the video processing method described above, and will not be repeated here.

In one embodiment, as shown in fig. 9, there is provided a video processing apparatus including: a first determination module 902, a determination and calculation module 904, a second determination module 906, a selection and update module 908, wherein:

a first determining module 902 configured to determine a first motion mask between adjacent video frames among a plurality of video frames of a video;

a determining and calculating module 904, configured to determine a first motion region and a second motion region of an adjacent video frame according to the first motion mask, and calculate an optical flow between the first motion region and the second motion region;

a second determining module 906, configured to determine pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the first motion area and the optical flow; the candidate points are determined based on the selected pixel points of the optical flow when the optical flow is greater than a first threshold value;

the selecting and updating module 908 is configured to select, from the pixel points corresponding to each candidate point and the pixel points corresponding to the positions in the plurality of video frames, the pixel point with the largest pixel value as the target pixel point, and according to the position and the pixel value corresponding to the target pixel point, the target video frame.

In one embodiment, the first determining module 902 is further configured to perform reduction processing on an adjacent video frame in the plurality of video frames of the video to obtain a reduced adjacent video frame; gray processing is carried out on the reduced adjacent video frames to obtain adjacent gray images; in a plurality of video frames of the video, performing difference processing on gray values corresponding to pixel points in adjacent gray images to obtain gray difference values; obtaining a difference gray scale map based on each gray scale difference; and moving the sliding frame in the difference gray level diagram according to the target step length, counting the number of pixels with gray level values larger than a preset gray level value in the sliding frame in the process of moving the sliding frame each time, assigning the pixels in the sliding frame to be a first gray level value when the counted number of the pixels is larger than the preset number, assigning the pixels in the sliding frame to be a second gray level value when the counted number of the pixels is smaller than or equal to the preset number, and obtaining a first motion mask until the assignment of the pixels in each region in the difference gray level diagram is completed.

In one embodiment, the determining and calculating module 904 is further configured to determine the target moving object according to the first moving region and the second moving region; determining a pixel point of the target moving object in the first moving area and determining a pixel point of the target moving object in the second moving area; a plurality of optical flows are determined based on the pixel points of the target moving object in the first motion region and the pixel points in the second motion region.

In one embodiment, the second determining module 906 is further configured to take the first threshold as the number of interpolated frames when the optical flow is greater than the first threshold; equally dividing the optical flow according to the number of the inserted frames to obtain an equal value; candidate points are determined based on the equipotential values and the pixels corresponding to the optical flow in the first motion region.

In one embodiment, the second determining module 906 is further configured to determine, when the optical flow is greater than or equal to the second threshold and not greater than the first threshold, a candidate point according to the optical flow and a pixel point corresponding to the optical flow in the first motion area; determining pixel points corresponding to the candidate points in the second motion area; and selecting the pixel point with the largest pixel value from the pixel points corresponding to the candidate points and the pixel points corresponding to the positions in the video frames as a target pixel point, and according to the position corresponding to the target pixel point and the pixel value of the target video frame.

In one embodiment, the select and update module 908 is further configured to obtain a number of trails; and selecting a tailing number of adjacent video frames from the plurality of video frames, updating the pixel points at the target positions in the target video frames, and obtaining the updated target video frames when the updating is completed.

In one embodiment, the selecting and updating module 908 is further configured to determine a frame number difference value according to the number of trails and the acceleration multiple when the number of trails is greater than the acceleration multiple; determining a first pixel point of a target video frame in a plurality of video frames according to the frame number difference value; acquiring a plurality of new video frames with the frame number equal to the acceleration multiple, and determining a second pixel point in the plurality of new video frames; selecting a pixel point with the maximum pixel value from the determined first pixel point and the determined second pixel point as a replacement pixel point; and updating the pixel point at the target position in the next target video frame according to the replaced pixel point.

The above-described embodiment, by determining a first motion mask between adjacent video frames among a plurality of video frames of a video; determining a first motion area and a second motion area of adjacent video frames according to the first motion mask, and calculating optical flow between the first motion area and the second motion area; the motion mask is used for determining the motion areas of the moving objects in the adjacent video frames so as to determine the optical flow between the motion areas, and compared with the traditional mode of directly calculating the optical flow between the complete video frames, the calculation amount is effectively reduced. Moreover, according to the pixel points corresponding to the optical flows in the first motion area and the optical flows, determining the pixel points corresponding to each candidate point; the candidate points are determined based on the selected pixel points of the optical flow when the optical flow is greater than a first threshold value; selecting a pixel point with the largest pixel value from the pixel points corresponding to each candidate point and the pixel points corresponding to the positions in a plurality of video frames as a target pixel point, selecting the pixel point with the largest pixel value from the pixel points corresponding to each candidate point and the pixel points corresponding to the plurality of video frames according to the position corresponding to the target pixel point and the pixel value, and screening the pixel points participating in video frame processing in a motion area by judging whether the optical flow is larger than a first threshold value, so that the operation amount is further reduced.

In one embodiment, as shown in fig. 10, a video processing apparatus includes: a grouping module 1002, an acquisition module 1004, a first determination module 1006, a second determination module 1008, a processing module 1010, a third determination module 1012, and an update module 1014, wherein:

a grouping module 1002, configured to group a plurality of video frames of a video to obtain at least two video frame groups;

an acquiring module 1004, configured to acquire a first video frame and a second video frame from each video frame group;

a first determining module 1006 for sequentially determining a second motion mask between the first video frame and the second video frame of each group;

a second determination module 1008 for determining an optical flow between the first video frame and the second video frame based on the second motion mask;

a processing module 1010, configured to use optical flows between the first video frame and the second video frame of each group as optical flows corresponding to each video frame group;

a third determining module 1012, configured to determine pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flows in the third motion area of the adjacent video frame in each video frame group and the optical flows in sequence; when the optical flow is larger than the first threshold value, the candidate points are determined based on the selected pixel values of the optical flow; the third motion region and the fourth motion region are motion regions of different video frames in adjacent video frames;

The updating module 1014 is configured to select, from among the determined pixel points, the pixel point with the largest pixel value as the target pixel point, and according to the position and the pixel value corresponding to the target pixel point, the target video frame.

In the above embodiment, at least two video frame groups are obtained by grouping a plurality of video frames of a video; acquiring a first video frame and a second video frame from each video frame group; determining a second motion mask between the first video frame and the second video frame of each group in turn; determining an optical flow between the first video frame and the second video frame based on the second motion mask; taking the optical flow between the first video frame and the second video frame of each group as the optical flow corresponding to each video frame group; the method has the advantages that the plurality of video frames share one optical flow, the calculated amount of the optical flow is effectively saved, and the pixel points corresponding to the candidate points are determined sequentially according to the pixel points corresponding to the optical flow in the third motion area of the adjacent video frames in each video frame group and the optical flow; when the optical flow is larger than the first threshold value, the candidate points are determined based on the selected pixel values of the optical flow; the third motion region and the fourth motion region are motion regions of different video frames in adjacent video frames; and selecting the pixel point with the largest pixel value from the determined pixel points as a target pixel point, and according to the position corresponding to the target pixel point and the pixel value, obtaining a target video frame. The method has the advantages that the long exposure time-delay shooting effect can be realized without additionally inserting frames into the video, the memory space is greatly saved, and further, the video processing efficiency is further improved and the video processing effect is optimized by sharing the optical flow with a plurality of video frames.

The respective modules in the video processing apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a terminal or a server, and the internal structure of the computer device is illustrated as a terminal, and may be shown in fig. 11. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input means. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a video processing method. The display unit of the computer equipment is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device, wherein the display screen can be a liquid crystal display screen or an electronic ink display screen, the input device of the computer equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on a shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by those skilled in the art that the structure shown in fig. 11 is merely a block diagram of a portion of the structure associated with the present application and is not limiting of the computer device to which the present application applies, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

In one embodiment, a computer device is provided that includes a memory having a computer program stored therein and a processor that implements the above embodiments when the processor executes the computer program.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, implements the above embodiments.

In one embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, implements the embodiments described above.

It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data are required to comply with the related laws and regulations and standards of the related countries and regions.

Those skilled in the art will appreciate that implementing all or part of the above-described methods in accordance with the embodiments may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The foregoing examples represent only a few embodiments of the present application, which are described in more detail and are not thereby to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.

Claims

1. A method of video processing, the method comprising:

2. The method of claim 1, wherein prior to determining a first motion mask between adjacent video frames in the plurality of video frames of the video, the method further comprises:

3. The method of claim 1, wherein the calculating optical flow between the first and second regions of motion comprises:

4. The method of claim 3, wherein before determining the pixel corresponding to each candidate point according to the pixel corresponding to the optical flow in the first motion region and the optical flow, the method further comprises:

5. The method according to claim 1, wherein the method further comprises:

6. The method according to claim 1, wherein after obtaining the target video frame according to the position and the pixel value corresponding to the target pixel point, the method further comprises:

obtaining the tailing quantity;

7. The method of claim 6, wherein after obtaining the target video frame according to the position and the pixel value corresponding to the target pixel, the method further comprises:

8. A method of video processing, the method comprising:

9. A video processing apparatus, the apparatus comprising:

10. A video processing apparatus, the apparatus comprising:

11. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 8 when the computer program is executed.

12. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 8.

13. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 8.