WO2021179954A1 - 视频处理方法、装置、设备及存储介质 - Google Patents

视频处理方法、装置、设备及存储介质 Download PDF

Info

Publication number
WO2021179954A1
WO2021179954A1 PCT/CN2021/078642 CN2021078642W WO2021179954A1 WO 2021179954 A1 WO2021179954 A1 WO 2021179954A1 CN 2021078642 W CN2021078642 W CN 2021078642W WO 2021179954 A1 WO2021179954 A1 WO 2021179954A1
Authority
WO
WIPO (PCT)
Prior art keywords
video frame
current
current video
processing
frame
Prior art date
Application number
PCT/CN2021/078642
Other languages
English (en)
French (fr)
Inventor
孟祥飞
颜乐驹
Original Assignee
百果园技术(新加坡)有限公司
孟祥飞
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 百果园技术(新加坡)有限公司, 孟祥飞 filed Critical 百果园技术(新加坡)有限公司
Publication of WO2021179954A1 publication Critical patent/WO2021179954A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234363—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the spatial resolution, e.g. for clients with a lower screen resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H04N21/440263—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by altering the spatial resolution, e.g. for displaying on a connected PDA

Definitions

  • the embodiments of the present application relate to the field of computer vision technology, such as video processing methods, devices, equipment, and storage media.
  • super-resolution processing can be performed on each frame of the video to enlarge the resolution and fill in more details while zooming in to enhance the overall look and feel of the processed video.
  • a deep neural network model in a machine learning algorithm is often used to process video frames, so as to restore low-resolution video frames to high-resolution image frames.
  • the use of the cyclic neural network model can better use the super-resolution processing result of the previous video frame to guide the super-resolution processing process of the current video frame. To ensure that the generated high-resolution video is more realistic.
  • the recurrent neural network model in the related technology is used for super-resolution processing of long videos (that is, the video frame contained is greater than 100 frames)
  • the use of the recurrent network structure often results in some timing in the processing of the static display area in the long video
  • the accumulated error makes the displayed video frame after processing show unnatural graphic distortion.
  • the proposed improvement plan needs to add a special loss function to the training process of the recurrent neural network model.
  • This improvement plan is equivalent to the training process that needs to modify the model, involving the modification of the training data arrangement and the hyperparameters.
  • the entire modification process is more complicated, and the improved scheme cannot repair the trained recurrent neural network model.
  • the improved scheme can only relieve the cumulative error of the timing in the static display area in the long video. , And cannot be completely eliminated.
  • the embodiments of the present application provide video processing methods, devices, equipment, and storage media to provide high-quality videos to video consumers.
  • an embodiment of the present application provides a video processing method, including:
  • super-resolution processing is performed based on the image information of the current video frame in combination with the first preamble information to obtain the first of the current video frame.
  • the first preamble information and the second preamble information are both determined based on the limited processing sequence to which the previous video frame of the current video frame belongs.
  • an embodiment of the present application provides a video processing device, including:
  • a processing sequence determining module configured to determine the limited processing sequence to which the current video frame belongs based on the current frame sequence number of the received current video frame and the predetermined current frame number division set and the next frame number division set;
  • the information processing module is configured to perform super-resolution processing based on the combination of the image information of the current video frame and the first preamble information when the current video frame belongs to both the current limited processing sequence and the next limited processing sequence to obtain the current A first processing result of a video frame, and performing super-resolution processing based on the combination of image information of the current video frame and second preamble information to obtain a second processing result of the current video frame;
  • the information determining module is configured to determine the target output information of the current video frame according to the obtained first processing result of the current video frame and the second processing result of the current video frame, and perform the current video according to the target output information Super-resolution display of frames;
  • the first preamble information and the second preamble information are both determined based on the limited processing sequence to which the previous video frame of the current video frame belongs.
  • an embodiment of the present application provides a computer device, including:
  • At least one processor At least one processor
  • the storage device is set to store at least one program
  • the at least one program is executed by the at least one processor, so that the at least one processor implements the video processing method provided by the embodiment of the first aspect of the present application.
  • an embodiment of the present application provides a computer-readable storage medium with a computer program stored on the computer-readable storage medium.
  • the computer program is executed by a processor, the video provided by the embodiment of the first aspect of the present application is implemented. Approach.
  • the video processing method, device, device, and storage medium when processing the received video frame, first divide the set and the next frame according to the current frame number of the current video frame and the predetermined current frame number The number is divided into sets to determine the limited processing sequence to which the current video frame belongs; and when the current video frame belongs to both the current limited processing sequence and the next limited processing sequence, the image information of the current video frame is combined with the first preamble information to perform super Resolution processing to obtain the first processing result of the current video frame, and super-resolution processing is performed by combining the image information of the current video frame with the second preamble information to obtain the second processing result of the current video frame.
  • the first processing result of the current video frame and the second processing result of the current video frame determine the target output information, and the target output information is equivalent to the result finally displayed to the user after the super-resolution processing of the current video frame.
  • this solution first divides the attribution of a limited processing sequence to the video frame to be subjected to super-resolution processing, and uses the limited processing sequence as a cycle to change the video super-resolution processing required
  • the technical solution introduces overlapping video frames that exist in two consecutive finite processing sequences to pass the comparison
  • the processing of overlapping video frames avoids the jump of the displayed picture of the video frame caused by the change of the preamble information between the two finite processing sequences, so as to enhance the smoothness of the video image display.
  • the high-fidelity display of the still image in the video is ensured.
  • Figure 1 shows the initial display of the 30th video frame to be super-resolution processing in the video
  • Figure 2 shows the effect display of the 30th video frame shown in Figure 1 after being processed by the super-resolution method in the related technology
  • Figure 3 shows the initial display of the 200th video frame to be super-resolution processing in the video
  • Figure 4 shows the effect of processing the 200th video frame shown in Figure 2 with the super-resolution method in the related technology
  • FIG. 5 shows a schematic flowchart of a video processing method provided in Embodiment 1 of the present application
  • FIG. 6 shows a schematic flowchart of a video processing method provided in Embodiment 2 of the present application.
  • FIG. 7 shows an effect display diagram of the video frame shown in FIG. 2 after being processed by the super-resolution processing method provided in this embodiment
  • FIG. 8 shows an implementation flowchart of determining target output information in this embodiment
  • FIG. 9 shows a structural block diagram of a video processing device provided in Embodiment 3 of the present application.
  • FIG. 10 shows a schematic diagram of the hardware structure of a computer device provided in the fourth embodiment of the present application.
  • this embodiment mainly performs super-resolution processing on the video.
  • the super-resolution processing when the super-resolution processing is performed, it is equivalent to zooming in the resolution of the video, and filling more while zooming in Details, to enhance the overall look and feel of the video.
  • the enlargement performed is not an enlargement of the total number of pixels contained in multiple video frames in the video. In fact, it can be regarded as an integral multiple enlargement of the width and height of the video (such as 2 times). Magnification and 4x magnification).
  • the improvement in the look and feel of the video after super-resolution processing can be reflected in two aspects.
  • the original resolution of each video frame in the video is 256*256
  • the super-resolution processing is not performed on it, and the video is placed in full-screen mode
  • a blurry video will be displayed.
  • the original resolution is enlarged to 512*512
  • the clarity of the displayed video is higher than that of the original video; for example, for the original resolution
  • the two video frames exhibit the same definition when displayed in equal proportions.
  • Figure 1 shows the 30th frame of the video to be super-resolution processing.
  • the initial display of the video frame shows the effect of processing the 30th video frame shown in Figure 1 with the super-resolution method in the related technology
  • Figure 3 shows the super-resolution processing in the video
  • the initial display of the 200th video frame shows the effect of the 200th video frame shown in Figure 2 after being processed by the super-resolution method in the related technology.
  • the main content of the video display where the video frames shown in Figures 1 to 4 are displayed is that a girl is playing football.
  • the total length of the entire video is 58 seconds, and the total number of video frames is 1483.
  • the two video frames displayed The frame numbers of are respectively the 30th and 200th frames in the video.
  • Figures 1 and 3 respectively show the original image content of the 30th and 200th frames in the video.
  • the width and height of the 30th frame and the 200th frame are the image content after 4 times magnification.
  • the screen content shown in Figure 2 can still better show the details of the screen.
  • Fig. 3 and Fig. 4 it can be seen that in Fig. 4, based on the super-resolution processing in the related technology, the second still area 11 shown in the 200th frame in Fig. 4 Compared with the image content of the first still area 10 in FIG. 3, significant image distortion has occurred.
  • the reason for the image distortion is that the use of cyclic neural network model for super-resolution processing results in errors in the output information after a long time sequence is accumulated.
  • the video processing method proposed in this embodiment can simply and effectively avoid image distortion in the processing method of the related technology.
  • the following embodiment gives an explanation of the video processing method provided in this embodiment.
  • FIG. 5 shows a schematic flow chart of a video processing method provided by Embodiment 1 of the present application.
  • the method is applicable to the situation where multiple video frames in the video perform super-resolution processing in real time.
  • the method can be executed by a video processing device.
  • the device can be implemented by software and/or hardware, and generally can be integrated on a computer device.
  • the execution subject of the provided method can be regarded as a background server that supports multimedia function application software (including live broadcast application software, video call software, video playback software, etc.) for background service support.
  • multimedia function application software including live broadcast application software, video call software, video playback software, etc.
  • the subject can perform super-resolution processing on the video received in real time or stored in advance.
  • a video processing method provided by Embodiment 1 of the present application includes S101 to S103.
  • S101 Determine the limited processing sequence to which the current video frame belongs according to the current frame sequence number of the received current video frame, the predetermined current frame number division set and the next frame number division set.
  • the current video frame can be regarded as the current video frame to be super-resolution processed, and the current video frame can be a frame in the video uploaded by the video producer, or it can be the real-time video sender.
  • the received current video frame may be a video frame captured in real time in a live video broadcast; or, the received current video frame may be a video frame captured in real time during a video call; or, the received current video frame may be a video frame captured by the user in real time.
  • the video frame currently to be played in the selected video file is regarded as the current video frame to be super-resolution processed, and the current video frame can be a frame in the video uploaded by the video producer, or it can be the real-time video sender.
  • a video frame sent to the video receiver may be a video frame captured in real time in a live video broadcast; or, the received current video frame may be a video frame captured in real time during a video call; or, the received current video frame may be a video frame
  • the video frame generated by the host can be captured in real time; in the video call application, the video frame generated by the video sender can be captured in real time; in addition, in the video playback application, the video selected by the user can be intercepted The video frame to be played in the file.
  • each video frame received by this executive body has a corresponding frame number in the video stream where it is located. In this step, the frame number of the current video frame in the video stream where it is located is marked as the current frame number. .
  • the current frame number division set and the next frame number division set can be understood as a frame number set based on selected frame numbers in the video stream, where the current frame number division set includes The frame sequence number belonging to the current limited processing processing, the next frame number division set includes the frame sequence number that should belong to the next limited processing sequence, and the frame sequence numbers included in the current frame number division set and the next frame number division set can be It is determined in advance according to the finite processing length of the set finite processing sequence (that is, the sequence length value of the set finite processing sequence).
  • this embodiment sets up a limited processing sequence with a limited processing length, and determines which frame numbers of the video frames in the current to-be-processed video stream should be in a limited processing sequence, so that only a limited processing sequence is considered.
  • the super-resolution processing result is cyclically input, and the super-resolution processing result is no longer used as the preamble information to participate in the super-resolution processing of the next finite processing sequence continuous with it, so as to reduce the number of video frames involved in the loop.
  • two consecutive finite processing sequences are recorded as the current finite processing sequence and the next finite processing sequence respectively.
  • this embodiment determines which limited processing sequence the current video frame received in real time belongs to by predetermining the current frame number division set and the next frame number division set.
  • the current frame sequence number of the current video frame exists in the current frame number division set, it can be determined that the current video frame belongs to the current limited processing sequence, or when the current frame sequence number exists in the next frame number division set, the current frame number can be determined The video frame belongs to the next finite processing sequence.
  • the current frame number equivalent to the current video frame is divided into the current frame number division set and the next frame number division set, which can be It is determined that the current video frame belongs to the current limited processing sequence and the next limited processing sequence at the same time.
  • the image information can be composed of pixel values constituting the current video frame, and can be represented by a three-dimensional matrix.
  • Two dimensions of the three-dimensional matrix (the first dimension and the second dimension) can respectively represent the width and height of the current video frame.
  • the remaining dimension (the third dimension) can represent the color (RGB) channel of the current video frame.
  • the information contained in the first preamble information and the second preamble information in this step are not the same.
  • the substance of the first preamble information and the second preamble information can be based on the attribute of the previous video frame of the current video frame.
  • the limited processing sequence is determined.
  • the preamble information required for super-resolution processing in this embodiment is actually related to the processing result of the super-resolution processing performed on the previous video frame of the current video frame to be processed, and then considering that it is in the next limit.
  • the preamble information of the first video frame in the processing sequence no longer continues the processing result of the previous video frame.
  • This embodiment considers the different preamble information used for super-resolution processing according to the attribute of the previous video frame of the current video frame.
  • the limited processing sequence is limited as follows.
  • the first preamble information is the third processing result of the previous video frame;
  • the first preamble information is the first processing result of the previous video frame;
  • the previous video frame of the current video frame is only When it belongs to the current limited processing sequence, the second preamble information is image information represented by all 0s; when the previous video frame of the current video frame belongs to both the current limited processing sequence and the next limited processing sequence, the first The second preamble information is the second processing result of the previous video frame.
  • this embodiment when the current video frame belongs to the current limited processing sequence and the next limited processing sequence at the same time, if the previous video frame adjacent to it only belongs to the current limited processing sequence, it can be considered that the previous video frame only has the current limited processing sequence.
  • the corresponding processing result under the current finite processing sequence the processing result is recorded as the third processing result of the previous video frame, and as the first preamble information of the current video frame; at this time, the current video frame is equivalent to the next finite processing sequence Because the first video frame in the next finite processing sequence no longer uses the preamble information of the current finite processing sequence, this embodiment sets the second preamble information as the initial image information represented by all 0s at this time.
  • the previous video frame adjacent to the current video frame belongs to both the current finite processing sequence and the next finite processing sequence, it is equivalent to the previous video frame being an overlapping frame.
  • the previous video frame is the sum of the current finite processing sequence and The next finite processing sequence corresponds to a super-resolution processing result
  • the previous video frame corresponds to a super-resolution processing result of the current finite processing sequence.
  • the processing result can be recorded as the first processing result of the previous video frame
  • the previous video frame corresponds to a super-resolution processing result for the next limited processing sequence.
  • This processing result can be recorded as the second processing result of the previous video frame, and the first processing result of the previous video frame is taken as the current
  • the first preamble information of the video frame uses the second processing result of the previous video frame as the second preamble information.
  • the current video frame belongs to the current limited processing sequence and the next limited processing sequence at the same time, the corresponding previous video frame does not belong to the next limited processing sequence. Therefore, there is no such situation.
  • the first preamble information and the second preamble information It should be noted that the operation of the super-resolution processing performed in this step has not changed, and it can be achieved by using the currently trained recurrent neural network model.
  • two processing results can be obtained respectively.
  • the two processing results are recorded as the first processing result of the current video frame and the current video frame.
  • This step can perform weighted summation processing on the two processing results, and use the weighted summation result as the final result of the super-resolution processing of the current video frame.
  • the final result is recorded as the target output information, and this step can directly
  • the target output information is the image information displayed to the video receiving user after the super-resolution processing of the current video frame.
  • this embodiment may respectively determine the weight value of the two processing results of the current video frame when weighting, and compare the first processing result of the current video frame and the second processing result of the current video frame according to the corresponding weight values.
  • the processing results are weighted and summed, and the output result after the weighted summation is recorded as the target output information, where the weight value of the two processing results can be determined in the form of an interpolation function (such as a linear interpolation function).
  • this solution first performs the attribution of a limited processing sequence to the video frame to be super-resolution processed, and uses the limited processing
  • the sequence is a cycle to change the preamble information required for video super-resolution processing, so as to eliminate the image distortion in the static area of the video accumulated over time in the related technical processing; at the same time, this technical solution introduces the overlap existing in two consecutive A video frame in a finite processing sequence to avoid the jump of the displayed picture of the video frame caused by the change of the preamble information between the two finite processing sequences by processing the overlapped video frames, thereby enhancing the video image The smoothness of the display. Therefore, while realizing the high-efficiency super-resolution processing of the video, the high-fidelity display of the still area picture in the video is ensured.
  • determining the limited processing sequence to which the current video frame belongs based on the current frame sequence number of the received current video frame, the predetermined current frame number division set, and the next frame number division set including: obtaining Receive the current frame sequence number of the current video, and perform a traversal search respectively in the current frame number division set and the next frame number division set according to the current frame sequence number; if the current frame sequence number only exists in the current frame number division Set, it is determined that the current video frame only belongs to the current limited processing sequence; if the current frame number exists in the current frame number division set and at the same time exists in the next frame number division set, then the current video frame is determined It belongs to the current limited processing sequence and the next limited processing sequence at the same time; if the current frame sequence number only exists in the next frame number division set, it is determined that the current video frame only belongs to the next limited processing sequence.
  • the process of determining the attribution of the limited processing sequence of the current video frame is given, which is mainly performed by dividing the current frame number of the current video frame and the current frame number into the set and the next frame number into the frame in the set.
  • the sequence numbers are matched to determine whether the current video frame is an overlapping frame that belongs to both the current limited processing sequence and the next limited processing sequence, thereby ensuring the effective progress of subsequent super-resolution processing.
  • Fig. 6 shows a schematic flow chart of a video processing method provided in the second embodiment of the present application.
  • the second embodiment is detailed on the basis of the above-mentioned embodiment.
  • super-resolution processing is performed based on the image information of the current video frame and the third preamble information, where the third preamble information is based on the previous video frame of the current video frame
  • the attribution of the limited processing sequence is determined; the third processing result of the current video frame obtained after processing is determined as the target output information of the current video frame, and the super-resolution display of the current video frame is performed according to the target output information .
  • This embodiment also includes: when the current video frame and the corresponding previous video frame belong to the next limited processing sequence, after monitoring that the current video frame is displayed in super-resolution, record that the current video frame belongs to the new The current limited processing sequence; the next frame number division set is recorded as the new current frame number division set, and the new next frame number division set is determined according to the new current frame number division set.
  • This embodiment also includes: recording the current video frame as the previous video frame, and when a new current video frame is received, returning to continue the operation of determining the limited processing sequence to which the new current video frame belongs.
  • a video processing method provided in the second embodiment of the present application includes S201 to S211.
  • S201 Determine the limited processing sequence to which the current video frame belongs according to the current frame sequence number of the received current video frame, the predetermined current frame number division set and the next frame number division set.
  • the following S202, S204, and S205 and other steps respectively associated with the above steps respectively show that the current video frame belongs to the current limited processing sequence and the next limited processing sequence at the same time, only belongs to the current limited processing sequence, and only It belongs to the related operations of super-resolution processing when it belongs to the next finite processing sequence.
  • a pre-trained recurrent neural network model can be optionally used for super-resolution processing.
  • the cyclic neural network model takes the image information of the video frame to be processed and the given preamble information as input data, and the output result of the cyclic neural network model is used as the processing result of super-resolution processing.
  • this embodiment optionally uses the image information of the current video frame and the first preamble information as the input of the cyclic neural network model
  • the output data obtained from the data is used as the first processing result of the current video frame; at the same time, optionally, the output data obtained by using the image information of the current video frame and the second preamble information as the input data of the cyclic neural network model is used as the current video The second processing result of the frame.
  • the image information and preamble information of the video frame are used as the cyclic neural network model to perform super-resolution processing.
  • the output processing result is also a three-dimensional matrix.
  • the first dimension and the second dimension of the three-dimensional matrix are also respectively. Represents the width and height of the video frame, but compared with the original image information of the video frame, the number of elements contained in these two dimensions is an integer multiple of the original image information in the two dimensions, and the specific value of the integer multiple is constructed It has been set when the neural network is looped. In addition, the number of color channels represented by its third dimension has not changed.
  • the processing result output by the recurrent neural network model is used as the preorder information of the following video frame
  • the processing result represented by the three-dimensional matrix needs to be post-processed.
  • the processing result is 512*512*3
  • the input data of the input recurrent neural network model must ensure that the number of elements in the first dimension and the second dimension is the same.
  • this embodiment needs to convert the three-dimensional matrix 512*512*3 into 256*256*12 to ensure that the processing result is the first in the form of adding color channels.
  • the number of elements in the dimension and the second dimension is the same as the number of specified elements, and the three-dimensional matrix 256*256*12 can be used as the pre-order information to be input next time.
  • S203 Determine the target output information of the current video frame according to the first processing result of the current video frame and the second processing result of the current video frame obtained after processing, and execute S207.
  • the weight values of the first processing result of the current video frame and the second processing result of the current video frame can be determined separately, and the weighted summation of the weight value and the corresponding processing result can obtain the target that can be finally displayed to the target user Output information.
  • the current video frame when the current video frame only belongs to the current limited processing sequence, the current video frame only needs to perform super-resolution processing once.
  • the processing result obtained by the super-resolution processing may be recorded as the first of the current video frame.
  • this embodiment optionally uses the image information of the current video frame and the third preamble information as the input data of the cyclic neural network model, and the processed output data is the third processing result of the current video frame.
  • the third preamble information also needs to be determined based on the limited processing sequence to which the previous video frame of the current video frame belongs.
  • the previous video frame of the current video frame only belongs to the current limited processing sequence
  • the previous video frame also only outputs one piece of data information recorded as the third processing result. Therefore, the third preamble information can be It is directly the third processing result of the previous video frame; in addition, because the current video frame only belongs to the current limited processing sequence, the previous video frame cannot belong to the current limited processing sequence and the next limited processing sequence at the same time. Therefore, it is also There is no third preamble in this situation; similarly, the previous video frame before it cannot only belong to the next limited processing sequence, and there is also no third preamble in this situation. ; But the previous video frame of the current video frame may be empty. In this case, the image information represented by all 0s can be used as the third preamble information.
  • the current video frame when the current video frame only belongs to the next finite processing sequence, the current video frame also only needs to perform super-resolution processing once in combination with the fourth preamble information, and this embodiment may also perform super-resolution processing.
  • the processing result is recorded as the fourth processing result of the current video frame. Therefore, this embodiment optionally uses the image information of the current video frame and the fourth preamble information as the input data of the cyclic neural network model, and the processed output data is still recorded as the fourth processing result of the current video frame.
  • the fourth preamble information also needs to be determined based on the limited processing sequence to which the previous video frame of the current video frame belongs.
  • the current video frame only belongs to the next limited processing sequence, considering the existence of overlapping frames, the previous video frame of the current video frame cannot belong to the current limited processing sequence only, so there is no such case
  • the fourth preamble information when the previous video frame of the current video frame belongs to both the current limited processing sequence and the next limited processing sequence, it is equivalent to two super-resolution processing of the previous video frame, and the processing results are respectively Recorded as the first processing result of the previous video frame and the second processing result of the previous video frame.
  • the second processing result of the previous video frame is regarded as the corresponding processing of the previous video frame in the next finite processing sequence
  • this embodiment can record the fourth preamble information as the second processing result of the previous video frame; in addition, when the previous video frame of the current video frame only belongs to the next limited processing sequence, the previous video frame is only The super-resolution processing is performed once, and the processed processing result is recorded as the fourth processing result of the previous video frame. Therefore, the fourth preamble information in this case is the fourth processing result of the previous video frame.
  • S206 Determine the obtained third processing result of the current video frame or the fourth processing result of the current video frame as the target output information of the current video frame.
  • the obtained third processing result of the current video frame can be directly determined as the target output information of the current video frame; the current video frame only belongs to the next limited processing sequence At this time, the obtained fourth processing result of the current video frame is directly determined as the target output information of the current video frame.
  • S207 Perform super-resolution display of the current video frame according to the target output information.
  • the target output information obtained in this step is used as the final processing result of the super-resolution processing of the current video frame, and is based on the target The output information performs super-resolution display of the screen content of the current video frame.
  • Figure 7 shows the The video frame shown in Fig. 2 is displayed with the effect of the super-resolution processing method provided in this embodiment. Assuming that the current video frame being processed is the 200th frame in the video, as shown in Fig. 7, based on this In the video picture corresponding to the 200th frame after the super-resolution processing provided by the embodiment, the third still area 12 can still display the details of the picture clearly compared with the second still area 11 in FIG. 2 without image distortion.
  • This step is equivalent to determining the timing of the new next frame number division set, and the timing determination can be achieved by determining whether the current video frame only belongs to the next limited processing sequence. It is understandable that when the current video frame only belongs to the next limited processing sequence, it is equivalent to that the video frames in the current limited processing sequence have completed the super-resolution processing, and the existing current frame number division set is no longer required.
  • the current frame number division set also includes video frames that may exist in the next frame number division set adjacent to the current frame number division set. Therefore, as long as it is determined that the current video frame only belongs to the next limited processing sequence, S209 can be started to update the next frame number division set.
  • the current video frame only belongs to the current limited processing sequence or the current video frame belongs to both the current limited processing sequence and the next limited processing sequence, it indicates that there are still unprocessed frame numbers in the current frame number division set, and S211 is required to re-align The new current video frame undergoes super-resolution processing.
  • the current video frame is displayed in super-resolution, it is equivalent to no connection with the related processing result of the previous video frame.
  • the current video frame can be displayed after the super-resolution display is completed.
  • the next limited processing sequence to which the frame belongs is recorded as the new current limited processing sequence.
  • next frame number division set can be recorded as the new current frame number division set to perform a new next frame. Determination of the number division set.
  • the determining a new next frame number division set according to the new current frame number division set may include: obtaining the sequence length value of the set limited processing sequence and the overlap contained in the limited processing sequence The number of overlapping frames of the frame, and the first frame sequence number in the new current frame number division set is obtained; according to the sequence length value, the number of overlapping frames, and the first frame sequence number, the number is determined to be the sequence length value Based on the determined number of new frame sequence numbers, a new next frame number division set is formed.
  • the limited processing length of the limited processing sequence (that is, the sequence length value of the limited processing sequence) used in this embodiment may be preset according to empirical values, and the overlapping frames contained in the limited processing sequence (ie, The number of overlapping frames (video frames existing in two adjacent finite processing sequences) may also be preset according to empirical values.
  • the set formula can be used to determine the first frame sequence number in the new next frame number division set Based on the newly determined first frame number and the limited processing length, other frame numbers in the new next frame number division set can be determined.
  • S t,1 1, N is 8, and k is 2, it can be determined that S (t+1), 1 is 7, and the new next frame number division set S (t+1)
  • the remaining frame numbers are 8, 9, 10, 11, 12, 13, and 14.
  • the current video frame can be changed to the previous video frame, and the newly received video frame can be recorded as the new current video frame, and then Return to S201 and continue the super-resolution processing operation in a loop until no new video frames are received.
  • the corresponding first processing result of the current video frame, the third processing result of the current video frame, the fourth processing result of the current video frame, and the current video is also changed to the first processing result of the previous video frame, the third processing result of the previous video frame, the fourth processing result of the previous video frame, and the second processing result of the previous video frame.
  • the second embodiment of the present application provides a video processing method, which provides the implementation process of super-resolution processing of the current video frame under the condition of belonging to three limited processing sequences; at the same time, it also provides a set of new next frame number divisions.
  • the determination process and the process of super-resolution processing of the video frame loop Using this method, the preamble information required for video super-resolution processing is changed with a finite processing sequence as a cycle, and video frames that overlap existing in two continuous finite processing sequences are introduced to pass the processing of the overlapped video frames To avoid the jump of the displayed picture of the video frame caused by the change of the preamble information between the two finite processing sequences, so as to enhance the smoothness of the video image display.
  • this embodiment does not need to change the super-resolution processing used
  • the training data and training methods of the cyclic neural network model can easily and effectively avoid image distortion, and realize the high efficiency and high fidelity of super-resolution processing.
  • FIG. 8 shows an implementation flow chart for determining target output information in this embodiment.
  • the second processing result, determining the target output information of the current video frame includes S2031 to S2033.
  • This embodiment considers setting overlapping frames that are both in the current finite processing sequence and the next finite processing sequence, and performing super-resolution processing on the overlapping frames twice to achieve smooth display of video pictures adjacent to the two finite processing sequences.
  • the two super-resolution processing results obtained from the current video frame as the overlapping frame respectively correspond to the current limited processing sequence and the next limited processing sequence.
  • This embodiment considers the first processing result of the current video frame and the current video
  • the influence of the second processing result of the frame on the final result of the current video frame respectively occupies a certain proportion, and the proportion respectively occupied is equivalent to the corresponding weight value, and the present embodiment can influence the final result according to the two results to two degrees. Set the appropriate weight value as a result.
  • the following operations may be used to determine the first weight value and the second weight value: obtain the sequence number corresponding to the current video frame in the current frame number division set; according to the sequence number and the selected sequence number
  • the interpolation function determines the weighting coefficient of the current video frame, where the weighting coefficient is greater than 0 and less than 1; the weighting coefficient is taken as the first weight value of the first processing result corresponding to the current video frame, and 1
  • the difference between the weighting coefficient and the weighting coefficient is used as the second weighting value of the second processing result corresponding to the current video frame.
  • the frame numbers in the current frame number division set and the next frame number division set are arranged in a time sequence order, and each frame sequence number can also correspond to a sequence number based on the time sequence order; in addition, this embodiment is At least one overlapping video frame is set in the limited processing sequence, and the frame numbers of the overlapping multiple video frames are continuous; when determining that the current video frame is an overlapping frame, if it is determined that there are other overlapping frames before it, according to the current The sequence numbers of the video frame and other consecutive overlapping frames before it can be known how many overlapping frames the current video frame is.
  • this embodiment uses an interpolation function to determine the degree of influence of each overlapped frame on the calculation of the target output information in the current finite processing sequence (this embodiment records the degree of influence as a weighting coefficient), and each overlapped frame is currently limited.
  • the corresponding weighting coefficient in the processing sequence can be directly regarded as the first weight value (greater than 0 and less than 1) corresponding to the first processing result of the overlapping frame, and the difference between 1 and the first weight value can be regarded as the The second weight value of the second processing result corresponding to the overlapping frame in the next finite processing sequence.
  • the interpolation function may be selected as a linear interpolation function.
  • the linear interpolation function can be expressed as: Wherein, P i represents the i-th weighting coefficients corresponding to the overlapping frames, i ⁇ [1, p], p is the number of overlapping frames overlapping a finite frame included in the processing sequence.
  • the weighting coefficient corresponding to the first overlapping frame is 0.67, which is equivalent to the first weighting value corresponding to the first processing result of the overlapping frame, which is 0.67.
  • the second weight value corresponding to the second processing result of the overlapping frame is 0.33; and the weighting coefficient corresponding to the second overlapping frame is 0.33, which is equivalent to that the first processing result of the overlapping frame corresponds to the first weight value of 0.33.
  • the second weight value corresponding to the second processing result of is 0.67.
  • the current video frame as an overlapping frame, after knowing the sequence number of its current frame number in the current frame number division set, and knowing the number of overlapping frames before it, the current video frame can be known
  • the video frame is the number of overlapping frames, and the corresponding weighting coefficient can be determined according to the interpolation function, and then the first weight value corresponding to the first processing result of the current video frame and the second processing result of the current video frame can be determined The corresponding second weight value.
  • the weighted sum processing performed is equivalent to multiplying 0.33 by 0.33.
  • this step may directly use the data information processed by the weighted summation in the above steps as the final target output information of the super-resolution processing.
  • the foregoing optional embodiment of the second embodiment provides an implementation process for determining the target output information when the current video frame is an overlapping frame.
  • the realization of the above method takes into account the distribution characteristics of the influence of overlapping frames on the final processing result after super-resolution processing in two adjacent finite processing sequences, and determines the weight values of the two processing results, and obtains them based on the determined weight values.
  • the target output information better avoids the occurrence of screen jumps at the adjacent two finite processing sequences, and ensures the smoothness of the video screen display.
  • Fig. 9 shows a structural block diagram of a video processing device provided in the third embodiment of the present application.
  • the device is suitable for super-resolution processing of video frames.
  • the device can be implemented by hardware and/or software, and generally Integrated on a computer device, as shown in FIG. 9, the device includes: a processing sequence determination module 31, an information processing module 32, and an information determination module 33.
  • the processing sequence determining module 31 is configured to determine the limited processing sequence to which the current video frame belongs according to the current frame sequence number of the received current video frame and the predetermined current frame number division set and the next frame number division set;
  • the information processing module 32 is configured to perform super-resolution processing based on the combination of the image information of the current video frame and the first preamble information when the current video frame belongs to both the current limited processing sequence and the next limited processing sequence.
  • the first processing result of the current video frame and performing super-resolution processing based on the combination of image information of the current video frame and the second preamble information to obtain the second processing result of the current video frame;
  • the information determining module 33 is configured to determine the target output information of the current video frame according to the first processing result of the current video frame and the second processing result of the current video frame obtained after processing, and perform all operations according to the target output information.
  • the first preamble information is different from the second preamble information.
  • the third embodiment of the present application provides a video processing device that uses a limited processing sequence as a cycle to change the preamble information required for video super-resolution processing, so as to eliminate the accumulation of time in the still area of the video in the related technical processing.
  • this technical solution introduces overlapping video frames existing in two consecutive finite processing sequences to avoid the change of the preamble information between the two finite processing sequences by processing the overlapped video frames The transition of the picture displayed by the video frame in order to enhance the smoothness of the video image display.
  • the high-fidelity display of the still image in the video is ensured.
  • FIG. 10 shows a schematic diagram of the hardware structure of a computer device provided in the fourth embodiment of the present application.
  • the computer device includes a processor and a storage device. At least one instruction is stored in the storage device, and the instruction is executed by the processor, so that the computer device executes the video processing method described in the foregoing method embodiment.
  • the computer equipment may include: a processor 40, a storage device 41, a display screen 42, an input device 43, an output device 44, and a communication device 45.
  • the number of processors 40 in the computer device may be at least one, and one processor 40 is taken as an example in FIG. 4.
  • the number of storage devices 41 in the computer device may be at least one, and one storage device 41 is taken as an example in FIG. 10.
  • the processor 40, the storage device 41, the display screen 42, the input device 43, the output device 44, and the communication device 45 of the computer equipment may be connected by a bus or other means. In FIG. 4, the connection by a bus is taken as an example.
  • the storage device 41 can be configured to store software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the embodiments of the present application (for example, in the video processing device provided in the above embodiments)
  • the storage device 41 may mainly include a storage program area and a storage data area.
  • the storage program area may store an operating device and an application program required for at least one function; the storage data area may store data created according to the use of computer equipment, and the like.
  • the storage device 41 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.
  • the storage device 41 may include a memory remotely provided with respect to the processor 40, and these remote memories may be connected to the computer device through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
  • the display screen 42 is set to display data according to instructions of the processor 40, and is also set to receive touch operations on the display screen 42 and send corresponding signals to the processor 40 or other devices.
  • the display screen 42 is an infrared screen, it also includes an infrared touch frame.
  • the infrared touch frame is arranged around the display screen 42, and it can also be set to receive infrared signals and send the infrared signals to the processor. 40 or other computer equipment.
  • the communication device 45 is configured to establish a communication connection with other computer equipment, and it may be a wired communication device and/or a wireless communication device.
  • the input device 43 can be set to receive input digital or character information, and to generate key signal input related to the user settings and function control of the computer equipment. It can also be a camera set to obtain images and a pickup computer to obtain audio in video data. equipment.
  • the output device 44 may include video computer equipment such as a display screen and audio computer equipment such as a speaker. It should be noted that the composition of the input device 43 and the output device 44 can be set according to actual conditions.
  • the processor 40 executes various functional applications and data processing of the computer equipment by running the software programs, instructions, and modules stored in the storage device 41, that is, realizes the above-mentioned video processing method.
  • the processor 40 executes at least one program stored in the storage device 41, the following operations are implemented: determine the current frame number according to the current frame number of the received current video frame and the predetermined current frame number division set and the next frame number division set.
  • the limited processing sequence to which the video frame belongs when the current video frame belongs to both the current limited processing sequence and the next limited processing sequence, super-resolution processing is performed based on the combination of the image information of the current video frame and the first preamble information to Obtain the first processing result of the current video frame, and perform super-resolution processing based on the combination of the image information of the current video frame and the second preamble information to obtain the second processing result of the current video frame; according to the current video obtained after processing
  • the first processing result of the frame and the second processing result of the current video frame determine the target output information of the current video frame, and perform super-resolution display of the current video frame according to the target output information.
  • the embodiments of the present application also provide a computer-readable storage medium.
  • a program in the storage medium is executed by a processor of a computer device, the computer device can execute the video processing method described in the foregoing embodiment.
  • the video processing method described in the foregoing embodiment includes: determining the to which the current video frame belongs according to the current frame sequence number of the received current video frame and the predetermined current frame number division set and the next frame number division set.
  • this application can be implemented by software and necessary general-purpose hardware, and of course, it can also be implemented by hardware.
  • the technical solution of this application essentially or the part that contributes to the related technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, Read-Only Memory (ROM), Random Access Memory (RAM), Flash memory (FLASH), hard disk or optical disk, etc., including several instructions to make a computer device (can be a robot, personal A computer, a server, or a network device, etc.) execute the video processing method described in any embodiment of the present application.
  • a computer device can be a robot, personal A computer, a server, or a network device, etc.
  • each part of this application can be implemented by hardware, software, firmware, or a combination thereof.
  • multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.
  • a suitable instruction execution device For example, if it is implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: Discrete logic circuits, ASICs with suitable combinational logic gate circuits, Programmable Gate Array (PGA), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Television Systems (AREA)
  • Transforming Electric Information Into Light Information (AREA)

Abstract

本申请实施例公开了视频处理方法、装置、设备及存储介质。该方法包括根据所接收当前视频帧的当前帧序号及预先确定的当前帧号划分集合和下一帧号划分集合，确定当前视频帧归属的有限处理序列；在当前视频帧同时属于当前有限处理序列和下一有限处理序列时，基于当前视频帧的图像信息与第一前序信息结合进行超分辨率处理以获得当前视频帧的第一处理结果，且基于当前视频帧的图像信息与第二前序信息结合进行超分辨率处理以获得当前视频帧的第二处理结果；根据获得的当前视频帧的第一处理结果和当前视频帧的第二处理结果，确定当前视频帧的目标输出信息，并按照目标输出信息进行当前视频帧的超分辨率显示。

Description

视频处理方法、装置、设备及存储介质
本申请要求在2020年3月13日提交中国专利局、申请号为202010175136.1的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及计算机视觉技术领域，例如涉及视频处理方法、装置、设备及存储介质。
背景技术
在计算机视觉领域，可通过对视频的各帧进行超分辨率处理来放大分辨率以及在放大的同时填充更多细节，以增强所处理视频的整体观感。
相关技术中对视频进行超分辨率处理的实现方案中，往往使用机器学习算法中的深度神经网络模型对视频帧进行处理，以此将低分辨率的视频帧恢复为高分辨率的图像帧。且研究发现相比如常规超分辨率处理的深度神经网络模型，采用循环神经网络模型能够更好的利用前一视频帧的超分辨率处理结果来指导当前视频帧的超分辨率处理过程，以此来保证所生成的高分辨率视频更为逼真。然而，采用相关技术中循环神经网络模型对长视频(即所包含视频帧大于100帧)进行超分辨率处理时，使用循环网络结构往往会在对长视频内静止显示区域的处理中产生一些时序累计的误差，使得处理后所显示的视频帧表现出不自然的图形畸变。
为避免该问题所提出的改进方案需要在循环神经网络模型的训练过程中加入一种特殊的损失函数，该改进方案相当于需要修改模型的训练过程，涉及到训练数据排列方式的修改和超参数的选取，整个修改过程较为复杂，且该改进方案无法对已训练好的循环神经网络模型进行修复，此外，该改进方案也只能对长视频内处于静止显示区域的时序累积误差起到缓解作用，并无法彻底消除。
发明内容
本申请实施例提供了视频处理方法、装置、设备及存储介质，以向视频的消费用户提供高质量视频。
第一方面，本申请实施例提供了一种视频处理方法，包括：
根据所接收当前视频帧的当前帧序号及预先确定的当前帧号划分集合和下一帧号划分集合，确定所述当前视频帧归属的有限处理序列；
响应于确定所述当前视频帧同时属于当前有限处理序列和下一有限处理序列，基于所述当前视频帧的图像信息与第一前序信息结合进行超分辨率处理以获得当前视频帧的第一处理结果，且基于所述当前视频帧的图像信息与第二前序信息结合进行超分辨率处理以获得当前视频帧的第二处理结果；
根据获得的当前视频帧的第一处理结果和当前视频帧的第二处理结果，确定所述当前视频帧的目标输出信息，并按照所述目标输出信息进行所述当前视频帧的超分辨率显示；
其中，所述第一前序信息和第二前序信息均基于所述当前视频帧的前一视频帧归属的有限处理序列确定。
第二方面，本申请实施例提供一种视频处理装置，包括：
处理序列确定模块，设置为根据所接收当前视频帧的当前帧序号及预先确定的当前帧号划分集合和下一帧号划分集合，确定所述当前视频帧归属的有限处理序列；
信息处理模块，设置为当所述当前视频帧同时属于当前有限处理序列和下一有限处理序列时，基于所述当前视频帧的图像信息与第一前序信息结合进行超分辨率处理以获得当前视频帧的第一处理结果，且基于所述当前视频帧的图像信息与第二前序信息结合进行超分辨率处理以获得当前视频帧的第二处理结果；
信息确定模块，设置为根据获得的当前视频帧的第一处理结果和当前视频帧的第二处理结果，确定所述当前视频帧的目标输出信息，并按照所述目标输 出信息进行所述当前视频帧的超分辨率显示；
其中，所述第一前序信息和第二前序信息均基于所述当前视频帧的前一视频帧归属的有限处理序列确定。
第三方面，本申请实施例提供了一种计算机设备，包括：
至少一个处理器；
存储装置，设置为存储至少一个程序；
所述至少一个程序被所述至少一个处理器执行，使得所述至少一个处理器实现本申请第一方面实施例提供的视频处理方法。
第四方面，本申请实施例提供了一种计算机可读存储介质，计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时，实现本申请第一方面实施例提供的视频处理方法。
本申请实施例提供的视频处理方法、装置、设备及存储介质中，对所接收的视频帧进行处理时，首先根据当前视频帧的当前帧序号以及预先确定的当前帧号划分集合及下一帧号划分集合，来确定当前视频帧归属的有限处理序列；并在当前视频帧同时属于当前有限处理序列和下一有限处理序列时，将当前视频帧的图像信息与第一前序信息结合进行超分辨率处理以获得当前视频帧的第一处理结果，且将当前视频帧的图像信息与第二前序信息结合进行超分辨率处理以获得当前视频帧的第二处理结果，之后可根据获得的当前视频帧的第一处理结果和当前视频帧的第二处理结果确定出目标输出信息，而该目标输出信息就相当于对该当前视频帧超分辨率处理后最终展示给用户的结果。相比于相关技术中的视频超分辨率处理，本方案首先对待进行超分辨率处理的视频帧进行有限处理序列的归属划分，并以有限处理序列为周期来变更视频超分辨率处理时所需的前序信息，以此来消除相关技术处理中随时间累计而在视频静止区域产生的图像畸变；同时，本技术方案引入重叠存在于连续两个有限处理序列中的视频帧，以通过对所重叠视频帧的处理来避免因两个有限处理序列间前序信息的变更而引起的视频帧所显示画面的跳变，以此来增强视频图像显示的平滑 性。由此，在实现视频高效率超分辨处理的同时，保证了视频中静止区域图像的高保真显示。
附图说明
图1给出了视频中待进行超分辨率处理的第30帧视频帧的初始显示图；
图2给出了对图1所显示第30帧视频帧采用相关技术中超分辨率方法处理后的效果显示图；
图3给出了视频中待进行超分辨率处理的第200帧视频帧的初始显示图；
图4给出了对图2所显示第200视频帧采用相关技术中超分辨率方法处理后的效果显示图；
图5给出了本申请实施例一提供的一种视频处理方法的流程示意图；
图6给出了本申请实施例二提供的一种视频处理方法的流程示意图；
图7给出了对图2所显示视频帧采用本实施例提供的超分辨率处理方法处理后的效果显示图；
图8给出了本实施例中进行目标输出信息确定的一种实现流程图；
图9给出了本申请实施例三提供的一种视频处理装置的结构框图；
图10给出了本申请实施例四提供的一种计算机设备的硬件结构示意图。
具体实施方式
需要说明的是，本实施例主要对视频进行超分辨率处理，对于一个视频而言，其在进行超分辨率处理时，相当于将视频的分辨率进行放大，并在放大的同时填充更多细节，来增强视频的整体观感，所进行的放大并不是对视频中多个视频帧所包含总像素数量的放大，实际可看做对视频的宽和高均进行整数倍的放大(如2倍放大和4倍放大)。
对于视频的观看用户而言，对视频进行超分辨率处理后的观感提升可从两方面来体现。示例性的，假设视频中每个视频帧的原有分辨率为256*256，在不 对其进行超分辨率处理时，将视频置于全屏模式下，将会展现一个较模糊的视频，而进行超分辨率处理后，如将原有分辨率放大至512*512后，在将视频置于全屏模式下时，所展现视频的清晰度高于原始视频的清晰度；又如，对于原有分辨率分别为256*256，以及512*512的视频帧而言，其在等比例显示时两视频帧所展现出的清晰度相同，若不进行超分辨率处理直接将256*256的视频帧放大至512*512，则将会展现一个较模糊的画面，若进行了超分辨率处理，将256*256的视频放大至512*512时，所展现的仍是一个清晰的画面。由此可知，进行超分辨率处理的视频能够给观看用户带来更好的观感。
相关技术中超分辨率处理后会因时序累计的误差，造成所显示视频帧静止区域显示畸变的问题，为便于更好的理解，图1给出了视频中待进行超分辨率处理的第30帧视频帧的初始显示图；图2给出了对图1所显示第30帧视频帧采用相关技术中超分辨率方法处理后的效果显示图；图3给出了视频中待进行超分辨率处理的第200帧视频帧的初始显示图；图4给出了对图2所显示第200视频帧采用相关技术中超分辨率方法处理后的效果显示图。其中，图1～图4中所展示视频帧所处视频展示的主要内容为一个女生在踢足球，整个视频的时间总长度为58秒，视频总帧数为1483，所显示的两个视频帧的帧序号分别为视频中的第30帧和第200帧。
由图1～图4可以看出，图1和图3分别展示了视频中第30帧和第200帧的原始图像内容，图2和图4分别给出了基于相关技术中超分辨率处理方法对第30帧和第200帧的宽和高进行4倍放大处理后的图像内容。将图1和图2进行比对，可以看出对第30帧图像采用相关技术中的超分辨率处理后，图2所显示的画面内容还能够较好的展现画面细节，此时因时序累积较短还不存在图像畸变；然而，将图3和图4进行比对，可以看出在图4中，基于相关技术中超分辨率处理后，图4中第200帧所显示第二静止区域11相较于图3内第一静止区域10的画面内容发生了明显的图像畸变。该图像畸变的原因在于采用循环神经网络模型进行超分辨率处理因较长时间的时序累积后，使得输出信息产生了误 差。
本实施例提出的视频处理方法，能够简单有效的避免相关技术的处理方法中图像畸变的情况，下述实施例给出了本实施例所提供视频处理方法的阐述。
实施例一
图5给出了本申请实施例一提供的一种视频处理方法的流程示意图，该方法适用于视频中的多个视频帧实时进行超分辨率处理的情况，该方法可以由视频处理装置执行，其中，该装置可以软件和/或硬件实现，并一般可集成在计算机设备上。
需要说明的是，本实施例可将所提供方法的执行主体看作给多媒体功能类应用软件(包括直播应用软件、视频通话软件以及视频播放软件等)进行后台服务支持的后台服务器，通过该执行主体可以对实时接收的或者预先存储的视频进行超分辨率处理。
如图5所示，本申请实施例一提供的一种视频处理方法，包括S101至S103。
S101、根据所接收当前视频帧的当前帧序号及预先确定的当前帧号划分集合和下一帧号划分集合，确定所述当前视频帧归属的有限处理序列。
在本实施例中，所述当前视频帧可看做当前待进行超分辨率处理的视频帧，所述当前视频帧可以是视频生产者所上传视频中的一帧，也可以是视频发送方实时发送给视频接收方的一个视频帧。可选的，所接收的当前视频帧可以为视频直播中实时捕获的视频帧；或者，所接收的当前视频帧为视频通话中实时捕获的视频帧；或者，所接收的当前视频帧为用户所选定视频文件中当前待播放的视频帧。
可以知道的是，在视频直播应用中可以实时捕获主播产生的视频帧；在视频通话应用中可以实时捕获视频发送方产生的视频帧；此外，在视频播放应用中，可以截取用户所选定视频文件中待播放的视频帧。同样可以知道的是，本执行主体所接收的每个视频帧在所处的视频流中均具备相应的帧序号，本步骤 将当前视频帧在所处视频流中的帧序号记为当前帧序号。
在本实施例中，所述当前帧号划分集合和下一帧号划分集合可以理解为基于从视频流中选定的帧序号构成的帧号集合，其中，当前帧号划分集合中包括了应当归属于当前有限处理处理的帧序号，下一帧号划分集合中包括应当归属于下一有限处理序列的帧序号，而且当前帧号划分集合以及下一帧号划分集合中所包括的帧序号可以根据所设置有限处理序列的有限处理长度(即所设定有限处理序列的序列长度值)来预先确定。
需要说明的是，相关技术中的超级分辨率处理采用循环神经网络实现时，如果待处理的视频帧数较多，随着视频帧超分辨率的结果不断作为前序信息并入输入数据中，就在视频的静止区域中产生一系列时序累计的误差，使得静止区域出现不自然的图像畸变。本实施例为避免该问题，设置了具备有限处理长度的有限处理序列，并确定当前待处理的视频流中哪些帧序号的视频帧应该处于一个有限处理序列中，以此仅考虑在一个有限处理序列中对超分辨处理结果进行循环输入，且超分辨处理结果不再作为前序信息参与与之连续的下一个有限处理序列的超分辨率处理，以此来消减因参与循环的视频帧数过多引起的时序累积误差，本实施例将连续的两个有限处理序列分别记为当前有限处理序列和下一有限处理序列。
考虑到本实施例所接收视频帧的实时性，本实施例通过预先确定当前帧号划分集合及下一帧号划分集合来确定实时接收的当前视频帧应该属于哪个有限处理序列。示例性的，可以在当前视频帧的当前帧序号存在于当前帧号划分集合时，确定当前视频帧属于当前有限处理序列，也可以在当前帧序号存在于下一帧号划分集合时，确定当前视频帧属于下一有限处理序列。
S102、当所述当前视频帧同时属于当前有限处理序列和下一有限处理序列时，基于所述当前视频帧的图像信息与第一前序信息结合进行超分辨率处理以获得当前视频帧的第一处理结果，且基于所述当前视频帧的图像信息与第二前 序信息结合进行超分辨率处理以获得当前视频帧的第二处理结果。
需要说明的是，对视频流中的视频帧进行有限处理序列归属确定时，如果两个相邻的有限处理序列中所包括的视频帧均不相同，对于处于当前有限处理序列末位以及处于下一有限处理序列首位的视频帧而言，两个相邻有限处理序列进行超分辨率处理时所需的前序信息发生了变更，其相当于下一有限处理序列首位的视频帧所采用的前序信息不再沿用前一视频帧的处理结果，从而使得相邻两有限处理序列的邻接处在显示内容上有所不同，由此发生显示内容跳变，影响视频的显示效果。
本实施例为保证视频超分辨处理后的显示效果，考虑将一部分视频帧重叠划分在两个相邻的有限处理序列中，对于重叠存在于相邻两个有限处理序列中的视频帧而言，可以基于其在两个有限处理序列中分别对应的超分辨率处理结果进行视频所显示内容的平滑过渡，以此来避免邻接的视频帧所显示内容跳变的情况。
若当前视频帧重叠存在于相邻两个有限处理序列中，相当于当前视频帧的当前帧序号既被划分在当前帧号划分集合中，也被划分在下一帧号划分集合中，由此可确定当前视频帧同时属于当前有限处理序列和下一有限处理序列。
本步骤确定当前视频帧为处于相邻两个有限处理序列中的重叠帧时，将分别沿用其在当前有限处理序列中对应的第一前序信息，以及其在下一有限处理序列中对应的第二前序信息，分别将第一前序信息和第二前序信息与当前视频帧自身的图像信息相结合进行超分辨率处理。
其中，所述图像信息可由构成当前视频帧的像素值构成，可采用一个三维矩阵表示，三维矩阵的其中两个维度(第一维度和第二维度)可分别表示当前视频帧的宽和高，余下的一个维度(第三维度)可以表示当前视频帧的颜色(RGB)通道。此外，本步骤中的第一前序信息以及第二前序信息包含的信息并不相同，第一前序信息以及第二前序信息的实质内容可根据当前视频帧的前一视频帧所 归属的有限处理序列来确定。
可以理解的是，本实施例进行超分辨率处理所需的前序信息实际与当前待处理视频帧的前一视频帧所进行超分辨率处理后的处理结果有关，再考虑到处于下一有限处理序列首位视频帧的前序信息不再延续前一视频帧的处理结果，本实施例考虑对进行超分辨率处理所使用的不同前序信息，根据当前视频帧的前一视频帧所归属的有限处理序列进行下述限定。
示例性的，在所述当前视频帧的前一视频帧仅属于当前有限处理序列时，所述第一前序信息为所述前一视频帧的第三处理结果；在所述当前视频帧的前一视频帧同时属于当前有限处理序列和下一有限处理序列时，所述第一前序信息为所述前一视频帧的第一处理结果；在所述当前视频帧的前一视频帧仅属于当前有限处理序列时，所述第二前序信息为以全0表示的图像信息；所述当前视频帧的前一视频帧同时属于当前有限处理序列和下一有限处理序列时，所述第二前序信息为所述前一视频帧的第二处理结果。
在本实施例中，在当前视频帧同时属于当前有限处理序列和下一有限处理序列时，若与其邻接的前一视频帧仅属于当前有限处理序列，则可认为该前一视频帧仅具备在当前有限处理序列下对应的处理结果，该处理结果记为前一视频帧的第三处理结果，并作为当前视频帧的第一前序信息；此时，当前视频帧相当于下一有限处理序列中的首位视频帧，因下一有限处理序列不再沿用当前有限处理序列的前序信息，本实施例此时将第二前序信息设定为初始的以全0表示的图像信息。
若与当前视频帧邻接的前一视频帧同时属于当前有限处理序列和下一有限处理序列时，相当于前一视频帧也为一个重叠帧，此时，前一视频帧对于当前有限处理序列和下一有限处理序列分别对应一个超分辨率处理的处理结果，前一视频帧对于当前有限处理序列对应一个超分辨率处理的处理结果，该处理结果可记为前一视频帧的第一处理结果，前一视频帧对于下一有限处理序列对应 一个超分辨率处理的处理结果，这个处理结果可记为前一视频帧的第二处理结果，并将前一视频帧的第一处理结果作为当前视频帧的第一前序信息，将前一视频帧的第二处理结果作为第二前序信息。
此外，因为当前视频帧已经同时属于当前有限处理序列和下一有限处理序列，所以其对应的前一视频帧不存在仅属于下一有限处理序列的情况，因此，也不存在该种情况下的第一前序信息和第二前序信息。需要说明的是，本步骤中所进行超分辨率处理的操作并未发生改变，采用当前已训练好的循环神经网络模型即可实现。
S103、根据获得的当前视频帧的第一处理结果和当前视频帧的第二处理结果，确定所述当前视频帧的目标输出信息，并按照所述目标输出信息进行所述当前视频帧的超分辨率显示。
采用上述步骤对作为重叠帧的当前视频帧进行两次超分辨率处理后，可分别获得两个处理结果，本实施例将这两个处理结果分别记为当前视频帧的第一处理结果和当前视频帧的第二处理结果。本步骤可对这两个处理结果进行加权求和处理，并将加权求和后的结果作为当前视频帧超分辨处理的最终结果，该最终结果记为目标输出信息，且本步骤可直接将该目标输出信息作为该当前视频帧超分辨率处理后显示给视频接收用户的图像信息。
示例性的，本实施例可以分别为当前视频帧的两个处理结果确定加权时所占的权重值，并按照所对应的权重值对当前视频帧的第一处理结果和当前视频帧的第二处理结果进行加权求和，加权求和后输出的结果记为目标输出信息，其中，两个处理结果的权重值可采用插值函数(如线性插值函数)的形式确定。
本申请实施例一提供的一种视频处理方法，相比于相关技术中的视频超分辨率处理，本方案首先对待进行超分辨率处理的视频帧进行有限处理序列的归属划分，并以有限处理序列为周期来变更视频超分辨率处理时所需的前序信息，以此来消除相关技术处理中随时间累计而在视频静止区域产生的图像畸变；同 时，本技术方案引入重叠存在于连续两个有限处理序列中的视频帧，以通过对所重叠视频帧的处理来避免因两个有限处理序列间前序信息的变更而引起的视频帧所显示画面的跳变，以此来增强视频图像显示的平滑性。由此，在实现视频高效率超分辨处理的同时，保证了视频中静止区域画面的高保真显示。
作为一个可选实施例，将根据所接收当前视频帧的当前帧序号及预先确定的当前帧号划分集合和下一帧号划分集合，确定所述当前视频帧归属的有限处理序列，包括：获取所接收当前视频的当前帧序号，并根据当前帧序号在所述当前帧号划分集合和下一帧号划分集合中分别进行遍历搜索；如果所述当前帧序号仅存在于所述当前帧号划分集合，则确定所述当前视频帧仅属于当前有限处理序列；如果所述当前帧序号存在于所述当前帧号划分集合同时存在于所述下一帧号划分集合，则确定所述当前视频帧同时属于当前有限处理序列和下一有限处理序列；如果所述当前帧序号仅存在于所述下一帧号划分集合，则确定所述当前视频帧仅属于下一有限处理序列。
在本可选实施例中，给出了当前视频帧进行有限处理序列归属确定的过程，其主要通过将当前视频帧的当前帧序号与当前帧号划分集合以及下一帧号划分集合中的帧序号进行匹配来确定，以此可以确定出当前视频帧是否为既属于当前有限处理序列又属于下一有限处理序列的重叠帧，进而保证后续超分辨处理的有效进行。
需要说明的是，本实施例不存在当前帧序号均不存在于两帧号划分集合的情况，因为两帧号划分集合也都是随着对视频帧的处理根据有限处理序列的有限处理长度实时变更确定。
实施例二
图6给出了本申请实施例二提供的一种视频处理方法的流程示意图，本实施例二以上述实施例为基础进行细化，在本实施例中，还包括了：当所述当前 视频帧仅属于下一有限处理序列时，基于所述当前视频帧的图像信息及第三前序信息进行超分辨处理，其中，所述第三前序信息基于所述当前视频帧的前一视频帧归属的有限处理序列确定；将处理后获得的当前视频帧的第三处理结果确定为所述当前视频帧的目标输出信息，并按照所述目标输出信息进行所述当前视频帧的超分辨率显示。
本实施例还包括了：在当前视频帧及相应的前一视频帧均属于下一有限处理序列时，监测到所述当前视频帧进行超分辨率显示后，记所述当前视频帧归属于新的当前有限处理序列；将所述下一帧号划分集合记为新的当前帧号划分集合，并根据所述新的当前帧号划分集合，确定新的下一帧号划分集合。
本实施例还包括了：将所述当前视频帧记为前一视频帧，并当接收到新的当前视频帧时，返回继续执行确定所述新的当前视频帧归属的有限处理序列的操作。
如图6所示，本申请实施例二提供的一种视频处理的方法，包括S201至S211。
S201、根据所接收当前视频帧的当前帧序号及预先确定的当前帧号划分集合和下一帧号划分集合，确定所述当前视频帧归属的有限处理序列。
本实施例下述S202、S204以及S205及其分别与上述步骤关联的其他步骤，分别给出了当前视频帧在同时属于当前有限处理序列和下一有限处理序列、仅属于当前有限处理序列以及仅属于下一有限处理序列时所进行超分辨率处理的相关操作。
S202、当所述当前视频帧同时属于当前有限处理序列和下一有限处理序列时，基于所述当前视频帧的图像信息与第一前序信息结合进行超分辨率处理以获得当前视频帧的第一处理结果，并基于所述当前视频帧的图像信息与第二前序信息结合进行超分辨率处理以获得当前视频帧的第二处理结果。
需要说明的是，本实施例可选采用预先训练好的循环神经网络模型进行超分辨率处理。该循环神经网络模型将待处理视频帧自身的图像信息以及给定的 前序信息作为输入数据，循环神经网络模型的输出结果则作为超分辨率处理的处理结果。
示例性的，在当前视频帧为处于相邻两有限处理序列中的重叠帧时，本实施例可选的将以所述当前视频帧的图像信息及第一前序信息作为循环神经网络模型输入数据获得的输出数据作为当前视频帧的第一处理结果；同时可选的，将以所述当前视频帧的图像信息及第二前序信息作为循环神经网络模型输入数据获得的输出数据作为当前视频帧的第二处理结果。
在本实施例中，采用视频帧的图像信息和前序信息作为循环神经网络模型进行超分辨率处理后所输出处理结果也是一个三维矩阵，该三维矩阵中的第一维度和第二维度也分别表示视频帧的宽和高，但与视频帧原始的图像信息相比，这两维度上所包含元素的数量为原始的图像信息在该两维度上的整数倍，该整数倍的具体值在构建循环神经网络时已经设定，此外，其第三维度表示的颜色通道数量未发生改变。
示例性的，假设原始的图像信息的三维矩阵为256*256*3，若超分辨处理为2倍放大处理，则输出的处理结果的三维矩阵为512*512*3。需要说明的是，将循环神经网络模型输出的处理结果作为后一视频帧的前序信息时，需要对以三维矩阵表示的处理结果进行后处理。以一个示例来说明该后处理的操作，假设处理结果的三维矩阵为512*512*3，因为输入循环神经网络模型的输入数据要保证第一维度和第二维度的元素个数相同，由此，当输入数据要求的三维矩阵为256*256*3时，本实施例需要将三维矩阵512*512*3转化为256*256*12，以通过增加颜色通道的形式来保证处理结果中第一维度和第二维度的元素个数与所规定元素个数相同，三维矩阵256*256*12则可作为下一次待输入的前序信息。
S203、根据处理后获得的当前视频帧的第一处理结果和当前视频帧的第二处理结果，确定所述当前视频帧的目标输出信息，执行S207。
示例性的，可以分别确定当前视频帧的第一处理结果以及当前视频帧的第 二处理结果的权重值，通过权重值与相应处理结果的加权求和，可以获得最终可展示给目标用户的目标输出信息。
S204、当所述当前视频帧仅属于当前有限处理序列时，基于所述当前视频帧的图像信息及第三前序信息进行超分辨率处理，并在处理后获取当前视频帧的第三处理结果，执行S206。
示例性的，在当前视频帧仅属于当前有限处理序列时，当前视频帧仅需进行一次超分辨率处理，本实施例可以将该超分辨率处理所获得的处理结果记为当前视频帧的第三处理结果。由此，本实施例可选的将当前视频帧的图像信息以及第三前序信息作为循环神经网络模型的输入数据，处理后的输出数据则为当前视频帧的第三处理结果。
其中，所述第三前序信息同样需要基于当前视频帧的前一视频帧归属的有限处理序列来确定。示例性的，在所述当前视频帧的前一视频帧仅属于当前有限处理序列时，前一视频帧也仅输出一个记为第三处理结果的数据信息，由此，第三前序信息可直接为前一视频帧的第三处理结果；此外，因为当前视频帧仅属于当前有限处理序列，其之前的前一视频帧不可能同时属于当前有限处理序列和下一有限处理序列，因此，也不存在处于该种情况下的第三前序信息；同样的，其之前的前一视频帧也不可能仅属于下一有限处理序列，同样也不存在处于该种情况下的第三前序信息；但当前视频帧的前一视频帧可能为空，此时，可以将以全0表示的图像信息作为第三前序信息。
S205、当所述当前视频帧仅属于下一有限处理序列时，基于所述当前视频帧的图像信息及第四前序信息进行超分辨处理，并在处理后获取当前视频帧的第四处理结果，执行S206。
示例性的，在当前视频帧仅属于下一有限处理序列时，当前视频帧同样仅需结合第四前序信息进行一次超分辨率处理，本实施例也可以将该超分辨率处理所获得的处理结果记为当前视频帧的第四处理结果。由此，本实施例可选的 将当前视频帧的图像信息以及第四前序信息作为循环神经网络模型的输入数据，处理后的输出数据仍旧记为当前视频帧的第四处理结果。
其中，所述第四前序信息同样需要基于当前视频帧的前一视频帧归属的有限处理序列来确定。示例性的，因当前视频帧仅属于下一有限处理序列，考虑到重叠帧的存在，所述当前视频帧的前一视频帧不可能仅属于当前有限处理序列，因此也不存在该种情况下的第四前序信息；在所述当前视频帧的前一视频帧同时属于当前有限处理序列和下一有限处理序列时，相当于前一视频帧进行了两次超分辨率处理，处理结果分别记为前一视频帧的第一处理结果和前一视频帧的第二处理结果，此时，前一视频帧的第二处理结果被认为在前一视频帧在下一有限处理序列中对应的处理结果，本实施例可将第四前序信息记为前一视频帧的第二处理结果；此外，在当前视频帧的前一视频帧仅属于下一有限处理序列时，前一视频帧也只进行一次超分辨率处理，且处理后的处理结果记为前一视频帧的第四处理结果，因此，该种情况下的第四前序信息为前一视频帧的第四处理结果。
S206、将获得的当前视频帧的第三处理结果或当前视频帧的第四处理结果确定为所述当前视频帧的目标输出信息。
本步骤可在当前视频帧仅属于当前有限处理序列的情况下，将获得的当前视频帧的第三处理结果直接确定为当前视频帧的目标输出信息；在当前视频帧仅属于下一有限处理序列时，将获得的当前视频帧的第四处理结果直接确定为当前视频帧的目标输出信息。
S207、按照所述目标输出信息进行所述当前视频帧的超分辨率显示。
在本实施例中，当前视频帧在上述给定的三种有限处理序列的归属情况下，本步骤将获得的目标输出信息作为当前视频帧进行超分辨率处理的最终处理结果，并基于该目标输出信息对当前视频帧的画面内容进行超分辨率显示。
接上述以时间总长度为58秒，视频总帧数为1483的视频进行超分辨处理 的实施例，若采用本实施例提供的视频处理方式对该视频进行处理，其中，图7给出了对图2所显示视频帧采用本实施例提供的超分辨率处理方法处理后的效果显示图，假设当前所处理的当前视频帧为该视频中的第200帧，则如图7所示，基于本实施例提供的超分辨率处理后第200帧对应显示的视频画面中，第三静止区域12相较于图2内的第二静止区域11仍能清晰的显示画面细节，并未出现图像畸变。
需要说明的是，本实施例上述S201至S207相当于实现了当前视频帧的超分辨率处理，考虑到视频不断产生新的视频帧，本实施例可以通过下述操作对视频的超分辨率处理进行上述各步骤的迭代处理。其中，下述S208至S210给出了新的下一帧号划分集合的确定过程。
S208、当前视频帧是否仅属于下一有限处理序列，在当前视频帧仅属于下一有限处理序列的情况下，执行S209；在当前视频帧仅属于当前有限处理序列或当前视频帧同时属于当前有限处理序列和下一有限处理序列的情况下，执行S211。
本步骤相当于确定新的下一帧号划分集合的时机判定，该时机判定可通过确定当前视频帧是否仅属于下一有限处理序列来实现。可以理解的是，在当前视频帧仅属于下一有限处理序列时，相当于归属于当前有限处理序列中视频帧均已完成了超分辨率处理，不再需要已有的当前帧号划分集合，而当前的帧号划分集合中也包含了可能存在于与当前的帧号划分集合相邻的下一帧号划分集合中的视频帧，因此，只要确定当前视频帧仅属于下一有限处理序列，就可启动S209进行下一帧号划分集合的更新。在当前视频帧仅属于当前有限处理序列或者当前视频帧同时属于当前有限处理序列和下一有限处理序列的情况下，表明当前帧号划分集合中还存在未处理的帧序号，需要执行S211重新对新的当前视频帧进行超分辨率处理。
S209、监测到所述当前视频帧进行超分辨率显示后，记所述当前视频帧所 归属的下一有限处理序列为新的当前有限处理序列。
需要说明的是，在当前视频帧进行超分辨率显示后，相当于与前一视频帧的相关处理结果再无联系，本步骤就可以在当前视频帧完成了超分辨率显示后，将当前视频帧所归属的下一有限处理序列记为新的当前有限处理序列。
S210、将所述下一帧号划分集合记为新的当前帧号划分集合，并根据所述新的当前帧号划分集合确定新的下一帧号划分集合，执行S211。
可以理解的是，当前的下一帧号划分集合中还存在未处理的帧序号，本步骤可以将该下一帧号划分集合记为新的当前帧号划分集合，以进行新的下一帧号划分集合的确定。
示例性的，所述根据所述新的当前帧号划分集合确定新的下一帧号划分集合，可以包括：获取所设定有限处理序列的序列长度值以及所述有限处理序列中所包含重叠帧的重叠帧个数，并获取所述新的当前帧号划分集合中的首个帧序号；根据所述序列长度值、重叠帧个数以及所述首个帧序号，确定数量为序列长度值的新帧序号，并基于所确定数量的新帧序号形成新的下一帧号划分集合。
在本实施例中，本实施例所采用的有限处理序列的有限处理长度(即有限处理序列的序列长度值)可以是根据经验值预先设定的，有限处理序列所包含的重叠帧(即，存在于相邻两有限处理序列中的视频帧)的重叠帧个数也可以是根据经验值预先设定的。在获取有限处理长度和重叠帧个数后，已知新的当前帧号划分集合中的首个帧序号时，可以采用设定公式确定出新的下一帧号划分集合中的首个帧序号，基于新确定的首个帧序号以及有限处理长度，可以确定出新的下一帧号划分集合中的其他帧序号。
其中，该设定公式可以表示为：S (t+1),1＝S t,1+N-k，其中，S (t+1),1表示新的下一帧号划分集合S (t+1)中的首个帧序号，S t,1为新的当前前帧号划分集合S t中的首个帧序号，N表示有限处理长度，k表示重叠帧个数。示例性的，假设S t,1为1，N 为8，k为2，则可确定出S (t+1),1为7，新的下一帧号划分集合S (t+1)中的其余帧序号分别为8、9、10、11、12、13以及14。
S211、将所述当前视频帧记为前一视频帧，并当接收到新的当前视频帧时，返回S201。
在本实施例中，接上述S208或S210，在完成上述步骤的相应处理后，可以将当前视频帧变更为前一视频帧，并将新接收到的视频帧记为新的当前视频帧，然后返回至S201循环持续进行超分辨率的处理操作，直至不再接收新的视频帧。
可以理解的是，在当前视频帧变更为前一视频帧后，其对应的当前视频帧的第一处理结果、当前视频帧的第三处理结果、当前视频帧的第四处理结果、以及当前视频帧的第二处理结果同样变更为前一视频帧的第一处理结果、前一视频帧的第三处理结果、前一视频帧的第四处理结果、及前一视频帧的第二处理结果。
本申请实施例二提供的一种视频处理方法，给出了当前视频帧在三种有限处理序列归属情况下进行超分辨处理的实现过程；同时给出了进行新的下一帧号划分集合的确定过程以及视频帧循环进行超分辨率处理的过程。利用该方法，以有限处理序列为周期来变更视频超分辨率处理时所需的前序信息，并引入重叠存在于连续两个有限处理序列中的视频帧，以通过对所重叠视频帧的处理来避免因两个有限处理序列间前序信息的变更而引起的视频帧所显示画面的跳变，以此来增强视频图像显示的平滑性。同时引入了新的下一帧号划分集合的确定，确保在有限处理序列划分下超分辨率处理的有效进行，相比于相关技术中的改进方法，本实施例无需改变超分辨率处理所采用的循环神经网络模型的训练数据和训练方式，就能简单有效避免图像畸变，实现超分辨率处理的高效和高保真。
作为本实施例的一个可选实施例，本可选实施例对目标输出信息的确定过程进行了细化。可选的，图8给出了本实施例中进行目标输出信息确定的一种 实现流程图，如图8所示，根据处理后获得的当前视频帧的第一处理结果和当前视频帧的第二处理结果，确定所述当前视频帧的目标输出信息，包括S2031至S2033。
S2031、确定所述当前视频帧的第一处理结果的第一权重值，以及所述当前视频帧的第二处理结果的第二权重值。
本实施例考虑设置既处于当前有限处理序列又处于下一有限处理序列的重叠帧，并对重叠帧进行两次超分辨率处理的方式来实现两个有限处理序列邻接处视频画面的平滑显示。示例性的，作为重叠帧的当前视频帧所获得的两个超分辨率处理结果分别对应于当前有限处理序列和下一有限处理序列，本实施例考虑当前视频帧的第一处理结果以及当前视频帧的第二处理结果对该当前视频帧最终结果的影响分别占据一定的比例，其分别占用的比例相当于各自对应的权重值，且本实施例可根据两结果对最终结果的影响力度为两结果设置合适的权重值。
示例性的，可以采用下述操作来确定第一权重值和第二权重值：获取所述当前视频帧在所述当前帧号划分集合中对应的排列序号；根据所述排列序号及选定的插值函数确定所述当前视频帧的加权系数，其中，所述加权系数大于0且小于1；将所述加权系数作为所述当前视频帧所对应第一处理结果的第一权重值，并将1与所述加权系数的差值作为所述当前视频帧所对应第二处理结果的第二权重值。
在本实施例中，处于当前帧号划分集合以及下一帧号划分集合中的帧序号均按照时序顺序排列，且每个帧序号还可基于时序顺序对应一个排列序号；此外，本实施例在有限处理序列中至少设置了一个重叠的视频帧，且所重叠的多个视频帧的帧序号连续；在确定当前视频帧为重叠帧时，若确定在其之前还连续存在其他重叠帧，根据当前视频帧及其之前其他连续重叠帧的排列序号，可以知道该当前视频帧为第几个重叠帧。
对于包含多个连续重叠帧的当前有限处理序列和下一有限处理序列而言，应当考虑当前有限处理序列中多个重叠帧所对应处理结果对目标输出信息计算的影响度将逐渐降低，而多个重叠帧在下一有限处理序列中所对应处理结果对目标输出信息计算的影响度将逐渐升高。由此，本实施例采用插值函数来确定每个重叠帧在当前有限处理序列中对目标输出信息计算的影响度(本实施例记该影响度为加权系数)，且每个重叠帧在当前有限处理序列中对应的加权系数可直接看做该重叠帧的第一处理结果所对应的第一权重值(大于0且小于1)，而1与该第一权重值的差值则可看做该重叠帧在下一有限处理序列中所对应的第二处理结果的第二权重值。
可以知道的是，本实施例并不对该插值函数做限定，只要保证当前有限处理序列中多个重叠帧所对应的加权系数逐渐在降低即可，本实施例可选该插值函数为线性插值函数，且该线性插值函数可表示为：
Figure PCTCN2021078642-appb-000001
其中，P i表示第i个重叠帧对应的加权系数，i∈[1，p]，p为一个有限处理序列中所包含重叠帧的重叠帧个数。
示例性的，当一个有限处理序列序列中包含2个重叠帧时，第1个重叠帧对应的加权系数为0.67，相当于该重叠帧的第一处理结果对应的第一权重值为0.67，该重叠帧的第二处理结果对应的第二权重值为0.33；而第二个重叠帧对应的加权系数为0.33，相当于该重叠帧的第一处理结果对应第一权重值为0.33，该重叠帧的第二处理结果对应的第二权重值为0.67。
因此，对于作为重叠帧的当前视频帧而言，在已知其当前帧序号在当前帧号划分集合中的排列序号后，且已知其之前所存在重叠帧的个数后，可以知道该当前视频帧为第几个重叠帧，由此根据插值函数可确定出其对应的加权系数，进而可确定出当前视频帧的第一处理结果对应的第一权重值以及当前视频帧的第二处理结果对应的第二权重值。
S2032、采用所述第一权重值及第二权重值分别对所述当前视频帧的第一处 理结果及当前视频帧的第二处理结果进行加权求和处理。
接上述实施例，假设当前视频帧的第一处理结果的第一权重值为0.33，当前视频帧的第二处理结果的第二权重值为0.67，所进行的加权求和处理相当于0.33乘以第一处理结果与0.67乘以第二处理结果的和。
S2033、将加权求和处理后的输出结果，作为所述当前视频帧的目标输出信息。
示例性的，本步骤可直接将上述步骤加权求和处理后的数据信息作为超分辨处理最终的目标输出信息。
本实施例二的上述可选实施例，给出了当前视频帧为重叠帧时进行目标输出信息确定的实现过程。上述方法的实现考虑了重叠帧在两相邻有限处理序列中进行超分辨率处理后对最终处理结果影响度的分配特点，进行了两个处理结果权重值的确定，基于所确定的权重值获得目标输出信息更好的避免了两有限处理序列邻接处画面跳变的产生，保证了视频画面显示的平滑性。
实施例三
图9给出了本申请实施例三提供的一种视频处理装置的结构框图，该装置适用于视频的视频帧进行超分辨率处理的情况，该装置可以由硬件和/或软件实现，并一般集成在计算机设备上，如图9所示，该装置包括：处理序列确定模块31、信息处理模块32以及信息确定模块33。
其中，处理序列确定模块31，设置为根据所接收当前视频帧的当前帧序号及预先确定的当前帧号划分集合和下一帧号划分集合，确定所述当前视频帧归属的有限处理序列；
信息处理模块32，设置为当所述当前视频帧同时属于当前有限处理序列和下一有限处理序列时，基于所述当前视频帧的图像信息与第一前序信息结合进行超分辨率处理以获得当前视频帧的第一处理结果，且基于所述当前视频帧的 图像信息与第二前序信息结合进行超分辨率处理以获得当前视频帧的第二处理结果；
信息确定模块33，设置为根据处理后获得的当前视频帧的第一处理结果和当前视频帧的第二处理结果，确定所述当前视频帧的目标输出信息，并按照所述目标输出信息进行所述当前视频帧的超分辨率显示；
其中，所述第一前序信息不同于所述第二前序信息。
本申请实施例三提供的一种视频处理装置，以有限处理序列为周期来变更视频超分辨率处理时所需的前序信息，以此来消除相关技术处理中随时间累计而在视频静止区域产生的图像畸变；同时，本技术方案引入重叠存在于连续两个有限处理序列中的视频帧，以通过对所重叠视频帧的处理来避免因两个有限处理序列间前序信息的变更而引起的视频帧所显示画面的跳变，以此来增强视频图像显示的平滑性。由此，在实现视频高效率超分辨处理的同时，保证了视频中静止区域图像的高保真显示。
实施例四
图10给出了本申请实施例四提供的一种计算机设备的硬件结构示意图，该计算机设备包括：处理器和存储装置。存储装置中存储有至少一条指令，且指令由所述处理器执行，使得所述计算机设备执行如上述方法实施例所述的视频处理方法。
参照图10，该计算机设备可以包括：处理器40、存储装置41、显示屏42、输入装置43、输出装置44以及通信装置45。该计算机设备中处理器40的数量可以是至少一个，图4中以一个处理器40为例。该计算机设备中存储装置41的数量可以是至少一个，图10中以一个存储装置41为例。该计算机设备的处理器40、存储装置41、显示屏42、输入装置43、输出装置44以及通信装置45可以通过总线或者其他方式连接，图4中以通过总线连接为例。
存储装置41作为一种计算机可读存储介质，可设置为存储软件程序、计算机可执行程序以及模块，如本申请实施例对应的程序指令/模块(例如，上述实施例所提供视频处理装置中的处理序列确定模块31、信息处理模块32以及信息确定模块33等)。存储装置41可主要包括存储程序区和存储数据区，其中，存储程序区可存储操作装置、至少一个功能所需的应用程序；存储数据区可存储根据计算机设备的使用所创建的数据等。此外，存储装置41可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他非易失性固态存储器件。在一些实例中，存储装置41可包括相对于处理器40远程设置的存储器，这些远程存储器可以通过网络连接至计算机设备。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
一般而言，显示屏42设置为根据处理器40的指示显示数据，还设置为接收作用于显示屏42的触摸操作，并将相应的信号发送至处理器40或其他装置。可选的，当显示屏42为红外屏时，其还包括红外触摸框，该红外触摸框设置在显示屏42的四周，其还可以设置为接收红外信号，并将该红外信号发送至处理器40或者其他计算机设备。
通信装置45，设置为与其他计算机设备建立通信连接，其可以是有线通信装置和/或无线通信装置。
输入装置43可设置为接收输入的数字或者字符信息，以及产生与计算机设备的用户设置以及功能控制有关的键信号输入，还可以是设置为获取图像的摄像头以及获取视频数据中音频的拾音计算机设备。输出装置44可以包括显示屏等视频计算机设备以及扬声器等音频计算机设备。需要说明的是，输入装置43和输出装置44的组成可以根据实际情况设定。
处理器40通过运行存储在存储装置41中的软件程序、指令以及模块，从而执行计算机设备的各种功能应用以及数据处理，即实现上述的视频处理方法。
处理器40执行存储装置41中存储的至少一个程序时，实现如下操作：根据所接收当前视频帧的当前帧序号及预先确定的当前帧号划分集合和下一帧号划分集合，确定所述当前视频帧归属的有限处理序列；当所述当前视频帧同时属于当前有限处理序列和下一有限处理序列时，基于所述当前视频帧的图像信息与第一前序信息结合进行超分辨率处理以获得当前视频帧的第一处理结果，且基于所述当前视频帧的图像信息与第二前序信息结合进行超分辨率处理以获得当前视频帧的第二处理结果；根据处理后获得的当前视频帧的第一处理结果和当前视频帧的第二处理结果，确定所述当前视频帧的目标输出信息，并按照所述目标输出信息进行所述当前视频帧的超分辨率显示。
本申请实施例还提供一种计算机可读存储介质，所述存储介质中的程序由计算机设备的处理器执行时，使得计算机设备能够执行如上述实施例所述的视频处理方法。示例性的，上述实施例所述的视频处理方法包括：根据所接收当前视频帧的当前帧序号及预先确定的当前帧号划分集合和下一帧号划分集合，确定所述当前视频帧归属的有限处理序列；当所述当前视频帧同时属于当前有限处理序列和下一有限处理序列时，基于所述当前视频帧的图像信息与第一前序信息结合进行超分辨率处理以获得当前视频帧的第一处理结果，且基于所述当前视频帧的图像信息与第二前序信息结合进行超分辨率处理以获得当前视频帧的第二处理结果；根据处理后获得的当前视频帧的第一处理结果和当前视频帧的第二处理结果，确定所述当前视频帧的目标输出信息，并按照所述目标输出信息进行所述当前视频帧的超分辨率显示。
需要说明的是，对于装置、计算机设备、存储介质实施例而言，由于其与方法实施例基本相似，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
通过以上关于实施方式的描述，所属领域的技术人员可以清楚地了解到，本申请可借助软件及必需的通用硬件来实现，当然也可以通过硬件实现。基于这样的理解，本申请的技术方案本质上或者说对相关技术做出贡献的部分可以 以软件产品的形式体现出来，该计算机软件产品可以存储在计算机可读存储介质中，如计算机的软盘、只读存储器(Read-Only Memory,ROM)、随机存取存储器(Random Access Memory,RAM)、闪存(FLASH)、硬盘或光盘等，包括若干指令用以使得一台计算机设备(可以是机器人，个人计算机，服务器，或者网络设备等)执行本申请任意实施例所述的视频处理方法。
值得注意的是，上述视频处理装置中，所包括的各个单元和模块只是按照功能逻辑进行划分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，各功能单元的名称也只是为了便于相互区分，并不用于限制本申请的保护范围。
应当理解，本申请的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，多个步骤或方法可以用存储在存储器中且由合适的指令执行装置执行的软件或固件来实现。例如，如果用硬件来实现，和在另一实施方式中一样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(Programmable Gate Array，PGA)，现场可编程门阵列((Field-Programmable Gate Array，FPGA)等。

Claims (15)

  1. 一种视频处理方法，包括：
    根据所接收当前视频帧的当前帧序号及预先确定的当前帧号划分集合和下一帧号划分集合，确定所述当前视频帧归属的有限处理序列；
    响应于确定所述当前视频帧同时属于当前有限处理序列和下一有限处理序列，将所述当前视频帧的图像信息与第一前序信息结合进行超分辨率处理以获得当前视频帧的第一处理结果，且基于所述当前视频帧的图像信息与第二前序信息结合进行超分辨率处理以获得当前视频帧的第二处理结果；
    根据获得的当前视频帧的第一处理结果和当前视频帧的第二处理结果，确定所述当前视频帧的目标输出信息，并按照所述目标输出信息进行所述当前视频帧的超分辨率显示；
    其中，所述第一前序信息不同于所述第二前序信息。
  2. 根据权利要求1所述的方法，其中，
    在所述当前视频帧的前一视频帧仅属于当前有限处理序列的情况下，所述第一前序信息为所述前一视频帧的第三处理结果；在所述当前视频帧的前一视频帧同时属于当前有限处理序列和下一有限处理序列的情况下，所述第一前序信息为所述前一视频帧的第一处理结果；
    在所述当前视频帧的前一视频帧仅属于当前有限处理序列的情况下，所述第二前序信息为以全0表示的图像信息；在所述当前视频帧的前一视频帧同时属于当前有限处理序列和下一有限处理序列的情况系，所述第二前序信息为所述前一视频帧的第二处理结果。
  3. 根据权利要求1所述的方法，还包括：
    响应于确定所述当前视频帧仅属于当前有限处理序列，基于所述当前视频帧的图像信息及第三前序信息进行超分辨率处理，在处理后获取当前视频帧的第三处理结果，并将获得的当前视频帧的第三处理结果确定为所述当前视频帧的目标输出信息；
    响应于确定所述当前视频帧仅属于下一有限处理序列，基于所述当前视频 帧的图像信息及第四前序信息进行超分辨处理，在处理后获取当前视频帧的第四处理结果，并将获得的当前视频帧的第四处理结果确定为所述当前视频帧的目标输出信息；
    按照所述目标输出信息进行所述当前视频帧的超分辨率显示。
  4. 根据权利要求3所述的方法，其中，
    在所述当前视频帧的前一视频帧仅属于当前有限处理序列情况下，所述第三前序信息为所述前一视频帧的第三处理结果，在所述当前视频帧的前一视频帧为空的情况下，所述第三前序信息为以全0表示的图像信息；
    在所述当前视频帧的前一视频帧同时属于当前有限处理序列和下一有限处理序列的情况下，所述第四前序信息为所述前一视频帧的第二处理结果；在所述当前视频帧的前一视频帧仅属于下一有限处理序列的情况下，所述第四前序信息为所述前一视频帧的第四处理结果。
  5. 根据权利要求3所述的方法，其中，所述超分辨率处理基于预先训练的循环神经网络模型进行；
    所述当前视频帧的第一处理结果为将所述当前视频帧的图像信息及所述第一前序信息作为所述循环神经网络模型的输入数据处理获得的输出数据；所述当前视频帧的第三处理结果为将所述当前视频帧的图像信息及所述第三前序信息作为所述循环神经网络模型的输入数据处理获得的输出数据；所述当前视频帧的第四处理结果为将所述当前视频帧的图像信息及所述第四前序信息作为所述循环神经网络模型的输入数据处理获得的输出数据；
    所述当前视频帧的第二处理结果为将所述当前视频帧的图像信息及所述第二前序信息作为所述循环神经网络模型的输入数据处理获得的输出数据。
  6. 根据权利要求1所述的方法，其中，
    所接收的当前视频帧为视频直播中实时捕获的视频帧；
    或者，所接收的当前视频帧为视频通话中实时捕获的视频帧；
    或者，所接收的当前视频帧为用户所选定视频文件中当前待播放的视频帧。
  7. 根据权利要求1所述的方法，还包括：
    响应于确定当前视频帧仅属于下一有限处理序列，监测到所述当前视频帧进行超分辨率显示后，记所述当前视频帧所属的下一有限处理序列为新的当前有限处理序列；
    将所述下一帧号划分集合记为新的当前帧号划分集合，并根据所述新的当前帧号划分集合确定新的下一帧号划分集合。
  8. 根据权利要求7所述的方法，其中，根据所述新的当前帧号划分集合确定新的下一帧号划分集合，包括：
    获取所设定有限处理序列的序列长度值以及所述有限处理序列中所包含重叠帧的重叠帧个数，并获取所述新的当前帧号划分集合中的首个帧序号；
    根据所述序列长度值、重叠帧个数以及所述首个帧序号，确定数量为序列长度值的新帧序号，并基于所确定数量的新帧序号形成新的下一帧号划分集合。
  9. 根据权利要求1所述的方法，还包括：
    将所述当前视频帧记为前一视频帧，并当接收到新的当前视频帧时，返回继续执行确定所述新的当前视频帧归属的有限处理序列。
  10. 根据权利要求1-9任一项所述的方法，其中，所述根据所接收当前视频帧的当前帧序号及预先确定的当前帧号划分集合和下一帧号划分集合，确定所述当前视频帧归属的有限处理序列，包括：
    获取所接收当前视频的当前帧序号，并利用当前帧号在所述当前帧号划分集合和下一帧号划分集合中分别进行遍历搜索；
    响应于所述当前帧序号仅存在于所述当前帧号划分集合，确定所述当前视频帧仅属于当前有限处理序列；
    响应于所述当前帧序号存在于所述当前帧号划分集合同时存在于所述下一帧号划分集合，确定所述当前视频帧同时属于当前有限处理序列和下一有限处理序列；
    响应于所述当前帧序号仅存在于所述下一帧号划分集合，确定所述当前视 频帧仅属于下一有限处理序列。
  11. 根据权利要求1-9任一项所述的方法，其中，根据处理后获得的当前视频帧的第一处理结果和当前视频帧的第二处理结果，确定所述当前视频帧的目标输出信息，包括：
    确定所述当前视频帧的第一处理结果的第一权重值，以及所述当前视频帧的第二处理结果的第二权重值；
    采用所述第一权重值及第二权重值分别对所述当前视频帧的第一处理结果及当前视频帧的第二处理结果进行加权求和处理；
    将加权求和处理后的输出结果，作为所述当前视频帧的目标输出信息。
  12. 根据权利要求11所述的方法，其中，确定所述当前视频帧的第一处理结果的第一权重值，以及所述当前视频帧的第二处理结果的第二权重值，包括：
    获取所述当前视频帧在所述当前帧号划分集合中对应的排列序号；
    根据所述排列序号及选定的插值函数确定所述当前视频帧的加权系数，其中，所述加权系数大于0且小于1；
    将所述加权系数作为所述当前视频帧的第一处理结果的第一权重值，并将1与所述加权系数的差值作为所述当前视频帧的第二处理结果的第二权重值。
  13. 一种视频处理装置，包括：
    处理序列确定模块，设置为根据所接收当前视频帧的当前帧序号及预先确定的当前帧号划分集合和下一帧号划分集合，确定所述当前视频帧归属的有限处理序列；
    信息处理模块，设置为当所述当前视频帧同时属于当前有限处理序列和下一有限处理序列时，基于所述当前视频帧的图像信息与第一前序信息结合进行超分辨率处理以获得当前视频帧的第一处理结果，且基于所述当前视频帧的图像信息与第二前序信息结合进行超分辨率处理以获得当前视频帧的第二处理结果；
    信息确定模块，设置为根据获得的当前视频帧的第一处理结果和当前视频 帧的第二处理结果，确定所述当前视频帧的目标输出信息，并按照所述目标输出信息进行所述当前视频帧的超分辨率显示；
    其中，所述第一前序信息不同于所述第二前序信息。
  14. 一种计算机设备，包括：
    至少一个处理器；
    存储装置，设置为存储至少一个程序；
    所述至少一个程序被所述至少一个处理器执行，使得所述至少一个处理器实现如权利要求1-12任一项所述的视频处理方法。
  15. 一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时，实现如权利要求1-12任一项所述的视频处理方法。
PCT/CN2021/078642 2020-03-13 2021-03-02 视频处理方法、装置、设备及存储介质 WO2021179954A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010175136.1A CN111246250B (zh) 2020-03-13 2020-03-13 视频处理方法、装置、设备及存储介质
CN202010175136.1 2020-03-13

Publications (1)

Publication Number Publication Date
WO2021179954A1 true WO2021179954A1 (zh) 2021-09-16

Family

ID=70864762

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/078642 WO2021179954A1 (zh) 2020-03-13 2021-03-02 视频处理方法、装置、设备及存储介质

Country Status (2)

Country Link
CN (1) CN111246250B (zh)
WO (1) WO2021179954A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111246250B (zh) * 2020-03-13 2022-07-01 广州市百果园信息技术有限公司 视频处理方法、装置、设备及存储介质
CN112597334B (zh) * 2021-01-15 2021-09-28 天津帕克耐科技有限公司 通信数据中心的数据处理方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109146776A (zh) * 2017-06-16 2019-01-04 陈馨瑶 一种基于时域相关性和运动补偿的视频超分辨率系统
CN110136066A (zh) * 2019-05-23 2019-08-16 北京百度网讯科技有限公司 面向视频的超分辨率方法、装置、设备和存储介质
CN110622502A (zh) * 2017-05-17 2019-12-27 三星电子株式会社 用于活动图像的超分辨率处理方法及其图像处理装置
CN111246250A (zh) * 2020-03-13 2020-06-05 广州市百果园信息技术有限公司 视频处理方法、装置、设备及存储介质

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2009087641A2 (en) * 2008-01-10 2009-07-16 Ramot At Tel-Aviv University Ltd. System and method for real-time super-resolution
US9258518B2 (en) * 2012-03-05 2016-02-09 Thomson Licensing Method and apparatus for performing super-resolution
CN103489173B (zh) * 2013-09-23 2016-08-17 百年金海科技有限公司 一种视频图像超分辨率重建方法
CN106251289A (zh) * 2016-07-21 2016-12-21 北京邮电大学 一种基于深度学习和自相似性的视频超分辨率重建方法
WO2018212599A1 (en) * 2017-05-17 2018-11-22 Samsung Electronics Co., Ltd. Super-resolution processing method for moving image and image processing apparatus therefor
CN107633482B (zh) * 2017-07-24 2020-12-29 西安电子科技大学 一种基于序列图像的超分辨率重建方法
US10621695B2 (en) * 2017-10-31 2020-04-14 Disney Enterprises, Inc. Video super-resolution using an artificial neural network
CN109102462B (zh) * 2018-08-01 2023-04-07 中国计量大学 一种基于深度学习的视频超分辨率重建方法
CN109819321B (zh) * 2019-03-13 2020-06-26 中国科学技术大学 一种视频超分辨率增强方法
KR20190110965A (ko) * 2019-09-11 2019-10-01 엘지전자 주식회사 이미지 해상도를 향상시키기 위한 방법 및 장치

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110622502A (zh) * 2017-05-17 2019-12-27 三星电子株式会社 用于活动图像的超分辨率处理方法及其图像处理装置
CN109146776A (zh) * 2017-06-16 2019-01-04 陈馨瑶 一种基于时域相关性和运动补偿的视频超分辨率系统
CN110136066A (zh) * 2019-05-23 2019-08-16 北京百度网讯科技有限公司 面向视频的超分辨率方法、装置、设备和存储介质
CN111246250A (zh) * 2020-03-13 2020-06-05 广州市百果园信息技术有限公司 视频处理方法、装置、设备及存储介质

Also Published As

Publication number Publication date
CN111246250A (zh) 2020-06-05
CN111246250B (zh) 2022-07-01

Similar Documents

Publication Publication Date Title
WO2022141819A1 (zh) 视频插帧方法、装置、计算机设备及存储介质
JP4157568B2 (ja) 画像の高解像度化方法及び装置
WO2021179954A1 (zh) 视频处理方法、装置、设备及存储介质
KR20210129583A (ko) 미디어 재생 디바이스에서의 콘텐츠 필터링
TWI382755B (zh) 影像處理電路及其方法
JP2002359778A (ja) フレーム補間式可変速度動画撮像システム
TW200833109A (en) Advanced deinterlacer for high-definition and standard-defintion video
TW201349852A (zh) 影像處理裝置與影像處理方法
WO2020253103A1 (zh) 视频图像处理方法、装置、设备及存储介质
WO2023005140A1 (zh) 视频数据处理方法、装置、设备以及存储介质
CN102291531A (zh) 图像处理装置、图像处理方法和程序
JP4157567B2 (ja) 動画像の高解像度化方法及び装置
WO2020259123A1 (zh) 一种调整图像画质方法、装置及可读存储介质
WO2022143385A1 (zh) 基于脉冲信号的显示方法、装置、电子设备及介质
WO2019228219A1 (zh) 一种去除视频抖动的方法及装置
US20100158403A1 (en) Image Processing Apparatus and Image Processing Method
WO2024067461A1 (zh) 图像处理方法、装置、计算机设备和存储介质
US20100039517A1 (en) Film cadence detection
CN114938461A (zh) 视频处理方法、装置、设备及可读存储介质
CN107071326B (zh) 视频处理方法及装置
WO2014115522A1 (ja) フレームレート変換装置及びフレームレート変換方法並びにフレームレート変換装置を備えた表示装置及び撮像装置
JP5130171B2 (ja) 画像信号処理装置および画像信号処理方法
JP2010073075A (ja) 画像信号処理装置、画像信号処理方法
CN110706169A (zh) 一种明星人像优化方法、装置以及存储装置
Prasantha An Approach for Frame Rate conversion of a Video

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21768511

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21768511

Country of ref document: EP

Kind code of ref document: A1