KR20220154940A - Apparatus and Method for Selecting Skip Frames - Google Patents

Abstract

The present invention provides a skip frame selecting apparatus and method. The apparatus comprises: a frame setting unit which sets skip-enable frames and fixed frames among multiple frames of an input image; a skip frame selection unit which is implemented as pre-learned artificial neural networks, determines whether each of skip-enable frames is skipped from skip-enable frames and frames adjacent to each of skip-enable frames according to learned method and activates or deactivates a skip flag; and an encoder which encodes remaining frames except for skip frames having the skip flag activated therein by a predetermined method and acquires an encoded image. The skip frame selection unit is pre-learned based on compression rates of the encoded image according to whether skip frames are excluded, and picture differences between a decoded image decoding the encoded image and a restoration image which restores the skip frames according to whether skip frames are excluded. Therefore, the present invention can improve the compression rates while maintaining a quality of video images.

Description

Skip frame selection apparatus and method {Apparatus and Method for Selecting Skip Frames}

The present invention relates to an apparatus and method for selecting a skip frame, and relates to an apparatus and method for selecting a skip frame using an artificial neural network.

In video compression techniques, a frame skipping technique has been proposed in which compression is performed by skipping some frames among multiple frames to reduce the total number of frames.

The frame skipping technique is a technique that reduces the number of frames by removing some frames in order to remove temporal redundancy in a video image of multiple frames, and is largely applied in two ways. The first is a method of simply skipping and transmitting frames to adjust the bit rate in a limited streaming environment. The second method is a method of selectively skipping frames according to the motion of an input image in consideration of frame restoration through frame interpolation. That is, the frame skipping technique has evolved from a technique of simply skipping frames at a fixed cycle to a technique of adaptively selecting and skipping frames by analyzing the motion of an input image, thereby improving the compression rate of video images.

However, these conventional frame skipping techniques select frames in a way that does not sufficiently consider changes in video quality during video encoding and decoding according to codecs or the quality of frames restored through frame interpolation. That is, when a fixed frame or an arbitrary frame is simply skipped to adjust the bit rate, even if the skipped frame is subsequently restored by frame interpolation, the image quality is greatly deteriorated, resulting in a very large loss of image quality compared to the gain of the bit rate. . In addition, even when frames are skipped in consideration of frame restoration through frame interpolation, the quality of frames generated by codec interpolation is degraded as frames selected in a predetermined manner based on the motion of the input video are skipped. However, there is a limitation that it is difficult to greatly improve the compression ratio.

Korean Registered Patent No. 10-2207736 (registered on 2021.01.20)

An object of the present invention is to provide an apparatus and method for selecting a skip frame by considering the quality of a frame obtained through frame interpolation after encoding and decoding.

Another object of the present invention is to provide an apparatus and method for selecting a skip frame capable of improving a compression rate while maintaining the quality of a video image.

In order to achieve the above object, an apparatus for selecting skip frames according to an embodiment of the present invention includes a frame setting unit configured to set skippable frames and fixed frames among a plurality of frames of an input image; Implemented as a pre-learned artificial neural network, skip frame selection for activating or inactivating a skip flag by determining whether to skip each of the skippable frames from each of the skippable frames and adjacent frames of each of the skippable frames according to the learned method. wealth; and an encoder that obtains an encoded image by encoding frames other than the skip frame in which the skip flag is activated in a predetermined manner, wherein the skip frame selector determines a compression rate of the encoded image and an encoded image according to whether the skip frame is excluded or not. It is pre-learned based on a picture quality difference between a decoded image obtained by decoding and a reconstructed image obtained by reconstructing the skip frame according to whether or not the skip frame is excluded.

The skip frame selection unit calculates the compression rate of an encoded image obtained by encoding a plurality of frames of the input image not excluding the skip frame and the quality of a decoded image obtained by decoding the same according to a predetermined method, so that the compression rate and the image quality are the axes. The compression rate of the encoded image obtained by projecting into the coordinates of the dimensional space and encoding a plurality of frames excluding the skip frame and the image quality of the reconstructed image obtained by decoding and reconstructing the skip frame are calculated and projected into the coordinates of the 2D space, and the two coordinates are calculated. It can be learned based on the displacement of the liver.

The skip frame selector may obtain a skip probability value for the skippable frame according to a learned method, and deactivate or activate a skip flag according to whether the obtained skip probability value is greater than or equal to a predetermined reference value.

The skip frame selector obtains a fitting function that follows the pattern of the compression rate and image quality projected in the two-dimensional space when the skip frame is not excluded, and the obtained fitting function and the compression rate projected when the skip frame is excluded It can be learned based on displacement between coordinates of image quality.

When the skip frame selector excludes the skip frame, a difference between a displacement between the projected compression rate and image quality coordinates and the fitting function and the skip probability value may be back-propagated and learned as a loss.

The skip frame selector determines whether a displacement between the projected compression rate and image quality coordinates and the fitting function is greater than or equal to a predetermined reference displacement when skip frames are excluded, and an activation or inactivation state of the skip flag. The loss obtained by comparison can be back-propagated and learned.

The frame setting unit may set a fixed frame at a predetermined interval from an initial frame among the plurality of frames and set remaining frames as skippable frames.

The frame setting unit applies each skippable frame and an adjacent frame of the skippable frame to the skip frame selector, except when a skip flag for the adjacent frame is already activated, and applies the next skippable frame and the adjacent frame. can

To achieve the above object, a method for selecting skip frames according to another embodiment of the present invention includes setting skippable frames and fixed frames among a plurality of frames of an input image; activating or inactivating a skip flag by determining whether each of the skippable frames is skipped from each of the skippable frames and adjacent frames of each of the skippable frames according to a learned method using a pre-learned artificial neural network; and obtaining an encoded image by encoding frames other than the skip frame in which the skip flag is activated in a predetermined manner, wherein the artificial neural network calculates a compression rate of the encoded image and the encoded image according to whether the skip frame is excluded or not. It is pre-learned based on the difference in quality between the decoded decoded image and the reconstructed image obtained by reconstructing the skip frame according to whether or not the skip frame is excluded.

Therefore, the skip frame selection apparatus and method according to an embodiment of the present invention considers the quality of frames obtained by frame interpolation after encoding and decoding according to a codec in advance to select skip frames, thereby maintaining the quality of video images. The compression rate can be improved.

1 shows an image compression and restoration system according to an embodiment of the present invention.
FIG. 2 is a diagram for explaining the concept of image compression and restoration of the image compression and restoration system of FIG. 1 .
FIG. 3 shows an example of a detailed configuration of the skip frame interpolator of FIG. 1 .
FIG. 4 shows an example of a detailed configuration of the frame selection learning unit of FIG. 1 .
FIG. 5 is a diagram for explaining the concept of learning the skip frame selector of FIG. 1 .
6 is a diagram for explaining loss according to compression rate and image quality.
7 shows a skip frame selection method according to an embodiment of the present invention.

In order to fully understand the present invention and its operational advantages and objectives achieved by the practice of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the described embodiments. And, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain component, it means that it may further include other components, not excluding other components unless otherwise stated. In addition, terms such as "... unit", "... unit", "module", and "block" described in the specification mean a unit that processes at least one function or operation, which is hardware, software, or hardware. And it can be implemented as a combination of software.

1 shows an image compression and restoration system according to an embodiment of the present invention, and FIG. 2 is a diagram for explaining the concept of image compression and restoration of the image compression and restoration system of FIG. 1 .

Referring to FIG. 1, an image compression and restoration system according to the present embodiment includes an image compression device 100 that receives an input image composed of a plurality of consecutive frames and compresses it in a predetermined method and a compressed image in a predetermined method It may include an image restoration device 200 that restores to .

A method of compressing an image by the video compression apparatus 100 can be largely divided into a compression technique based on intra-prediction and a compression technique based on inter-prediction. The intra-prediction technique can be regarded as a pixel skipping technique, which compresses to minimize spatial redundancy by removing predictable pixels in a picture using the property that adjacent pixels within the same frame have high correlation with each other. In addition, the inter-prediction technique uses the property that a plurality of consecutive frames has high temporal correlation and removes predictable frames from the plurality of frames to compress them so that temporal redundancy is minimized. It can be seen as a frame skipping technique.

The video compression apparatus 100 may perform compression using at least one of intra-prediction and inter-prediction. However, the frame skipping technique for removing frames based on inter-prediction is based on the compression rate and the image quality of the restored frame. The loss is large, so it is hardly used for compression in practice. Therefore, in this embodiment, the image compression device 100 considers the quality of the frame of the image to be restored, selects a frame capable of preventing deterioration of image quality and simultaneously increases the compression rate, skips the selected frame, and compresses the frame. The skip technique can be practically used for image compression.

Accordingly, the video compression device 100 of FIG. 1 may be referred to as a skip frame selection device according to the present invention. The skip frame selection device 100 may include a frame setting unit 110 , a skip frame selection unit 120 and an encoder 130 .

The frame setting unit 110 receives an image composed of a plurality of consecutive frames to be compressed, and sets skippable frames that can be removed and fixed frames that must not be removed among the plurality of frames of the applied image. As described above, since the frame skipping technique based on inter-prediction is based on the temporal correlation between the skipped frame and adjacent frames, if a number of frames are consecutively skipped, the temporal correlation is lowered and the skipped frame can be restored normally. An impossible problem arises.

In order to prevent this problem, in the skip frame selection device 100 of the present embodiment, the frame setter 110 can divide a plurality of consecutive frames into skippable frames that can be removed and fixed frames that must not be removed. At this time, the frame setting unit 110 may set skippable frames and fixed frames in a plurality of frames according to a predetermined cycle.

As the simplest example, the frame setting unit 110 may alternately set a plurality of frames into fixed frames and skippable frames according to time order. That is, odd-numbered frames may be set as fixed frames, and even-numbered frames may be set as skippable frames. This is to ensure that the skippable frame can be easily restored as two frames temporally adjacent to the skippable frame are set as fixed frames.

In addition, the frame setting unit 110 may set fixed frames in a cycle of 4 after the initial frame in a plurality of frames, and set frames between the set fixed frames as skippable frames. When the fixed frame and the skippable frame are alternately set, there is an advantage in that the skippable frame can be easily restored. However, in the present embodiment, since all skippable frames are not skipped in consideration of image quality degradation, the compression rate is improved. may not be large. Accordingly, the skip frame selection apparatus 100 increases the number of skippable frames by reducing the ratio of fixed frames and increasing the ratio of skippable frames, so that not only even-numbered frames but also some odd-numbered frames can be skipped. By doing so, the compression ratio can be improved. As described above, when the fixed frames are set in a cycle of 4 frames, 3 frames between the fixed frames are set as skippable frames.

When a fixed frame and a skippable frame are set among a plurality of frames, the frame setting unit 110 sequentially selects a predetermined number of frames in a predetermined order from the plurality of frames and transmits them to the skip frame selection unit 120 . For example, the frame setter 110 may sequentially transmit three consecutive frames to the skip frame selector 120 . At this time, the frame setter 110 may differently select a frame to be transmitted to the skip frame selector 120 according to the settings of the fixed frame and the skippable frame.

If the frame setting unit 110 alternately sets fixed frames and skippable frames in a plurality of frames and transmits three frames each, the frame setting unit 110 first transmits the first to third frames, , Then, in a manner in which the third to fifth frames are transmitted, three frames may be sequentially selected and transmitted in such a manner that the skippable frame is located in the center. This is because the fixed frame is positioned at the center of three frames and transmitted, and even if the skip frame selector 120 selects the skip frame, the fixed frame cannot be skipped, so it is meaningless.

In addition, the frame setter 110 may select frames so that the skip frames selected by the previous skip frame selector 120 are not transmitted to the skip frame selector 120 again. As described above, even if three frames are set as skippable frames, if two or more consecutive frames are skipped, it is difficult to restore the skipped frames normally. In order to prevent this problem from occurring, the frame setting unit 110, when the skip frame selection unit 120 selects a previously transmitted frame as a skip frame, performs inter-prediction based on the previously selected skip frame; That is, a predetermined number of frames are delivered to the skip frame selection unit 120 in an order excluding skip frames so as not to estimate frame interpolation possibility.

For example, even when the second to fourth frames are set as skippable frames, the first to third frames are transmitted to the skip frame selector 120, and the skip frame selector 120 determines that the second frame is a skip frame. If selected, since it has already been determined that the second frame will be skipped, the third frame cannot be predicted from the second and fourth frames. Therefore, the frame setting unit 110 does not need to determine whether to skip the third frame by transmitting the second to fourth frames to the skip frame selector 120 . In this case, the frame setting unit 110 may skip the second frame selected as the skip frame, select the third to fifth frames, and transmit them to the skip frame selection unit 120 .

The frame setting unit 110 receives the skip flag output as a result of determining whether or not the frame applied by the skip frame selection unit 120 is skipped, and assigns the skip flag to a plurality of frames to be transmitted to the skip frame selection unit 120. You can decide whether or not to include the corresponding frame.

The skip frame selection unit 120 is implemented as a pre-learned artificial neural network, determines whether or not to skip a frame at a predetermined position among a plurality of frames applied from the frame setting unit 110 according to the learned method, and determines whether or not to skip the frame according to the determination result. Outputs skip flags accordingly.

The skip frame selection unit 120 may be implemented as an example of a convolution neural network (CNN) including a plurality of convolution kernels and fully connected layers. When a plurality of frames are applied, the skip frame selector 120 determines whether to skip by analyzing the possibility of restoring a frame positioned in the middle among the plurality of frames in consideration of the picture quality of the frame to be restored according to a pre-learned method. In this case, the skip frame selector 120 may determine whether to skip by considering frame quality changes according to encoding and decoding processes of the encoder 130 and the decoder 210, which will be described later. In addition, the value of the skip flag may be changed and output according to the determination result. Here, the skip flag may have a value of 1 when it is determined that the corresponding frame is skipped, whereas it may be output as a binary value to have a value of 0 when it is determined that the corresponding frame is not skipped. However, in some cases, the skip frame selection unit 120 may be configured to obtain a skip probability value, inactivate or activate a skip flag to 0 or 1 according to whether the obtained skip probability value is greater than or equal to a predetermined reference value, and output the result.

As described above, since the skip frame selector 120 is implemented as an artificial neural network, it must be trained in advance to perform normal operations, and a learning method of the skip frame selector 120 will be described later.

The encoder 130 receives frames determined as not to be skipped among a plurality of frames input to the frame setting unit 110 according to the skip flag output from the skip frame selection unit 120, and encodes and encodes them according to a predetermined method. output the video In this case, the encoder 130 may obtain an encoded image by encoding a skip flag for each of a plurality of frames together, and may transmit the encoded image to the image restoration apparatus 200 in the form of a bitstream.

Meanwhile, the image restoration device 200 receives the encoded image transmitted from the skip frame selection device 100, decodes it according to a predetermined method, and recovers the skipped frame from the decoded image to restore the input image.

The image restoration apparatus 200 may include a decoder 210 and a skip frame interpolator 220 . First, the decoder 210 receives an encoded image and decodes it according to a predetermined method to obtain a decoded image. Here, the decoder 210 performs decoding in a manner corresponding to the designated encoding method of the encoder 130, and the encoding method of the encoder 130 and the decoding method of the decoder 210 are codecs set to be used in the video compression and restoration system. (Codec) pre-specified.

Also, the decoder 210 may obtain a skip flag included in the encoded image together with the decoded image by decoding the encoded image.

The skip frame interpolator 220 receives the decoded image and the skip flag from the decoder 210, determines the skip frame excluded during transmission in the skip frame selection device 100 according to the applied skip flag, and determines the skip frame determined. The skip frame is reconstructed according to the frame interpolation technique based on the frames adjacent to the frame.

That is, as shown in FIG. 2, the video compression and restoration system of the present invention receives input video, which is a video input composed of a plurality of frames, determines and removes skip frames to be removed among the plurality of frames, and encodes the video of the remaining frames. The input video may be reconstructed by transmitting the encoded video in a bitstream format, decoding the encoded video transmitted in the bitstream format, and restoring the skipped frame using a frame interpolation technique.

At this time, the skip frame selection apparatus 100 determines whether a plurality of frames are skipped in consideration of the quality change of the reconstructed image according to the encoding and decoding of the encoder 130 and the decoder 210 and the frame interpolation of the skip frame interpolator 220. It is possible to obtain an effect of improving a compression rate according to frame skipping while preventing deterioration in image quality of a restored image by skipping and transmitting frames by determining .

The frame selection learning unit 300 is provided during learning and allows the skip frame selection unit 120 to learn. The skip frame selection unit 120 implemented as an artificial neural network must be trained in advance before actual use of the image compression and restoration system. The decoder 210 is trained to determine whether to skip a frame according to the quality change of the reconstructed image according to the encoding and decoding and the frame interpolation of the skip frame interpolator 220. A detailed description of the frame selection learning unit 300 will be described later.

FIG. 3 shows an example of a detailed configuration of the skip frame interpolator of FIG. 1 , and FIG. 4 shows an example of a learning method of the skip frame interpolator of FIG. 3 .

A frame interpolation technique for obtaining a middle frame between two frames using correlation between two frames is a well-known technique and can be implemented in various ways. The conventional frame interpolation technique is mainly a numerical analysis technique, and a technique of detecting blocks similar to each other block by block in adjacent frames and reconstructing each block of an intermediate frame based on a change in position of the similar block and a difference in block value is used. However, with the recent development of artificial neural networks, most of the frame interpolation techniques are also performed using artificial neural networks. Therefore, in this embodiment, it is assumed that the skip frame interpolator 220 is implemented with artificial neural networks to restore skip frames. .

Referring to FIG. 3 , the skip frame interpolator 220 may include a feature extraction unit 221, a feature interpolation unit 222, a feature correlation unit 223, and a frame acquisition unit 224.

The feature extractor 221 receives a predetermined number of frames adjacent to the skip frame designated by the skip flag among a plurality of frames of the image decoded by the decoder 210 (here, two as an example), and uses a pre-learned method. Accordingly, a plurality of feature maps are obtained by extracting features of each applied frame. The feature extractor 221 may include first and second feature extractors 2211 and 2312 .

The first feature extractor 2211 extracts a feature from a frame in temporal order based on the skip frame in multiple frames and outputs a first feature map, and the second feature extractor 2212 outputs a first feature map. A second feature map is output by extracting a feature by receiving a frame sequentially next to the frame based on the skip frame.

The feature interpolator 222 receives the first and second feature maps respectively obtained from the first and second feature extractors 2211 and 2312 of the feature extractor 221, and the applied first and second features. An interpolation feature map is obtained by interpolating the map according to a predetermined method. At this time, the feature interpolator 222 may obtain an interpolated feature map by interpolating the first and second feature maps according to a predetermined mathematical decoration. An interpolation feature map may be obtained.

The feature correlation unit 223 receives the first and second feature maps and the interpolation feature map, and reflects the temporal correlation between the first feature map, the interpolation feature map, and the second feature map according to a pre-learned method, thereby generating the first and second feature maps. First and second hidden maps and interpolation hidden maps corresponding to the second feature map and the interpolation feature map, respectively, are output. For example, the feature correlation unit 223 may be implemented as a long short-term memory (LSTM) that reflects temporal correlation among artificial neural networks.

The frame acquisition unit 224 may include first and second frame acquisition units 2241 and 2342 and an interpolation frame acquisition unit 2243 . The first frame acquisition unit 2241 receives the first hidden map and acquires a first frame corresponding to a previous frame of the skip frame according to a previously learned method, and the second frame acquisition unit 2242 acquires the second hidden map. may be applied and a second frame corresponding to a frame following the skip frame may be acquired according to a pre-learned method. Further, the interpolation frame acquisition unit 2243 receives the interpolation hidden map and obtains an interpolation frame corresponding to the skip frame according to a pre-learned method. Each of the first and second frame acquisition units 2241 and 2342 and the interpolation frame acquisition unit 2243 may be implemented as, for example, a convolutional neural network.

As such, the skip frame interpolator 220 obtains interpolated frames by receiving temporally adjacent previous and subsequent frames from the skip frame designated by the skip flag, and obtains interpolated frames, as well as first and subsequent frames of the interpolated frame. 2 frames are also acquired again so that the skip frame interpolator 220 does not stop at simply obtaining an interpolated frame, but compensates for the degradation of picture quality that may occur in the encoding and decoding processes in the encoder 130 and the decoder 210. It is for

The frame acquisition unit 224 composed of an artificial neural network compares the first and second frames and the interpolation frame with corresponding frames of the input image, i.e., skip frames selected from the input image and frames before and after the skip frame, thereby resulting in loss. It can be learned by calculating , and backpropagating the calculated loss. At this time, the process of inputting the frame of the input image to the skip frame interpolator 220 includes encoding and decoding processes of the encoder 130 and the decoder 210 designated by the codec. Therefore, when the skip frame interpolator 220 compares the first and second frames and the interpolated frame with the corresponding frame of the input image and learns according to the acquired loss, the skip frame interpolator 220 simply obtains only the interpolated frame. Instead, it is possible to compensate for deterioration in picture quality of the first and second frames that may occur during encoding and decoding processes.

Meanwhile, the skip frame interpolator 220 may also be configured not to reacquire first and second frames, which are frames before and after the interpolated frame. In this case, the first and second frame obtainers 2241 and 2342 can be omitted. In some cases, the feature correlation unit 223 may also be omitted, so that the interpolation frame obtaining unit 2243 directly obtains an interpolation frame from an interpolation feature map. In this way, when the skip frame interpolator 220 is configured to obtain only the interpolated frame, the skip frame interpolator 220 compares only the difference between the skip frame removed by the skip frame selection device 100 and the interpolated frame to reduce loss. By computing and backpropagating the computed loss, it can be learned.

4 shows an example of a detailed configuration of the frame selection learning unit of FIG. 1, FIG. 5 is a diagram for explaining the concept of learning the skip frame selection unit of FIG. 1, and FIG. It is a drawing for

Referring to FIG. 4 , the frame selection learning unit 300 may include a compression rate calculating unit 310 , an image quality measuring unit 320 , a 2D projection unit 330 and a loss calculating unit 340 .

Referring to FIG. ₅ , _the operation of the frame selection learning unit 300 of FIG. 4 is described. A compression rate (R ₁ ) in the case of encoding in the encoder 130 according to a pre-specified video codec is calculated and obtained. In addition, the compression rate calculation unit 310 determines that the remaining frames (x ₁ , x ₂ ) except for the frame (x) selected to be skipped by the skip frame selector 120 among the plurality of frames (x ₁ , x, x ₂ ) It is obtained by calculating the compression rate (R ₂ ) when encoded by the encoder 130.

Meanwhile, the image quality measurer 320 converts the decoded image decoded by the decoder 210 to the corresponding frame of the input image after a plurality of frames (x ₁ , x, x ₂ ) of the input image are encoded and transmitted without being skipped. Compared with the frame (x ₁ , x, x ₂ ), the quality (D ₁ ) of the decoded image is calculated in comparison with the plurality of frames (x ₁ , x, x ₂ ) of the input image. In addition, the image quality measurer 320 performs encoding by skipping at least one frame (x) among a plurality of frames (x ₁ , x, x ₂ ) of the input image, and then decodes the skip frame by the decoder 210. A plurality of reconstructed frames interpolated and obtained by the interpolator 220 are compared with frames (x ₁ , x, x ₂ ) of the input image, and prepared for a plurality of frames (x ₁ , x, x ₂ ) of the input image The image quality (D ₂ ) of the restored frame is calculated.

That is, compression rates (R ₁ , R ₂ ) and frame quality (D ₁ , D ₂ ) according to frame skipping are calculated, respectively.

The 2D projection unit 330 converts the compression rate (R ₁ , R ₂ ) and the frame quality (D ₁ , D ₂ ) calculated according to whether frames are skipped, as shown in FIG. 6 , into two axes It is projected onto coordinates on a two-dimensional space. In FIG. 6, the blue line (3 Frame) represents the distortion level of frame quality in comparison to the compression rate when no frame is skipped, and the red line (2 Frame) represents the distortion level of frame quality in comparison to the compression rate due to frame skipping. indicates

Accordingly, the loss calculation unit 340 calculates the compression rate and image quality (R ₁ , D ₁ ) in a state where no frame is skipped and the compression rate and image quality (R ₂ , D ₂ ) in a state where the frame is skipped in the two-dimensional coordinate system according to the skip flag. A displacement (D) between the frames is calculated, and if the calculated displacement (D) is greater than or equal to a predetermined reference displacement (D _ref ), it is determined that the frame should not be skipped, and a skip discrimination value may be output. At this time, the skip flag output from the skip frame selector 120 must be inactivated so as not to skip the frame. On the other hand, if the calculated displacement (D) is less than the predetermined reference displacement (D _ref ), it is determined that the frame should be skipped, and a skip determination value is output. At this time, the skip flag output from the skip frame selector 120 should be activated to skip the frame.

Accordingly, the loss calculation unit 340 compares the activation or inactivation state of the skip flag actually output from the skip frame selection unit 120 in the learning process with the skip discrimination value determined by the loss calculation unit 340 to calculate the loss and reverse the result. By propagating, the skip frame selector 120 can be trained.

However, when the skip flag is output as a binary value as described above and the loss calculation unit 340 also determines whether the skip flag is the same or not, the learning time may be increased. Accordingly, when the skip frame selector 120 outputs a skip probability value and is configured to activate or deactivate a skip flag based on the skip probability value, the difference between the obtained skip probability value and the normalized value of the calculated displacement (D) is the loss. Learning can also be performed by calculating and backpropagating.

Meanwhile, as shown in the blue line of FIG. 6 , it is a very cumbersome task to acquire all of the two-dimensional patterns according to the compression ratio and image quality (R ₁ , D ₁ ) in a non-skipping state. Accordingly, when the relationship between the compression rate (R ₁ ) and the frame quality (D ₁ ) is obtained when the frame is not skipped, it is expressed as a cubic function (F(x) = a ₃ x ³ + a ₂ x ² as shown in the purple line). + a ₁ x+ a ₀ ) to obtain a fitting function, and the displacement between the fitting function and the compression rate and image quality (R ₂ , D ₂ ) in the state of skipping frames You can also calculate the loss by calculating (D).

7 shows a skip frame selection method according to an embodiment of the present invention.

Referring to FIGS. 1 to 6, the skip frame selection method of FIG. 7 is described. In the skip frame selection method (S10), fixed and skippable frames among a plurality of frames of an input image are set in a predetermined manner (S11). ). In this case, first of the plurality of frames, a fixed frame may be set at a predetermined interval from an initial frame, and the remaining frames may be set as skippable frames.

When the fixed and skippable frames are set, a pre-learned artificial neural network is sequentially and continuously applied by a predetermined number to determine whether to skip skippable frames (S12). In this case, skip frames previously determined to be skipped are not included.

The learned artificial neural network determines whether to skip a frame at a predetermined position among applied frames according to the learned method (S13). In this case, when a frame is skipped from the input frames, the artificial neural network is encoded and decoded by a designated codec, and the skipped frame is interpolated to learn to determine skipping in consideration of the quality of the reconstructed image.

If it is determined as a skip frame, a skip flag is activated. In this case, the skip flag may have a binary value of 0 or 1, and for example, the skip flag may be activated and output as 1. In some cases, a skip probability value may be obtained from an artificial neural network, and a skip flag may be deactivated or activated according to whether the obtained skip probability value is greater than or equal to a predetermined reference value.

When the skip flag is activated, an encoded image is obtained by sequentially encoding the frames other than the frame corresponding to the activated skip flag according to a method specified by the codec (S15). At this time, the skip flag is also encoded and included in the encoded image.

The skip frame selection method ( S10 ) may be completed by obtaining an encoded image, but the skipped, encoded, and compressed image should then be reconstructed similarly to the input image. Accordingly, an image restoration method will be additionally described below. In the image restoration method (S20), first, an encoded frame is received and decoded according to a method specified by a codec to obtain a decoded image (S21). As the encoded image is decoded, the skip flag is restored in the decoded image.

Then, it is determined whether an activated skip flag is detected among the recovered skip flags (S22). If an activated skip flag is detected, previous and subsequent frames temporally adjacent to the detected skip flag are extracted (S23). Then, an interpolated frame is obtained by interpolating the extracted adjacent frame in a predetermined manner (S24). In this case, the interpolation frame may be obtained according to a predetermined mathematical method or may be obtained using a pre-learned artificial neural network. In addition, when obtained using an artificial neural network, the artificial neural network may be trained in advance to compensate for distortion of adjacent frames generated during encoding and decoding processes according to a codec.

When interpolation frames are acquired, interpolation frames are arranged between adjacent frames to obtain a reconstructed image obtained by reconstructing an input image (S25).

Here, the artificial neural network that determines whether or not to skip a frame is obtained by decoding the compression rate encoded without skipping the frame (R ₁ ), the compression rate encoded by skipping the frame (R ₂ ), and the encoded images encoded without skipping the frame. The distortion level (D ₁ ) of the decoded image is encoded and decoded by skipping frames, and the displacement (D) is calculated according to the distortion level (D ₂ ) of a plurality of restored frames inserted by interpolating the skipped frames, and the calculated displacement It can be learned by backpropagating by calculating the error between the normalized displacement and the skip probability value as a loss. Alternatively, it may be learned by comparing skipping according to displacement (D) with the binary value of the skip flag and performing backpropagation with a loss.

The method according to the present invention may be implemented as a computer program stored in a medium for execution on a computer. Here, computer readable media may be any available media that can be accessed by a computer, and may also include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, including read-only memory (ROM) dedicated memory), random access memory (RAM), compact disk (CD)-ROM, digital video disk (DVD)-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

Although the present invention has been described with reference to the embodiments shown in the drawings, this is only exemplary, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom.

Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

100: skip frame selection device 200: image restoration device
300: frame selection learning unit 110: frame setting unit
120: skip frame selector 130: encoder
210: decoder 220: skip frame interpolator
310: compression ratio calculation unit 320: image quality measurement unit
330: 2-dimensional projection unit 340: loss calculation unit

Claims (16)

a frame setting unit configured to set a skippable frame and a fixed frame among a plurality of frames of an input image;
Implemented as a pre-learned artificial neural network, skip frame selection for activating or inactivating a skip flag by determining whether to skip each of the skippable frames from each of the skippable frames and adjacent frames of each of the skippable frames according to the learned method. wealth; and
An encoder for obtaining an encoded image by encoding the remaining frames except for the skip frame in which the skip flag is activated in a predetermined manner,
The skip frame selection unit pre-learns based on a compression rate of an encoded image according to whether the skip frame is excluded or a quality difference between a decoded image obtained by decoding the encoded image and a reconstructed image obtained by reconstructing the skip frame according to whether or not the skip frame is excluded. skip frame selector. The method of claim 1, wherein the skip frame selector
The compression rate of the encoded image obtained by encoding a plurality of frames of the input image not excluding the skip frame and the quality of the decoded image obtained by decoding the same are calculated according to a predetermined method, and the compression rate and the image quality are calculated as coordinates in a two-dimensional space as axes. Projecting, calculating the compression rate of the encoded image encoding a plurality of frames excluding the skip frame and the quality of the reconstructed image obtained by decoding it and reconstructing the skip frame, and projecting it to the coordinates of the two-dimensional space, based on the displacement between the two coordinates Learned skip frame selection device. The method of claim 2, wherein the skip frame selector
A skip frame selection device for obtaining a skip probability value for the skippable frame according to a learned method, and inactivating or activating a skip flag according to whether or not the obtained skip probability value is greater than or equal to a predetermined reference value. The method of claim 3, wherein the skip frame selector
A fitting function following a pattern of compression rate and picture quality projected in the two-dimensional space when the skip frame is not excluded is obtained, and a fitting function is obtained, and when the skip frame is excluded, the coordinates between the projected compression rate and the picture quality are obtained. A skip frame selection device that learns based on displacement. The method of claim 4, wherein the skip frame selector
An apparatus for selecting skip frames in which, when skip frames are excluded, a difference between a displacement between a projected compression ratio and image quality coordinates and the fitting function and the skip probability value is back-propagated as a loss and learned. The method of claim 4, wherein the skip frame selector
Loss obtained by comparing the activation or inactivation state of the skip flag with a skip discrimination value determined according to whether the displacement between the coordinates of the projected compression rate and image quality and the fitting function when excluding the skip frame is equal to or greater than a predetermined reference displacement A skip frame selector that is learned through backpropagation. The method of claim 1, wherein the frame setting unit
A skip frame selection device for setting fixed frames at predetermined intervals from an initial frame among the plurality of frames and setting remaining frames as skippable frames. The method of claim 7, wherein the frame setting unit
A skip frame selector for applying each skippable frame and an adjacent frame of the skippable frame to the skip frame selector, except for a case where a skip flag for the adjacent frame is already activated, and a next skippable frame and an adjacent frame. . setting a skippable frame and a fixed frame among a plurality of frames of an input image;
activating or inactivating a skip flag by determining whether each of the skippable frames is skipped from each of the skippable frames and adjacent frames of each of the skippable frames according to a learned method using a pre-learned artificial neural network; and
Obtaining an encoded image by encoding remaining frames other than the skip frame in which the skip flag is activated in a predetermined manner,
The artificial neural network is pre-learned based on the compression rate of the encoded image according to whether or not the skip frame is excluded and the quality difference between the decoded image obtained by decoding the encoded image and the reconstructed image obtained by reconstructing the skip frame according to whether or not the skip frame is excluded How to select skip frames. The method of claim 9, wherein the artificial neural network
The compression rate of the encoded image obtained by encoding a plurality of frames of the input image not excluding the skip frame and the quality of the decoded image obtained by decoding the same are calculated according to a predetermined method, and the compression rate and the image quality are calculated as coordinates in a two-dimensional space as axes. Projecting, calculating the compression rate of the encoded image encoding a plurality of frames excluding the skip frame and the quality of the reconstructed image obtained by decoding it and reconstructing the skip frame, and projecting it to the coordinates of the two-dimensional space, based on the displacement between the two coordinates Learned skip frame screening method. 11. The method of claim 10, wherein activating or deactivating the skip flag
A skip frame selection method for obtaining a skip probability value for the skippable frame according to a learned method, and inactivating or activating a skip flag according to whether or not the obtained skip probability value is greater than or equal to a predetermined reference value. The method of claim 11, wherein the artificial neural network
A fitting function following a pattern of compression rate and picture quality projected in the two-dimensional space when the skip frame is not excluded is obtained, and a fitting function is obtained, and when the skip frame is excluded, the coordinates between the projected compression rate and the picture quality are obtained. A skip frame screening method learned based on displacement. The method of claim 12, wherein the artificial neural network
A method for selecting skip frames in which, when skip frames are excluded, a difference between a displacement between a projected compression ratio and image quality coordinates and the fitting function and the skip probability value is back-propagated as a loss and learned. The method of claim 12, wherein the artificial neural network
Loss obtained by comparing the activation or inactivation state of the skip flag with a skip discrimination value determined according to whether the displacement between the coordinates of the projected compression rate and image quality and the fitting function when excluding the skip frame is equal to or greater than a predetermined reference displacement A method for selecting skip frames that are learned through backpropagation. 10. The method of claim 9, wherein setting the fixed frame
A skip frame selection method of setting a fixed frame at a predetermined interval from an initial frame among the plurality of frames and setting remaining frames as skippable frames. 16. The method of claim 15, wherein setting the fixed frame
Each skippable frame and an adjacent frame of the skippable frame are applied to the artificial neural network, except when a skip flag for the adjacent frame is already activated, and a next skippable frame and an adjacent frame are applied.

Images