KR102136468B1 - Deep Learning based Video Frame Rate Conversion Method and Apparatus - Google Patents

Abstract

Provided is a method and apparatus for converting a video frame rate that interpolates frames at high quality and high speed while coping with various color image formats based on deep learning. An apparatus for converting a video frame rate according to an embodiment of the present invention includes an adjustment unit that adjusts a resolution for an image of each channel; A generation unit that generates an intermediate image from two consecutive images of t seconds and t+1 seconds whose resolution is adjusted by the adjusting unit; And an enhancement unit for improving the resolution of the intermediate image generated by the generation unit.
As a result, in the deep learning-based video frame rate conversion, the input of deep learning can be applied to various color formats such as YCbCr as well as the RGB color format. Frame rate conversion becomes possible.

Description

Deep Learning based Video Frame Rate Conversion Method and Apparatus}

The present invention relates to a method for converting a frame rate of a video, and more particularly, to a method and apparatus for increasing a frame rate at high speed by applying a frame interpolation technique and a resolution increase technique based on a deep learning technique.

1) Overview of video frame rate conversion technique

A video is composed of a set of continuous still images. In a video, a still image is called a frame, and the number of frames per unit time is called a frame rate of a video. For example, if it consists of 24 frames per second, the frame rate is 24 fps (frame per second).

The frame rate is determined by the intention of the photographer, the format of the image, and the limitations of the camera. In order for an observer to feel the image as a continuous screen, a frame rate of a certain amount or more is required, and if it is lower than this, the movement does not appear smooth. This phenomenon may vary depending on the size of the display, lighting, viewing distance, and the like.

To improve this, increasing the frame rate of the video by post-processing is called video frame rate conversion.

2) Prior art

The simplest way to increase the video frame rate is to repeat the frame. For example, when a 30 fps image is increased to a 60 fps image, one frame is repeatedly output for each frame. However, in this method, since the amount of information in the video is the same and the continuity for movement has not changed, the discomfort the viewer feels is the same.

To solve this, a technique for generating a virtual frame using successive frames has been developed. That is, an intermediate image of t+0.5 seconds is newly generated using an image between t seconds and t+1 seconds, and this is called a frame interpolation technique.

Various methods of frame interpolation have been developed and generally go through the following two steps. The first step is a step of acquiring a motion or an optical flow, and the second step is a step of generating an intermediate frame (Synthesis) based on the motion information.

In order for the motion of the object to appear smooth in the video, the intermediate image must correspond to the middle of the motion between the two images. Therefore, it is very important to accurately find the optical flow that contains the motion information of the object. Various techniques based on this have been proposed.

Recently, a deep learning algorithm has appeared, and is widely used in various fields such as computer vision and speech recognition, and shows superior performance compared to the conventional method. In line with this, various frame interpolation techniques using deep learning have appeared. These techniques have been studied continuously in recent years as they showed better interpolation results than conventional methods based on optical flow and synthesis.

3) Problems in the prior art

Conventional deep learning based technology assumes that the input image has a color format in RGB format. However, most videos are compressed and stored in YCbCr color format. In addition, the color difference signals Cb and Cr are sub-sampled, so the resolution is different from the brightness signal Y.

Therefore, in order to apply a general video to an existing method, it is necessary to up-sampling the color difference signal to match the brightness signal and image size, and then convert the YCbCr color format to RGB again. These color conversion tasks are inefficient and operate as a factor that makes real-time operation difficult, especially when the image size increases.

The second problem of the existing method is a problem due to the network structure, and for large resolutions such as 4K (3840x2160) resolution, it is impossible to operate due to insufficient GPU memory, or it shows a very slow operation speed. Such a phenomenon makes it difficult to apply a frame interpolation method using deep learning to commercial applications that require real-time computation.

The present invention has been devised to solve the above problems, and an object of the present invention is to provide a method and apparatus for converting a video frame rate that interpolates frames at high quality and high speed while coping with various color image formats based on deep learning. In the offer.

According to an embodiment of the present invention for achieving the above object, the apparatus for converting a video frame rate includes an adjustment unit that adjusts a resolution for an image of each channel; A generation unit that generates an intermediate image from two consecutive images of t seconds and t+1 seconds whose resolution is adjusted by the adjusting unit; And an enhancement unit for improving the resolution of the intermediate image generated by the generation unit.

The controller may subsample the resolution of the images of each channel in the same way.

The generating unit includes: an encoder unit for calculating the characteristics of the image with the adjusted resolution; A decoder unit that restores the calculated characteristics by the resolution size of the image; It may include; a synthesis unit for generating an intermediate image by using the output of the decoder and the two images whose resolution is adjusted by the adjustment unit.

The enhancement unit may improve the resolution of the intermediate image generated by the generation unit to the resolution of the original image.

The enhancement unit includes: a first adjustment unit that adjusts the resolution of the intermediate image to improve detail; A detail enhancement unit that improves details of an intermediate image whose resolution is adjusted using the original image; The first adjustment unit may include a second adjustment unit that adjusts the resolution of the intermediate image whose resolution is not adjusted.

The detail enhancement unit may receive only a selectively selected channel.

The selected channel may be a brightness signal.

The detail enhancement unit may improve the detail of the intermediate image by referring to the intermediate information generated by the generation unit.

The detail enhancement unit may be to use a deep learning network.

On the other hand, according to another embodiment of the present invention, the video frame rate converter comprises the steps of adjusting the resolution for the image of each channel; Generating an intermediate image from two consecutive images of t seconds and t+1 seconds with the adjusted resolution; And improving the resolution of the generated intermediate image.

On the other hand, according to another embodiment of the present invention, the video frame rate conversion apparatus is an adjustment unit for adjusting the resolution for the image of each channel; It includes; an enhancement unit for improving the resolution of the intermediate image generated by two consecutive images of t seconds and t+1 seconds whose resolution is adjusted by the adjusting unit.

On the other hand, according to another embodiment of the present invention, the video frame rate converter comprises the steps of adjusting the resolution for the image of each channel; And improving the resolution of the intermediate image generated by two consecutive images of t seconds and t+1 seconds whose resolution is adjusted by the adjustment unit.

As described above, according to embodiments of the present invention, in deep learning-based video frame rate conversion, the input of deep learning can be applied to various color formats such as YCbCr as well as the RGB color format. High-quality, high-speed frame rate conversion is possible using the resolution enhancement method at the same time.

1: Frame interpolation technology
Figure 2: Drawing of the invention
3: Subsampling of chrominance signal, Y: brightness signal, Cb, Cr: chrominance signal
4: Representative drawings of the present invention
5: A conceptual diagram of an intermediate frame generation unit and a resolution enhancement unit
6: Relationship between input and output resolution of the input image adjustment unit and the detail enhancement unit
Figure 7: Example of the present invention for YCbCr 4:2:0 format
8: Example 1 of the present invention for YCbCr 4:2:2 format
Figure 9: Example 2 of the present invention for YCbCr 4:2:2 format
Figure 10: Example 1 of the present invention for YCbCr 4:4:4 or RGB format
Figure 11: Example 2 of the present invention for YCbCr 4:4:4 or RGB format

Hereinafter, the present invention will be described in more detail with reference to the drawings.

The apparatus for converting a video frame rate according to an embodiment of the present invention includes an input image adjustment unit 110, an intermediate frame generation unit 120, and a resolution enhancement unit 130, as shown in FIG. 2.

Most videos are converted from RGB format to YCbCr format. In YCbCr, Y is a brightness signal and Cb and Cr are color difference signals. In general, color difference signals of YCbCr images of most moving pictures are used as sub-sampling.

Representative subsampling techniques of YCbCr include YCbCr4:4:4, YCbCr4:2:2, and YCbCr4:2:0 methods as shown in FIG. 2. YCbCr4:4:4 means an image format that does not subsample the color difference signals Cb and Cr signals, and the resolutions of Cb, Cr and Y are the same. In the case of YCbCr 4:2:2, the color difference signal is subsampled by 2 in the horizontal direction, and in YCbCr 4:2:0, the color difference signal is subsampled by 2 in the horizontal and vertical direction. In this way, the resolutions of the color difference signal and the brightness signal are different from each other by color format.

Let each of Y, Cb, and Cr be one color channel. In general, a deep learning network input must have the same resolution between channels so that each color channel can be processed simultaneously.

However, as mentioned above, the resolution between color channels is different according to the color format. In order to match the resolution between color channels, an embodiment of the present invention proposes a method of adjusting an input signal.

In FIG. 2, the input image adjusting unit 110 adjusts the resolution of the image of each channel to be the same before applying it to deep learning. That is, each channel is sub-sampled to have the same resolution.

For details, suppose that the input signal is composed of N channels, and the resolution for each channel is H ₁ × W ₁ , ..., H _N × W _N. Here, H is the vertical resolution of the image and W is the horizontal resolution.

The input image adjusting unit 110 transforms horizontal and vertical resolutions of all channels into the same resolution H _M x W _M through sub-sampling. H _M x W _M It is recommended that is not larger than the resolution of the channel with the largest resolution. This is because when the input resolution of a deep learning network increases, the amount of computation increases proportionally.

The input image adjusting unit 110 may reduce the computation amount of the interpolation processor by adjusting the resolution of the image. For reference, when processing by a separate network for each channel, the motions of the brightness and the color difference signal are predicted differently, and the signal in which the brightness signal and the color difference signal are synthesized may be inconsistent. In this way, the input image adjusting unit 110 adjusts the resolution of the input signal for each channel to make an image to be used as an input to the network.

The network structure proposed in the embodiment of the present invention is largely composed of two steps as shown in FIG. 5. The first is a network that generates an intermediate image and is referred to as an intermediate frame generator. This network may be a network that generates existing intermediate images. The second network is a network that adjusts the generated intermediate image to match the original resolution and is referred to as a resolution enhancement unit 130.

A video frame rate conversion apparatus according to an embodiment of the present invention will be described in more detail with reference to FIG. 4.

,

)을 입력으로 받는다. 두 프레임은 입력 영상 조정부(110)를 거쳐 두 프레임 간의 모든 채널의 영상 해상도는 일치된 상태가 된다.The intermediate frame generation unit 120 is an image of two consecutive frames of t seconds and t+1 seconds (

,

) As input. The two frames pass through the input image adjusting unit 110 and the image resolutions of all channels between the two frames are matched.

The intermediate frame generation unit 120 includes an encoder unit 121, a decoder unit 122, and a synthesis unit 123.

The encoder unit 121 is composed of a convolutional layer (convolutioal layer), a pooling layer (Pooling layer), an activation layer (Activation layer), and the like.

The output of the encoder unit 121 is an input of the decoder unit 122, and the decoder unit 122 is composed of a deconvolutional layer and an activation layer, but is not limited thereto.

The output of the decoder unit 122 and the network input enter the input of the synthesis unit 123 to generate a first intermediate (interpolated) image. The resolution of the primary intermediate image is the same as the resolution of the input of the intermediate frame generator 120. In the decoder unit 122, not only the output of the encoder unit 121 but also the intermediate result of the encoder unit 121 can be received as an input, and the performance can be improved by using the same.

5 shows the concepts of the encoder unit 121, the decoder unit 122, and the synthesis unit 123.

In the encoder unit 121, the resolution of the image is compressed to calculate a feature of the image, and the decoder unit 122 restores the characteristic by the resolution size of the image. The intermediate information of the encoder unit 121 is directly transmitted to the decoder unit 122, which is represented by an arrow in FIG.

In addition, intermediate information between the encoder unit 121 and the decoder unit 122 is also transmitted to the detail enhancement unit 132 and recycled.

The resolution enhancement unit 130 increases the resolution of the channel signal having the reduced resolution. The resolution of all channels of the primary intermediate frame generated in the intermediate frame generator 120 is H _M x W _M to be. The resolution of the original image is H ₁ x W ₁ , ..., H _N x W _N Therefore, the resolution enhancement unit 130 adjusts the size of the image to match the resolution of the original image. Since the resolution of the primary intermediate frame is smaller than the original resolution, the resolution enhancement unit 130 mainly performs an increase in resolution.

The resolution enhancement unit 130 is composed of an intermediate image adjustment unit 131, a detail enhancement unit 132, and an output image adjustment unit 133, of which only the detail enhancement unit 132 uses a deep learning network to generate a large amount of computation. in need.

Therefore, the input used in the detail enhancement unit 132 may be used only selectively selected channels, not all channels. In general, since most of the detail of the brightness signal is concentrated, it is recommended to use only the brightness signal as the input of the detail enhancement unit 132.

The intermediate image adjustment unit 131 adjusts the resolution of the input to be used in the detail enhancement unit 132. 6 shows a relationship between input and output resolutions of the input image adjustment unit 110 and the detail enhancement unit 132. The resolution of the n-th channel of the original image is H _n x W _n And the n-th channel is applied to the detail enhancement unit 132.

The resolution of the input/output of the detail enhancement unit 132 should be the same as the original resolution, and for this purpose, the resolution of the primary intermediate frame is adjusted by the input image adjustment unit 110 to be the same as the original resolution of the image size. The method of adjustment may use filters such as bilinear and bicubic, and is not limited to a specific method.

와

이다. 원본 입력 영상은 비록 다른 시간의 영상들이지만 원본 해상도의 디테일들을 보존하고 있다. 따라서 디테일 향상부(132)에는 이 정보를 사용하여 1차 중간 영상의 디테일을 개선하도록 한다.The input of the detail enhancement unit 132 is the first intermediate image and the original input image

Wow

to be. The original input image preserves the details of the original resolution, although the images are of different times. Therefore, the detail enhancement unit 132 uses this information to improve the detail of the primary intermediate image.

In addition, as an input of the resolution enhancement unit 130, intermediate results of the intermediate frame generation unit 120 are received as input. The intermediate process of the intermediate frame generation unit 120 is received as an input by the detail enhancement unit 132. This input includes the feature of the image, and this feature is used in the detail enhancement unit 132 to provide higher performance.

The output image adjusting unit 133 finally adjusts the channel resolution of each interpolated frame to be the same as the original channel resolution. The channel in which the resolution is improved in the detail enhancement unit 132 is the same as the resolution of the original image, but the image of the other channel may be different from the original resolution. In other words, the resolution adjustment is performed in the output image adjustment unit 133 only for the channel of the interpolated image different from the original resolution.

In an embodiment of the present invention, training of the network is recommended to first learn the intermediate frame generation unit 120 and then learn the detail enhancement unit 132. Or, after learning as above, it is recommended to learn the entire network at one time end-to-end. As another method, after learning the intermediate frame generation unit 120, it is recommended to learn the intermediate frame and the detail enhancement unit 132 at the same time.

The following shows various spherical examples of the present invention.

와

이라고 하자. 전체 네트워크의 출력은

이며 t’ 은 0과 1사이의 값이다. 즉 출력 영상은 t 와 t+1 사이의 영상이 된다. 여기서 t는 시간을 의미한다. 입력 영상이 YCbCr 컬러 포맷을 갖는다고 하면 입력 영상

와

는

와

로 표현할 수 있다.

와

은 밝기 신호를,

은 색차 신호를 의미한다. 입력 영상 조정부(110)의 입력은

와

이 되며 출력은

와

이 된다.

은 서브 샘플링된 영상을 의미하며

사이의 해상도는 모두 같아야 한다. 다시 말하면 채널 간 해상도는 같아야 한다. The input of the entire network is two consecutive frames in the video.

Wow

Let's say. The output of the entire network

And t 'is a value between 0 and 1. That is, the output image is an image between t and t +1. Where t means time. Assuming that the input image has a YCbCr color format, the input image

Wow

The

Wow

Can be expressed as

Wow

The brightness signal,

Means a color difference signal. The input of the input image adjusting unit 110 is

Wow

And the output is

Wow

It becomes.

Means sub-sampled image

The resolutions between them should all be the same. In other words, the resolution between channels should be the same.

과

은 모두 H/2 x W/2의 해상도를 갖는다. 출력 영상의 해상도는 갈 컬러 채널 중 가장 작은 해상도를 갖는 채널보다 작거나 같아야 한다. 즉, YCbCr 4:2:0영상의 경우에는 출력 해상도는 H/2 x W/2 보다 같거나 작아야 한다. 다른 실시예로 YCbCr 4:2:2 영상에 대하여 살펴보자. Y의 해상도를 H x W라고 하면 Cb, Cr의 해상도는 H x W/2이다. 그럼 입력 영상 조정부(110)의 출력 해상도는 H x W/2 이거나 그 이하여야 한다. 따라서 사용자의 설정에 따라

과

의 해상도는 H x W/2 가 될 수 도 있고, H/2 x W/2 등도 될 수 있다. An embodiment of the input image adjustment unit 110 will be described with reference to a video having a YCbCr 4:2:0 color format as shown in FIG. 7. When the resolution of the brightness signal Y is H x W, the color difference signals Cb and Cr have a resolution of H/2 x W/2. The output resolution of the video signal from the input image adjusting unit 110 is set to H/2 x W/2. In this case, only the signal of Y is subsampled horizontally and vertically twice to generate H/2 x W/2.

and

All have H/2 x W/2 resolution. The resolution of the output image should be less than or equal to the channel with the smallest resolution among the color channels to go. That is, in the case of a YCbCr 4:2:0 image, the output resolution should be equal to or smaller than H/2 x W/2. As another example, let's look at the YCbCr 4:2:2 image. If the resolution of Y is H x W, the resolution of Cb and Cr is H x W/2. Then, the output resolution of the input image adjusting unit 110 should be H x W/2 or less. Therefore, depending on the user's settings

and

The resolution may be H x W/2 or H/2 x W/2.

과

이다. 도 7의 YCbCr 4:2:0 포맷의 경우에는

과

이 같고 t 프레임도 마찬가지다. 중간 프레임 생성부의 출력은

이 된다.

는 t와 t+1 프레임 사이에서 보간된 중간 프레임이며

와

의 해상도는 동일하다. 중간 프레임 생성부(120)는 인코더부, 디코더부, 합성부로 구성되나 이에 한정되지 않는다. 기존의 프레임 보간 방법을 사용하여도 무관하다. 인코더부는 콘볼루셔널 레이어, 풀링 레이어, 활성 레이어로 구성되어 있다. 아래 도 처럼 입력 영상에 대하여 컨볼루션을 수행하고 풀링 레이어를 수행하고 활성 레이어를 수행한다. 풀링 레이어는 영상의 해상도를 감소시킨다. 일반적으로 가로, 세로 각각 2배식 감소시킨다. 컨볼루셔널 레이어의 출력을

, 풀링 레이어의 출력을

,활성 레이어의 출력을

이라고 한다. L은 레이어의 순서를 의미하며 이 각각의 레이어의 출력은 해상도 증가부의 입력이 된다. 디코더부는 인코더부에서 줄어든 해상도를 다시 증가시키면서 특징을 추출한다. 해상도 증가는 Bilinear 필터 또는 디콘볼루션 레이어 등 다양한 방법을 사용할 수 있으며 특정 방법에 제한되지 않는다. 각각의 업샘플링 레이어의 출력을

이라고 정의하며 L은 레이어의 순서를 의미한다. 각각의 업샘플링 레이어의 출력

은 해상도 향상부(130)의 입력으로 사용된다. 디코더부의 최종 출력

와

,

은 합성부의 입력이 된다. 합성은 콘볼루션 필터, 와핑 (Warping) 등 다양한 방법이 사용될 수 있으며 본 발명에서는 특정 방법에 제한되지 않는다. 합성부의 출력은

이 되며 이를 1차 중간 프레임으로 부른다.The input of the intermediate frame generator 120 is

and

to be. In the case of the YCbCr 4:2:0 format of FIG. 7

and

This is the same, and so is the t frame. The output of the intermediate frame generator

It becomes.

Is the intermediate frame interpolated between t and t+1 frames

Wow

The resolution is the same. The intermediate frame generation unit 120 includes an encoder unit, a decoder unit, and a synthesis unit, but is not limited thereto. It is not necessary to use the existing frame interpolation method. The encoder unit is composed of a convolutional layer, a pooling layer, and an active layer. As shown below, convolution is performed on the input image, a pooling layer is performed, and an active layer is performed. The pooling layer reduces the resolution of the image. In general, it is reduced by a factor of 2 in each of the horizontal and vertical directions. The output of the convolutional layer

, The output of the pooling layer

, Output the active layer

It is said. L denotes the order of the layers, and the output of each layer becomes an input of a resolution increasing unit. The decoder unit extracts features while increasing the reduced resolution in the encoder unit again. The increase in resolution may use various methods such as a bilinear filter or a deconvolution layer, and is not limited to a specific method. The output of each upsampling layer

And L is the order of the layers. Output of each upsampling layer

Is used as an input of the resolution enhancement unit 130. Decoder output

Wow

,

Is the input of the synthesis section. Synthesis may be a variety of methods, such as convolution filter, warping (Warping) is not limited to a specific method in the present invention. The output of the synthesizer

This is called the first intermediate frame.

The resolution of the first intermediate frame has a smaller resolution than the original image. Therefore, the resolution enhancement unit 130 increases the resolution of the first intermediate frame to perform the operation to have the same resolution as the original image. The input and output of the resolution enhancement unit 130 may be variously defined as inputs and outputs of the resolution enhancement unit 130, and some embodiments will be described below.

First, let's say that the original input image is a YCbCr 4:2:0 color format, and the resolution of Y is H x W, Cb, and Cr is H/2 X W/2. The input image adjustment unit 110 subsamples the resolution of Y to match the resolution of Cb and Cr with H/2 x W/2. Therefore, the resolution of the intermediate frame generation unit 120 is H/2 x W/2 in the case of Y, Cb, Cr, and in the case of Y, the resolution is reduced compared to the original image, but in the case of Cb, Cr, the resolution has not changed. Only the image is increased using the resolution enhancement unit 130. The resolution enhancement unit 130 is composed of two steps: an intermediate image adjustment unit 131 and a detail enhancement unit 132. The intermediate image adjustment unit 131 upsamples the first intermediate frame to make it equal to the size of the original image.

이고 출력은 H x W 해상도를 갖는 영상

이다. 디테일 향상부(132)의 입력은

와 원영상

,

이다.

의 디테일은 고해상도 디테일을 가지고 있는

,

의 협력을 받아 더욱 정교하게 복원될 수 있다. 기존의 다중 프레임 (multi-frame) 해상도 향상 방법은 여러 장의 저해상도 영상을 사용하여 고해상도 영상을 사용하였다. 그러나 본 발명에서는 저해상도 영상과 고해상도 영상을 동시에 이용하여 해상도 향상을 꾀하였다. 디테일 향상부(132)는 딥러닝 네트워크로 구성되어 있으며 출력은

이다. 디테일 향상부(132)를 위한 딥러닝 네트워크는 컨벌루셔널 레이어, 풀링 레이어, 활성 레이어 등으로 구성되어 있으며, 본 논문에서는 특정 네트워크 형태로 제한하지 않는다. 디테일 향상부(132)는 중간 프레임 생성부(120)의 중간 결과물을 입력으로 받아 디테일 향상부(132)의 중간 과정에 더하거나 적층하여 (concatenation) 성능을 개선할 수 있다. 중간 생성부의 중간 결과물과 디테일 향상부(132)의 중간 결과물의 해상도는 일치해야한다.

와 1차 중간 프레임의 색차 신호

,

이 최종 결과물이 된다. In the above case, the input of the intermediate image adjustment unit 131 is a first intermediate frame having a resolution of H/2 x W/2

And the output is H x W resolution image

to be. The input of the detail enhancement unit 132 is

And original video

,

to be.

The details of the

,

It can be restored more elaborately with the cooperation of. In the conventional multi-frame resolution enhancement method, high resolution images were used using multiple low-resolution images. However, in the present invention, the resolution is improved by simultaneously using a low-resolution image and a high-resolution image. The detail enhancement unit 132 is composed of a deep learning network and the output is

to be. The deep learning network for the detail enhancement unit 132 is composed of a convolutional layer, a pooling layer, and an active layer, and is not limited to a specific network type in this paper. The detail enhancement unit 132 may receive an intermediate result of the intermediate frame generation unit 120 as an input and add or stack the intermediate result of the detail enhancement unit 132 to improve the performance of the concatenation. The resolution of the intermediate result of the intermediate generation unit and the intermediate result of the detail enhancement unit 132 must match.

And 1st intermediate frame color difference signal

,

This is the final result.

와 1차 H x W/2의 해상도를 가지는 중간 프레임의 색차 신호

,

이 최종 결물이 된다. Let us consider an embodiment in which the original input image is a YCbCr 4:2:2 color format. Let the resolution of Y be H x W, the resolution of Cb and Cr be H x W/2. As shown in FIG. 8, the resolution of Y is subsampled by the input image adjusting unit 110 to match the resolution of Cb and Cr with H x W/2. Therefore, the resolution of the intermediate frame generation unit 120 is H x W/2 in the case of Y, Cb, Cr, and in the case of Y, the resolution is reduced compared to the original image, but in the case of Cb, Cr, the resolution has not changed, so only the Y image The resolution is increased by using the resolution enhancement unit 130. The operation method of the resolution enhancement unit 130 is the same as in the YCbCr 4:2:0 format above. H x W resolution

And 1st H x W/2 resolution intermediate frame color difference signal

,

This is the final payment.

Another example of the YCbCr 4:2:2 color format is illustrated in FIG. 9. Let the resolution of Y be H x W, the resolution of Cb and Cr be H x W/2. The input adjustment unit subsamples the resolution of Y, Cb, and Cr to H/2 x W/2. And it is processed in the same way as YCbCr 4:2:0 above. When this process is performed, the output of the detail enhancement unit 132 has a resolution of H x W for the Y signal, and a signal of H/2 x W/2 for the Cb and Cr signals. Since the signals of Cb and Cr are different from the original signal, the output image adjusting unit 133 adjusts the resolution of Cb and Cr to be the same as the original signal. As the adjustment method, a bicubic filter or the like can be used, and is not limited to a specific method. In this way, compared to the above method, the size of the input image processed in the network is small, so there is an advantage of saving computational amount and memory. And the advantage of being able to share the network used in YCbCr 4:2:0.

Let's consider the case where YCbCr 4:4:4 or RGB input image is input. In YCbCr 4:4:4 or RGB color format video, the resolution of all channels is the same, so an intermediate frame may be generated even if the input image adjusting unit 110 and the resolution enhancing unit 130 are not used. However, in order to reduce the computation amount, the input image adjustment unit 110 and the resolution enhancement unit 130 may be used in the present invention. For example, suppose the input image is in RGB format and the resolution of each channel is H x W. The input image adjustment unit 110 subsamples each channel to a size of H/2 x W/2 and then inputs it as an input of the intermediate frame generation unit 120. In addition, the first incremental frame having a resolution of H/2 x W/2 is restored to the resolution of H x W in the resolution enhancement unit 130. The advantage of speed in the entire process comes from that the computation amount of the intermediate frame generator 120 is higher than that of the resolution enhancement unit 130. Therefore, the intermediate frame generation unit 120 having a high computation amount is processed with a small resolution and is restored by the resolution enhancement unit 130 to reduce the computation amount. YCbCr 4:4:4 can be applied in a similar manner. If the resolution of each channel is H x W, the input image controller converts H/2 x W/2. Then, the Y, Cb, and Cr results of H/2 x W/2 are generated by the intermediate frame generator 120. The primary intermediate frame may be applied to the resolution enhancement unit 130 in two ways. The first method is to pass the deep learning network of the detail enhancement unit 132 for all channels Y, Cb, and Cr as shown in FIG. 10, as in the RGB image described above. The second method is to restore the high-definition image through only the Y region through the deep learning network as shown in FIG. 11 as in the YCbCr 4:2:2 method described above, and restore Cb and Cr through simple upscaling in the output image adjustment unit. Since Cb and Cr generally have a small detail area, they can be restored through simple upscaling. Such a method can be restored with a smaller computational amount than using a deep learning network for all channel images. In addition, it has the advantage that it can be used in common with the YCbCr4:2:0 network, so it has a high versatility.

On the other hand, the technical idea of the present invention can be applied to a computer-readable recording medium containing a computer program that performs functions of the apparatus and method according to the present embodiment. Further, the technical idea according to various embodiments of the present invention may be implemented in the form of computer-readable codes recorded on a computer-readable recording medium. The computer-readable recording medium can be any data storage device that can be read by a computer and stores data. Of course, the computer-readable recording medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, hard disk drive, or the like. In addition, computer-readable codes or programs stored on a computer-readable recording medium may be transmitted through a network connected between computers.

In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention pertains without departing from the gist of the present invention claimed in the claims. In addition, various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical idea or prospect of the present invention.

110: input image adjustment unit
120: intermediate frame generating unit
121: encoder unit
122: decoder unit
123: synthesis section
130: resolution enhancement unit
131: middle image adjustment unit
132: detail enhancement unit
133: output image adjustment unit

Claims (12)

Adjuster for adjusting the resolution for the image of each channel;
A generation unit that generates an intermediate image from two consecutive images of t seconds and t+1 seconds whose resolution is adjusted by the adjusting unit; And
Includes; an enhancement unit for improving the resolution of the intermediate image generated by the generation unit,
In the middle video,
t+0.5 second video,
The adjustment unit,
Video frame rate converter, characterized in that for sub-sampling the resolution for each channel of the video.
delete The method according to claim 1,
The generating part,
An encoder unit for calculating the characteristics of the image with the adjusted resolution;
A decoder unit that restores the calculated characteristics by the resolution size of the image;
And a synthesis unit that generates an intermediate image using the outputs of the decoder and the two images whose resolution is adjusted by the adjustment unit.
The method according to claim 1,
The improvement department,
Video frame rate converter, characterized in that to improve the resolution of the intermediate image generated by the generator to the resolution of the original image.
The method according to claim 4,
The improvement department,
A first adjustment unit for adjusting the resolution of the intermediate image to improve detail;
A detail enhancement unit that improves details of an intermediate image whose resolution is adjusted using the original image;
And a second adjustment unit that adjusts the resolution of the intermediate image whose resolution is not adjusted by the first adjustment unit.
The method according to claim 5,
The detail enhancement part,
Video frame rate converter, characterized in that only the selected channel is input.
The method according to claim 6,
The selected channel,
Video frame rate conversion device characterized in that the brightness signal.
The method according to claim 5,
The detail enhancement part,
A video frame rate conversion device characterized by improving the detail of the intermediate image by referring to the intermediate information generated by the generation unit.
The method according to claim 5,
The detail enhancement part,
A video frame rate conversion device characterized by using a deep learning network.
Adjusting a resolution for an image of each channel;
Generating an intermediate image from two consecutive images of t seconds and t+1 seconds with the adjusted resolution; And
And improving the resolution of the generated intermediate image.
In the middle video,
t+0.5 second video,
The adjustment step,
Video frame rate converter, characterized in that for sub-sampling the resolution for each channel of the video.
Adjuster for adjusting the resolution for the image of each channel;
It includes; an enhancement unit for improving the resolution of the intermediate image generated by two consecutive images of t seconds and t+1 seconds in which the resolution is adjusted by the adjusting unit;
In the middle video,
t+0.5 second video,
The adjustment unit,
Video frame rate converter, characterized in that for sub-sampling the resolution for each channel of the video.
Adjusting a resolution for an image of each channel;
Including the step of improving the resolution of the intermediate image generated by two consecutive images of t seconds and t+1 seconds, the resolution of which is adjusted by the adjustment unit,
In the middle video,
t+0.5 second video,
The adjustment step,
Video frame rate converter, characterized in that for sub-sampling the resolution for each channel of the video.

Images