KR20210019780A - System and method for processing image - Google Patents

Abstract

The present invention provides an image processing system. According to one embodiment of the present invention, the image processing system comprises: a learning module to generate learning data corresponding to a super resolution (SR) learning model including a first image quality convolution neural network to convert an image of a first image quality corresponding to input resolution into target resolution, a second image quality convolution neural network to convert an image of a second image quality corresponding to the input resolution and lower than the first image quality into the target resolution, and an image quality selection convolution neural network to select a combination of the first and second image quality convolution neural networks having the highest responsiveness to each image quality of the input resolution; and an image processing module to carry out SR processing for converting a prescribed image of the input resolution lower than the target resolution into the target resolution based on the learning data. Since the SR learning model does not correspond to any one specific image quality, image quality degradation of an image reconstructed by SR processing can be prevented regardless of whether an input image and the SR learning model correspond to the same image quality.

Description

Image processing system and method {SYSTEM AND METHOD FOR PROCESSING IMAGE}

The present invention relates to a system and method for performing SR (Super Resolution) processing based on a learning model.

In recent years, as display devices become larger and higher in resolution, the importance of up-scaling image processing for converting a low-resolution image into a high-definition, high-resolution image is increasing. In particular, in the case of a display device of UHD (Ultra High Definition) level resolution (e.g., 4K (3840 x 2160) or 8K (7680 x 4320)), in order to properly display an image of a lower resolution than UHD level resolution, It is necessary to have an up-scaling image processing, that is, a super resolution (SR) processing function, which converts an image of a lower resolution than the UHD-class resolution into a UHD-class resolution.

An example of such an SR processing method is a method of restoring an image of ultra-high resolution (e.g., UHD-level resolution) by matching several low-resolution (for example, FHD-class resolution (1920 x 1080)) images. have. In this case, there is a problem that an excessive amount of computation is required for motion compensation-based matching.

Another example of the SR processing method may be a bicubic interpolation method in which an ultra-high resolution image is reconstructed based on a neighboring pixel value of each pixel of a low-resolution image. In this case, there is a problem in that the image quality of the reconstructed ultra-high resolution image is blurred.

To improve this, a learning-based (e.g., deep learning) SR processing has been proposed. In the learning-based SR processing, a learning model is constructed by learning features of a process of converting a low-resolution image into an ultra-high-resolution image, and SR processing is performed based on the learning model.

In this case, there is an advantage in that an excessive amount of computation is not required for SR processing, and deterioration of image quality of a reconstructed image can be prevented. However, as each learning model is constructed based on a characteristic of a specific image quality, there is a problem in that the responsiveness to an image of an image quality different from the specific image quality is low. That is, when a learning model corresponding to a specific image quality is applied to SR processing for an input image having a quality different from that of a specific image quality, the training model does not respond to the input image, so that an output image of ultra high resolution according to SR processing is not derived. Otherwise, the output image may be derived in a state in which noise due to excessive SR processing is inserted. Accordingly, depending on whether the learning model corresponds to the same quality as the input image, there is a problem in that the quality of the image restored by the SR processing is deteriorated. In particular, in the case of a video streaming service such as IPTV, since input images of various quality are supplied, it is difficult to easily apply the learning-based SR processing.

The present invention is an image processing system capable of preventing deterioration in quality of an image reconstructed by SR processing regardless of whether a learning model and an input image correspond to the same quality in performing SR (Super Resolution) processing based on a learning model. And to provide a method thereof.

In addition, the present invention is to provide an image processing system and method that can be easily applied to SR processing of input images including various image quality.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention that are not mentioned can be understood by the following description, and will be more clearly understood by examples of the present invention. In addition, it will be easily understood that the objects and advantages of the present invention can be realized by the means shown in the claims and combinations thereof.

An example of the present invention is a learning module for generating training data corresponding to an SR (Super Resolution) learning model, and SR processing for converting an image of an input resolution lower than a predetermined target resolution to a target resolution based on the training data. It provides an image processing system including an image processing module to implement.

Here, the SR learning model includes a first quality convolution neural network for converting an image of a first quality corresponding to the input resolution to the target resolution, and a first quality convolution neural network corresponding to the input resolution and is higher than the first quality. Selecting a combination of a second quality convolutional neural network for converting an image of low second quality to the target resolution, and the first and second quality convolutional neural networks having the highest responsiveness to each quality of the input resolution It includes an image quality selection convolutional neural network.

That is, the SR learning model does not correspond to any one specific image quality, and may be reacted to images of various image quality by the first and second image quality convolutional neural networks and the image quality selection convolutional neural network.

Accordingly, the image processing module may convert an input image having an arbitrary image quality of the input resolution into the target resolution based on the SR learning model. Therefore, it is not necessary to examine whether the input image and the SR learning model correspond to the same image quality, and deterioration of the quality of the image reconstructed by the SR processing can be prevented.

In addition, since the image processing module performs SR processing based on the SR learning model, it can be easily applied even when input images of various quality, such as an image streaming service, are supplied.

The picture quality module includes a first picture quality learning unit for deriving a plurality of first upscaling parameters corresponding to the first picture quality convolutional neural network, and a plurality of second upscaling parameters corresponding to the second picture quality convolutional neural network And a second picture quality learning unit, and a picture quality selection learning unit for deriving a plurality of picture quality selection parameters corresponding to the picture quality selection convolutional neural network.

Here, the first quality learning unit repeats the learning process using each pixel of the first training image having the first quality and each pixel of the first correct answer image corresponding to the first training image and having the target resolution. The plurality of first upscaling parameters are derived. Similarly, the second image quality learning unit repeats a learning process using each pixel of the second learning image having the second image quality and each pixel of the second correct answer image corresponding to the second learning image and having the target resolution. The plurality of second upscaling parameters are derived.

In addition, the image quality selection learning unit repeats a learning process using each pixel of the third learning image having an arbitrary image quality of the input resolution and each pixel of the third correct answer image corresponding to the third learning image and having the target resolution. The plurality of picture quality selection parameters are derived.

In this way, as the first and second image quality learners and the image quality selection learners perform a learning process in units of pixels, the SR learning model becomes a result of learning features for SR processing in units of pixels. Accordingly, the image processing module may perform SR processing on a pixel-by-pixel basis based on the SR learning model. Therefore, since resolution information included in the header of the input video is unnecessary, storage of the input video is unnecessary.

In addition, since the SR processing may be performed on a pixel-by-pixel basis, the SR processing may also be performed on input images having various image quality. In addition, since the storage of the input image is unnecessary and the amount of calculation is reduced due to the SR processing in units of pixels, the timing controller of the display device may be sufficiently implemented as some functions.

In addition, another example of the present invention generates training data corresponding to a super resolution (SR) learning model including a first quality convolution neural network, a second quality convolution neural network, and a quality selection convolution neural network. The step of, constructing the SR (Super Resolution) learning model based on the training data, and SR processing of converting an input image having an arbitrary image quality of an input resolution into a target resolution based on the constructed SR learning model. It provides an image processing method including the step of performing.

The generating of the training data includes deriving a plurality of first upscaling parameters corresponding to the first quality convolutional neural network, and deriving a plurality of second upscaling parameters corresponding to the second quality convolutional neural network. Deriving a plurality of image quality selection parameters corresponding to the image quality selection convolutional neural network, and based on the plurality of first upscaling parameters, the plurality of second upscaling parameters, and the plurality of image quality selection parameters, the And generating training data.

In the step of deriving the plurality of first upscaling parameters, learning using each pixel of the first training image having a first image quality and each pixel of the first correct answer image corresponding to the first training image and having the target resolution Repeat the process. And, in the step of deriving the plurality of second upscaling parameters, each pixel of the second learning image having a second image quality and each pixel of the second correct answer image corresponding to the second learning image and having the target resolution are selected. Repeat the learning process used. Also, in the step of deriving the plurality of image quality selection parameters, each pixel of the third learning image having an arbitrary image quality of the input resolution and each pixel of the third correct answer image corresponding to the third learning image and having the target resolution are selected. Repeat the learning process used.

And, in the step of performing the SR processing, the input image is converted into the target resolution in pixel units based on the SR learning model.

An image processing system according to an embodiment of the present invention includes first and second image quality convolutional neural networks corresponding to different image quality of an input resolution lower than a predetermined target resolution, and a system having the highest responsiveness to each image quality of the input resolution. An input based on a learning module that generates training data for constructing an SR learning model including a quality selection convolutional neural network corresponding to the combination of the first and second quality convolutional neural networks, and the SR training model built according to the training data And an image processing module that performs SR processing for converting an input image having a resolution into a target resolution. As described above, the SR learning model is not a structure composed of a convolutional neural network corresponding to any one image quality, but a structure including the first and second image quality convolutional neural networks and the image quality selection convolutional neural network.

That is, a combination of the first and second quality convolutional neural networks corresponding to the quality of the input image can be selected by the quality selection convolutional neural network, regardless of whether the input image and the SR learning model correspond to the same quality. , Deterioration of the quality of an image reconstructed by SR processing can be prevented.

In addition to this, it can be easily applied even when an input image having various image quality is supplied, such as an image streaming service.

In addition, as the learning module generates an SR learning model by learning in units of pixels, the image processing module may perform SR processing in units of pixels. Therefore, even when images of various quality are mixed in any one image frame, it can be easily applied. Accordingly, since the memory for storing the image frame and the amount of computation corresponding to the image frame are unnecessary, it can be easily implemented by a timing controller of a display device.

1 is a diagram showing an image processing system according to an embodiment of the present invention.
2 is a diagram illustrating an example of an SR learning model corresponding to training data by the learning module of FIG. 1.
3 is a diagram illustrating an example of the learning module of FIG. 1 for deriving the SR learning model of FIG. 2.
FIG. 4 is a diagram illustrating an example of an operation of a first image quality learning unit of FIG. 3.
5 is a diagram illustrating an example of an operation of a second image quality learning unit of FIG. 3.
6 is a diagram illustrating an example of the reactivity of each of first and second quality convolutional neural networks corresponding to an image of a first quality.
7 is a diagram showing an example of the reactivity of each of first and second quality convolutional neural networks corresponding to an image of a second quality.
8 is a diagram illustrating an example of an operation of the image quality selection learning unit of FIG. 3.
9 is a diagram illustrating an example of the image processing module of FIG. 1.
10 is a diagram illustrating an example of an operation of the image processing module of FIG. 9.
11 is a diagram illustrating an image processing method according to an embodiment of the present invention.
12 is a diagram illustrating an example of an arbitrary display device.
13 is a diagram showing another example of an SR learning model corresponding to training data by the learning module of FIG. 1.
14 is a diagram illustrating another example of the learning module of FIG. 1 for deriving the SR learning model of FIG. 13.
15 is a diagram illustrating an image processing method according to another embodiment of the present invention.

The above-described objects, features, and advantages will be described later in detail with reference to the accompanying drawings, and accordingly, one of ordinary skill in the art to which the present invention pertains will be able to easily implement the technical idea of the present invention. In describing the present invention, if it is determined that a detailed description of known technologies related to the present invention may unnecessarily obscure the subject matter of the present invention, a detailed description will be omitted. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to indicate the same or similar elements.

Hereinafter, it means that an arbitrary component is disposed on the "top (or lower)" of the component or the "top (or lower)" of the component, the arbitrary component is arranged in contact with the top (or bottom) of the component. In addition, it may mean that other components may be interposed between the component and any component disposed on (or under) the component.

In addition, when a component is described as being "connected", "coupled" or "connected" to another component, the components may be directly connected or connected to each other, but other components are "interposed" between each component. It is to be understood that "or, each component may be "connected", "coupled" or "connected" through other components.

Hereinafter, an image processing system and method according to each embodiment of the present invention will be described with reference to the accompanying drawings.

1 is a diagram showing an image processing system according to an embodiment of the present invention.

2 is a diagram illustrating an example of an SR learning model corresponding to training data by the learning module of FIG. 1. 3 is a diagram illustrating an example of the learning module of FIG. 1 for deriving the SR learning model of FIG. 2.

FIG. 4 is a diagram illustrating an example of an operation of a first image quality learning unit of FIG. 3. 5 is a diagram illustrating an example of an operation of a second image quality learning unit of FIG. 3.

6 is a diagram illustrating an example of a reactivity of each of first and second quality convolutional neural networks corresponding to an image of a first quality. 7 is a diagram showing an example of the reactivity of each of first and second quality convolutional neural networks corresponding to an image of a second quality. 8 is a diagram illustrating an example of an operation of the image quality selection learning unit of FIG. 3.

The image processing system and method according to an embodiment of the present invention perform SR processing based on an SR learning model derived by repeated learning of SR processing for restoring an image of an input resolution lower than a predetermined target resolution to a target resolution. do.

As shown in FIG. 1, the image processing system 100 according to an embodiment of the present invention generates training data corresponding to an SR learning model for converting an image of an input resolution lower than a predetermined target resolution to a target resolution. SR processing for constructing an SR learning model based on the learning module 110 to perform and the learning data provided from the learning module 110, and converting an input image of an input resolution into an output image of a target resolution based on the constructed SR learning model. It includes an image processing module 120 to perform. Here, the target resolution may be 4K (3840 x 2160) or 8K (7680 x 4320), which is a UHD (Ultra High Definition) level resolution. In addition, the input resolution may be 1K (1920 x 1080), which is an FHD-level resolution.

As shown in FIG. 2, the SR learning module 200 by the learning module 110 includes a first quality convolutional neural network (convolutional neural network) for converting an image of a first quality corresponding to an input resolution into a target resolution; Convolution Neural Network (CNN) 201, a second quality convolutional neural network 202 corresponding to the input resolution and converting an image of a second quality lower than the first quality into a target resolution, and each quality of the input resolution. A picture quality selection convolutional neural network 210 for selecting a combination of the first and second picture quality convolutional neural networks 201 and 202 having the highest responsiveness is included.

The first quality convolutional neural network 201 corresponds to features of an operation that converts an image of a first quality corresponding to an input resolution to a target resolution. The first quality convolutional neural network 201 includes a plurality of first upscaling parameters a1, a2, a3, b1, b2, b3, c1, c2, and c3.

Similarly, the second image quality convolutional neural network 202 corresponds to an input resolution and corresponds to features of an operation for converting an image of a second image quality lower than the first image quality to a target resolution. The second picture quality convolutional neural network 202 includes a plurality of second upscaling parameters a4, a5, a6, b4, b5, b6, c4, c5, and c6.

For example, when the input resolution is FHD level, the first image quality may be 1080p (1920 x 1080), and the second image quality lower than the first image quality may be 720p (1280 x 720).

FIG. 2 shows that the first image quality convolutional neural network 201 is composed of three layers (a, b, c), and three first upscaling parameters (a1, a2, a3) are provided in each layer (a, b, c). )(b1, b2, b3) (c1, c2, c3) are arranged as an example, but this is only an example, and the first quality convolutional neural network 201 includes multiple layers and multiple layers It is natural that the first upscaling parameter may be included.

Similarly, it is natural that the second picture quality convolutional neural network 202 may also include a plurality of layers and a plurality of second upscaling parameters disposed in each layer.

The image quality selection convolutional neural network 210 corresponds to features of an operation that selects a combination of the first and second image quality convolutional neural networks 201 and 202 with the highest responsiveness to each image quality of the input resolution. The picture quality selection convolutional neural network 22 includes a plurality of picture quality selection parameters d1, d2, d3, e1, e2, and e3.

2, the image quality selection convolutional neural network 210 is composed of two layers (d, e), and three image quality selection parameters (d1, d2, d3) (e1, e2, e3) in each layer (d, e). ) Is arranged, but this is only an example, and it is natural that the quality selection convolutional neural network 210 may include a plurality of layers and a plurality of quality selection parameters disposed in each layer.

In this way, in order to construct the SR learning model 200 including the first and second image quality convolutional neural networks 201 and 202 and the image quality selection convolutional neural network 210, the learning module 110 corresponds to each neural network. Includes a learning department.

Specifically, as shown in FIG. 3, the learning module 110 includes a plurality of first upscaling parameters (a1, a2, a3, b1 in FIG. 2) corresponding to the first image quality convolutional neural network (201 in FIG. 2). , b2, b3, c1, c2, c3), a plurality of second upscaling parameters corresponding to the first image quality learning unit 111 and the second image quality convolutional neural network (202 in FIG. 2) (a4 in FIG. 2) , a5, a6, b4, b5, b6, c4, c5, c6) to derive a second image quality learning unit 112, a plurality of image quality selection parameters corresponding to the image quality selection convolutional neural network (210 in FIG. 2) (Fig. 2, d1, d2, d3, e1, e2, e3) of the image quality selection learning unit 113, and a plurality of first upscaling parameters, a plurality of second upscaling parameters, and a plurality of image quality selection parameters And a learning data generator 114 that generates learning data based on the learning data.

The first image quality learning unit 111 is configured by learning using a plurality of first learning images having a first image quality of the input resolution and a plurality of first correct answer images corresponding to the plurality of first learning images and having a target resolution. 1 A plurality of first upscaling parameters corresponding to the image quality convolutional neural network 201 are derived.

The first quality learning unit 111 repeats a learning process using each pixel of each first learning image and each pixel of each first correct answer image to derive a plurality of first upscaling parameters.

The first image quality learning unit 111 presets a plurality of first upscaling parameters to predetermined initial values, and updates a plurality of first upscaling parameters through each learning process. Here, the first image quality learning unit 111 may update the plurality of first upscaling parameters by modifying or deleting at least one of the plurality of first upscaling parameters or adding one or more first upscaling parameters. .

Specifically, as shown in FIG. 4, the first image quality learning unit 111 corresponds to a plurality of preset first upscaling parameters in any one learning process for deriving a plurality of first upscaling parameters. Based on the first quality convolutional neural network 201, a learning output pixel corresponding to a learning input block including any one target pixel in the first training image is derived. Then, the first quality learning unit 111 derives a difference between the correct answer pixel corresponding to the target pixel and the learning output pixel among the first correct answer image, and performs a plurality of first upscaling based on the difference between the correct answer pixel and the learning output pixel. Update parameters.

If the difference between the correct answer pixel and the learning output pixel is maintained within a threshold of the same range and more than a predetermined threshold number of times while repeating the learning process, the first quality learning unit 111 stops repetition of the learning process, and A plurality of first upscaling parameters (a1, a2, a3, b1, b2, b3, c1, c2, c3) corresponding to the learning output pixels whose difference is within the threshold of the same range may be finally output.

As shown in FIG. 3, the second image quality learning unit 112 corresponds to a plurality of second learning images having a second image quality of the input resolution and a plurality of second learning images, and a plurality of second correct answer images having a target resolution. A plurality of second upscaling parameters corresponding to the second image quality convolutional neural network 202 are derived through learning using.

The second image quality learning unit 112 repeats the learning process using each pixel of each second learning image and each pixel of each first correct answer image to derive a plurality of second upscaling parameters.

The second image quality learning unit 112 presets a plurality of second upscaling parameters to predetermined initial values, and updates a plurality of second upscaling parameters through each learning process. Here, the second image quality learning unit 112 may update the plurality of second upscaling parameters by modifying or deleting at least one of the plurality of second upscaling parameters or adding one or more second upscaling parameters. .

Specifically, as shown in FIG. 5, the second image quality learning unit 112 corresponds to a plurality of preset second upscaling parameters in any one learning process for deriving a plurality of second upscaling parameters. Based on the second image quality convolutional neural network 202, a learning output pixel corresponding to a learning input block including any one target pixel among the second training images is derived. Subsequently, the second image quality learning unit 112 derives a difference between the correct answer pixel corresponding to the target pixel and the learning output pixel among the second correct answer image, and performs a plurality of second upscaling based on the difference between the correct answer pixel and the learning output pixel. Update parameters.

If the difference between the correct answer pixel and the learning output pixel is maintained within a threshold of the same range and more than a predetermined threshold number of times while repeating the learning process, the second image quality learning unit 112 stops repetition of the learning process, and A plurality of second upscaling parameters (a4, a5, a6, b4, b5, b6, c4, c5, c6) corresponding to the learning output pixels whose difference is within the threshold of the same range may be finally output.

Meanwhile, since the first and second picture quality convolutional neural networks 201 and 202 are the result of learning for each picture quality, they exhibit normal reactivity for picture quality in a range similar to the corresponding picture quality. Here, the normal reactivity corresponds to the target image quality of the target resolution, and may be a relatively high level of reactivity.

As an example, the first quality convolutional neural network 201 is derived by learning to convert a plurality of first training images having a first quality into a target resolution. Accordingly, the first image quality convolutional neural network 201 may exhibit a normal reactivity with respect to the first image quality or an image quality in a range adjacent to the first image quality.

That is, as shown in FIG. 6A, an upscaling process for restoring a first input image Q1 having a first image quality or an image quality in a range adjacent to the first image quality to a target resolution is performed as a first image quality corresponding to the first image quality. When implemented based on the image quality convolutional neural network 201, features of the first image quality convolutional neural network 201 may react normally. As a result of the upscaling process for the first input image Q1 by the normal reactivity of the first image quality convolutional neural network 201, an output image T1-1 with a target resolution having a normal image quality may be derived. . Here, the normal image quality indicates within the target image quality range of the target resolution during the upscaling process.

On the other hand, as shown in FIG. 6B, the up-scaling process for restoring the second input image Q2 having an image quality different from the first image quality, such as the second image quality, to the target resolution, is performed by the first image quality convolutional neural network 201. ), the features of the first picture quality convolutional neural network 201 cannot normally respond to the upscaling process. As the upscaling process is not performed due to the abnormal reactivity of the first quality convolutional neural network 201, as a result of the upscaling process for the second input image Q2, the output image T1-2 of the target resolution ) May be derived as a low image quality without a clear image.

In addition, the second image quality convolutional neural network 202 is derived by learning to convert a plurality of second training images having a second image quality lower than the first image quality into a target resolution. Accordingly, the second image quality convolutional neural network 202 may exhibit a normal reactivity with respect to the second image quality or an image quality in a range adjacent to the second image quality.

That is, as shown in FIG. 7A, an upscaling process for restoring a second input image Q2 having a second image quality or an image quality in a range adjacent to the second image quality to a target resolution is performed as a second image quality corresponding to the second image quality. When implemented based on the image quality convolutional neural network 202, features of the second image quality convolutional neural network 202 may react normally. By the normal reactivity of the second image quality convolutional neural network 202, as a result of the upscaling process for the second input image Q2, an output image T2-2 with a target resolution having a normal image quality can be derived. have.

On the other hand, as shown in FIG. 7B, the up-scaling process for restoring the first input image Q1 having an image quality different from the second image quality, such as the first image quality, to the target resolution, is performed by the second image quality convolutional neural network 202. ), the features of the second image quality convolutional neural network 202 cannot normally respond to the upscaling process. As the upscaling process is excessively performed due to the abnormal reactivity of the second image quality convolutional neural network 202, as a result of the upscaling process for the first input image Q1, the output image T2-1 of the target resolution ) Can be derived with low image quality with a large amount of noise inserted.

In this way, since the reactivity of each of the first and second quality convolutional neural networks 201 and 202 corresponds to the quality of the input images Q1 and Q2, the first and second quality convolutional neural networks 201 and 202 for the input image The quality of the input image can be learned based on the reactivity of ).

That is, as shown in FIG. 3, the image quality selection learning unit 113 includes the first and second image quality convolutional neural networks 201 and 202 by the first and second image quality learning units 111 and 112 and the input resolution. A plurality of image quality selection convolutional neural networks 210 corresponding to a plurality of third training images and a plurality of third training images each having an arbitrary image quality of and through learning using a plurality of third correct answer images having a target resolution Derive quality selection parameters. Here, the plurality of third learning images correspond to various image quality of input resolution. As an example, some of the plurality of third learning images may have an image quality similar to the first image quality, and some of the third learning images may have an image quality similar to the second image quality.

Like the first and second image quality learning units 111 and 112, the image quality selection learning unit 113 performs a learning process in pixel units. That is, the quality selection learning unit 113 repeats the learning process using the first and second quality convolutional neural networks 201 and 202, each pixel of each third training image, and each pixel of each third correct answer image Derive the quality selection parameters of.

The image quality selection learning unit 113 presets a plurality of image quality selection parameters to predetermined initial values, and updates the plurality of image quality selection parameters through each learning process. Here, the quality selection learning unit 113 may update the plurality of quality selection parameters by modifying or deleting at least one of the plurality of quality selection parameters or adding one or more quality selection parameters.

Specifically, as shown in FIG. 8, the image quality selection learning unit 113 includes the first and second image quality convolution neural networks 201 and 202 in any one learning process for deriving a plurality of image quality selection parameters. Based on the image quality selection convolutional neural network 210 corresponding to a plurality of preset image quality selection parameters, a learning output pixel corresponding to a learning input block including any one target pixel among the third training images is derived. Subsequently, the quality selection learning unit 113 derives a difference between the correct answer pixel corresponding to the target pixel and the learning output pixel among the third correct answer image, and updates a plurality of quality selection parameters based on the difference between the correct answer pixel and the learning output pixel. do.

If the difference between the correct answer pixel and the learning output pixel is maintained within a threshold of the same range and more than a predetermined threshold number of times while repeating the learning process, the quality selection learning unit 113 stops repetition of the learning process and A plurality of image quality selection parameters d1, d2, d3, e1, e2, and e3 corresponding to the learning output pixels whose difference is within the threshold of the same range may be finally output.

Subsequently, as shown in FIG. 3, the learning data generation unit 114 includes a plurality of first upscaling parameters by the first quality learning unit 111 and a plurality of second upscaling parameters by the second quality learning unit 112. Learning data is generated by collecting a scaling parameter and a plurality of image quality selection parameters by the image quality selection learning unit 113.

As described above, according to an embodiment of the present invention, the learning module 110 includes first and second image quality convolutional neural networks 201 and 202 corresponding to different image quality and image quality selection convolutional neural network 210 Learning data corresponding to the SR learning model 200 is generated. Accordingly, the SR learning model 200 does not exhibit normal reactivity only for any one image quality, but may exhibit a normal reactivity to any image quality of the input resolution.

In addition, each of the first and second image quality learning units 111 and 112 and the image quality selection learning unit 113 of the learning module 110 builds a convolutional neural network through learning performed in units of pixels. That is, the SR learning model 200 does not correspond to an image unit, but corresponds to a pixel unit. Since the SR learning model 200 can react to SR processing for pixels having an arbitrary image quality of an input resolution, it can be easily applied to SR processing for a part of an image.

In addition, the learning module 110 provides learning data corresponding to the SR learning module 200 constructed through pixel-based learning. Therefore, the image processing module 120 performs SR processing on a pixel-by-pixel basis based on the SR learning module 200 constructed from learning data by the learning module 110.

9 is a diagram illustrating an example of the image processing module of FIG. 1. 10 is a diagram illustrating an example of an operation of the image processing module of FIG. 9.

9, in the image processing system 100 according to an embodiment of the present invention, the image processing module 120 includes a memory unit 121 for holding learning data provided from the learning module 110, And constructing an SR learning model based on the learning data of the memory unit 121, performing SR processing on the input image of the input resolution based on the constructed SR learning model, and outputting the target resolution restored by the SR processing. It may include an SR engine unit 122 that outputs an image.

As mentioned above, the training data provided by the learning module 110 corresponds to the SR learning module 200 constructed through pixel-based learning. Therefore, the SR engine unit 122 performs SR processing on a pixel-by-pixel basis based on the SR learning module 200.

That is, as shown in FIG. 10, the SR engine unit 122 constructs the SR learning module 200 based on the learning data of the memory unit 121.

Here, the training data may include parameters necessary for building the SR learning model 200. That is, as shown in FIG. 2, when the SR learning model 200 includes the first and second image quality convolutional neural networks 201 and 202 and the image quality selection convolutional neural network 210, the training data is 1 quality selection parameters (a1, a2, a3, b1, b2, b3, c1, c2, c3), multiple second quality selection parameters (a4, a5, a6, b4, b5, b6, c4, c5, c6) , And a plurality of image quality selection parameters d1, d2, d3, e1, e2, and e3.

The SR engine unit 122 SR-processes the transform target block including each target pixel of the input image II based on the constructed SR learning model 200 to sequentially generate each transform output pixel. Subsequently, the SR engine unit 122 may generate an output image OI by collecting the generated transformed output pixels.

Next, an image processing method according to an embodiment of the present invention will be described.

11 is a diagram illustrating an image processing method according to an embodiment of the present invention.

11, in the image processing method according to an embodiment of the present invention, the learning module 110 includes first and second quality convolutional neural networks 201 and 202 and a quality selection convolutional neural network 210. Generating training data corresponding to the included SR learning module 200 (S10), the image processing module 120 constructing an SR learning model based on the training data (S20), and the image processing module 120 And performing SR processing of converting an input image having an arbitrary image quality of an input resolution into a target resolution based on the constructed SR learning model (S30).

Here, the SR learning module 200 includes a first quality convolutional neural network 201 for converting an image of a first quality corresponding to an input resolution to a target resolution, and a second quality convolutional network 201 corresponding to the input resolution and lower than the first quality. Select a combination of a second quality convolutional neural network 202 for converting a quality image to a target resolution, and the first and second quality convolutional neural networks 201 and 202 with the highest responsiveness to each quality of the input resolution It includes an image quality selection convolutional neural network 210 to perform.

Specifically, the step of generating the learning data (S10) is performed through learning using a plurality of first learning images having a first image quality and a plurality of first correct answer images having a target resolution corresponding to the plurality of first learning images. 1 Step of deriving a plurality of first upscaling parameters (a1, a2, a3, b1, b2, b3, c1, c2, c3 in FIG. 2) corresponding to the image quality convolutional neural network 201 (S11), the second A plurality of second training images corresponding to the second image quality convolutional neural network 202 through learning using a plurality of second training images having image quality and a plurality of second correct answer images corresponding to the plurality of second training images and having a target resolution Step of deriving upscaling parameters (a4, a5, a6, b4, b5, b6, c4, c5, c6 in FIG. 2) (S12), a plurality of third training images each having an arbitrary image quality of the input resolution and a plurality of A plurality of quality selection parameters corresponding to the image quality selection convolutional neural network 210 (d1, d2, d3, e1 in FIG. 2) through learning using a plurality of third correct answer images corresponding to the third training image and having a target resolution. Deriving e2, e3) (S13), and a plurality of first upscaling parameters a1, a2, a3, b1, b2, b3, c1, c2, c3, and a plurality of second upscaling parameters a4, a5, a6, b4, b5, b6, c4, c5, c6) and generating training data based on a plurality of quality selection parameters d1, d2, d3, e1, e2, e3 (S14). .

As shown in FIG. 4, in the step of deriving a plurality of first upscaling parameters (S11), the first quality learning unit 111 of the learning module 110 includes each pixel of each first learning image and each first The learning process using each pixel of the correct answer image is repeated to derive a plurality of first upscaling parameters. Specifically, in any one learning process, the first image quality learning unit 111 is based on a first image quality convolutional neural network corresponding to a plurality of preset first upscaling parameters. A learning output pixel, which is a result of performing the SR processing on the learning input block including, is derived, and a plurality of first upscaling parameters are calculated based on the difference between the correct answer pixel corresponding to the target pixel and the learning output pixel among the first correct answer image. Update.

In addition, the first image quality learning unit 111 stops the learning process and stops the learning process when a learning output pixel whose difference from the correct answer pixel is within the threshold within the same range in the learning process of the threshold number of times is derived. A plurality of first upscaling parameters corresponding to the lusion neural network may be output.

5, in the step (S12) of deriving a plurality of second upscaling parameters, the second image quality learning unit 112 of the learning module 110 includes each pixel of each second learning image and each second The learning process using each pixel of the correct answer image is repeated to derive a plurality of second upscaling parameters. Specifically, in any one learning process, the second image quality learning unit 112 includes any one target pixel among the second learning images based on the second image quality convolutional neural network corresponding to a plurality of preset second upscaling parameters. A learning output pixel, which is a result of performing the SR processing on the learning input block including, is derived, and a plurality of second upscaling parameters are determined based on the difference between the correct answer pixel corresponding to the target pixel and the learning output pixel of the second correct answer image. Update.

And, in the learning process of the threshold number of times, when a learning output pixel whose difference from the correct answer pixel is within the threshold of the same range is derived, the learning process is stopped, and the second quality of the current learning process is A plurality of second upscaling parameters corresponding to the convolutional neural network may be output.

As shown in FIG. 8, in the step (S13) of deriving a plurality of quality selection parameters, the quality selection learning unit 113 of the learning module 110 includes first and second quality convolutional neural networks and each of the third training images. The learning process using each pixel of and each pixel of each third correct answer image is repeated to derive a plurality of quality selection parameters. Specifically, in any one learning process, the first and second image quality convolutional neural networks and the image quality selection convolutional neural network corresponding to a plurality of preset image quality selection parameters include any one target pixel among the third training images. A learning output pixel, which is a result of performing SR processing on the learning input block, is derived, and a plurality of quality selection parameters are updated based on the difference between the correct answer pixel corresponding to the target pixel and the learning output pixel of the third correct answer image.

And, in the learning process of the threshold number of times, the quality selection learning unit 113 stops the learning process, and when a learning output pixel whose difference from the correct answer pixel is within the threshold of the same range is derived, and stops the learning process, and selects the quality of the current learning process. A plurality of quality selection parameters corresponding to the neural network may be output.

Thereafter, the training data generation unit 114 of the learning module 110 may generate training data by collecting a plurality of first upscaling parameters, a plurality of second upscaling parameters, and a plurality of quality selection parameters. (S14)

The learning module 110 provides training data to the image processing module 120.

The memory unit 121 of the image processing module 120 stores learning data provided from the learning module 110.

The SR engine unit 122 of the image processing module 120 constructs an SR learning module based on the learning data of the memory unit 121. (S20)

Further, the SR engine unit 122 derives each pixel of the output image by performing SR processing on each pixel of the input image based on the constructed SR learning module. (S30)

As described above, according to an embodiment of the present invention, the learning module 110 includes the first and second image quality convolutional neural networks 201 and 202 and the image quality selection convolutional neural network 210 through pixel-based learning. Learning data corresponding to the SR learning model 200 is derived. Accordingly, the image processing module 120 may perform SR processing on pixels of an arbitrary quality based on the SR learning model 200 built according to the training data, regardless of the quality of the input image. Accordingly, the image processing module 120 does not need to have a means for determining the image quality and selecting an SR learning model corresponding to the image quality.

If the SR learning model is a result of learning for each quality, the image processing module must have a means for analyzing the quality of the input image and selecting an SR learning model corresponding to the quality of the input image. In addition, since the image processing module can analyze the quality of the input image only after all the input images have been received, a means for holding the entire information of the input image must be provided. Therefore, there is a problem that it is not easy to implement the image processing module. Further, there is a problem in that real-time display of the image is limited due to the delay time required to analyze the quality of the input image after receiving the entire information of the input image.

In order to reduce the delay time required for image quality analysis, the image processing module may perform SR processing of the current input image based on the SR learning model corresponding to the image quality of the previous input image. In this case, if the image quality of the current input image is different from the image quality of the previous input image, since the SR processing cannot be performed normally, the image quality of the output image restored by the SR processing is deteriorated. Accordingly, there is a problem in that there is a limitation in improving the reliability and accuracy for SR processing.

On the other hand, according to an embodiment of the present invention, the learning module 110 includes the first and second image quality convolutional neural networks 201 and 202 and the image quality selection convolutional neural network 210 through pixel-based learning. Learning data corresponding to the learning model 200 is generated. In addition, the image processing module 120 may perform SR processing on pixels of arbitrary quality based on the SR learning model 200.

Accordingly, since the image processing module 120 does not need to analyze the image quality, a delay time consumed in analyzing the image quality may be excluded. Thereby, there is an advantage to be advantageous for real-time display.

In addition, since the SR learning model 200 responds to an arbitrary image quality of each pixel, deterioration of the image quality of the output image restored by the SR process can be prevented. Accordingly, reliability and accuracy for SR processing can be improved, and thus display quality can be improved.

The image processing module 120 has an advantage in being implemented as a partial function of a timing controller of a display device.

12 is a diagram illustrating an example of an arbitrary display device.

As shown in FIG. 12, the display device 300 includes a display panel 310 and panel driving units 320, 330, and 340 that drive the display panel 310.

The display panel 310 defines a plurality of pixel areas P 301 arranged in a matrix in the display area AA, and includes data lines D1 to Dm and gate lines G1 for driving each pixel area 301. ~Gn). The data lines D1 to Dm correspond to each of the plurality of pixel regions 301, each of which is formed of pixel regions arranged in a vertical direction. The scan lines G1 to Gn correspond to each horizontal line made of pixel regions arranged in a horizontal direction among the plurality of pixel regions 301.

In addition, the panel driving units 320, 330, and 340 drive the data driving unit 320 for driving the data lines D1 to Dm of the display panel 310 and the gate lines G1 to Gn of the display panel 310 And a timing controller 340 that controls driving timings of the data driver 320 and the gate driver 330 based on the gate driver 330 and the image signal.

The timing controller 340 is a circuit controller that supplies a data processing unit 341 for processing a signal for an image signal input from the outside, and a control signal for controlling driving of the data driving unit 320 and the gate driving unit 330 (342) may be included.

The data processing unit 341 provides a reconstructed image by SR processing the image signal input from the outside according to the resolution of the display panel 310, rearranges digital video data (RGB) corresponding to the reconstructed image, and rearranges the digital video. Data (RGB') can be supplied.

The circuit control unit 342 operates the data driver 320 based on timing signals such as a vertical synchronization signal Vsync, a horizontal synchronization signal Hsync, a dot clock signal DCLK, and a data enable signal DE. A data control signal DDC for controlling a signal and a gate control signal GDC for controlling an operation timing of the gate driver 330 may be supplied.

The gate driver 330 sequentially supplies a gate signal for selecting the pixel region 301 of a horizontal line in which data is to be written based on the gate control signal GDC to each of the gate lines G1 to Gn.

The data driver 320 converts the rearranged digital video data RGB' into an analog data voltage based on the data control signal DDC. In addition, the data driver 320 supplies a data signal VDATA to the pixel region 301 of each horizontal line during each horizontal period based on the rearranged digital video data RGB'.

As described above, since the image processing system according to the exemplary embodiment of the present invention can perform SR processing on pixels of arbitrary quality, the amount of calculation and storage means for each image are unnecessary, so some functions of the timing controller 340 There is a more suitable advantage to be equipped with.

Meanwhile, an embodiment of the present invention exemplifies an SR learning model 200 including first and second image quality convolutional neural networks 201 and 202 corresponding to first and second image quality.

Alternatively, the SR learning model 200 may include a plurality of image quality convolutional neural networks corresponding to different image quality of the input resolution.

13 is a diagram illustrating another example of an SR learning model corresponding to training data by the learning module of FIG. 1. 14 is a diagram illustrating another example of the learning module of FIG. 1 for deriving the SR learning model of FIG. 13. 15 is a diagram illustrating an image processing method according to another embodiment of the present invention.

As shown in FIG. 13, the SR learning model 200' according to another embodiment of the present invention corresponds to an input resolution and converts an image of a third quality lower than the second quality into a target resolution. The first, second, and third image quality convolutional neural networks 201, 202, and 203 further including a convolutional neural network 203, wherein the image quality selection convolutional neural network 210 ′ has a higher responsiveness to each image quality of the input resolution. Except for selecting a combination of ), since it is the same as the embodiment of the present invention, redundant descriptions will be omitted below.

According to another embodiment of the present invention, the SR learning model 200' includes first, second, and third image quality convolutional neural networks 201, 202, and 203 corresponding to different image quality of the input resolution, and input resolution. And a picture quality selection convolutional neural network 210' for selecting a combination of the first, second, and third picture quality convolutional neural networks 201, 202, and 203 corresponding to each picture quality of.

The third image quality convolutional neural network 203 corresponds to an input resolution and corresponds to features of an operation that converts an image of a third image quality lower than the second image quality to a target resolution. The third image quality convolutional neural network 203 includes a plurality of third upscaling parameters a7, a8, a9, b7, b8, b9, c7, c8, and c9.

For example, when the input resolution is FHD, the first quality is 1080p (1920 x 1080), the second quality lower than the first quality is 720p (1280 x 720), and the third quality lower than the second quality is 360p. (640 x 360).

In the illustration of FIG. 13, the third image quality convolutional neural network 203 is composed of three layers (a, b, c), and three third upscaling parameters (a7, a8) in each layer (a, b, c). , a9) (b7, b8, b9) (c7, c8, c9) is illustrated to be arranged, but this is only an example, and the third quality convolutional neural network 203 is arranged in a plurality of layers and each layer. It is natural that a number of third upscaling parameters can be included.

The image quality selection convolutional neural network 210' is an operation that selects a combination of the first, second, and third image quality convolutional neural networks 201, 202, and 203 with the highest responsiveness to each image quality of the input resolution. Corresponds to

As shown in FIG. 14, the learning module 110 ′ according to another embodiment of the present invention further includes a third image quality learning unit 115, a third image quality learning unit 115, and an image quality selection learning unit ( 113 ′) and the learning data generation unit 114 ′ are the same as the exemplary embodiment illustrated in FIG. 3 except that the output of the third image quality learning unit 115 is further supplied, and thus a redundant description will be omitted below. .

According to another embodiment of the present invention, the learning module 110' includes a first quality learning unit 111, a second quality learning unit 112, a third quality learning unit 115, and a quality selection learning unit 113. ') and a learning data generation unit 114'.

The third image quality learning unit 115 corresponds to a plurality of fourth learning images having a third image quality of the input resolution and a plurality of fourth learning images, and uses a plurality of fourth correct answer images having a target resolution. A plurality of third upscaling parameters corresponding to the image quality convolutional neural network 203 are derived.

The third image quality learning unit 115 derives a plurality of third upscaling parameters by repeating a learning process using each pixel of each fourth learning image and each pixel of each fourth correct answer image.

The third image quality learning unit 115 presets a plurality of third upscaling parameters to predetermined initial values, and updates a plurality of third upscaling parameters through each learning process. Here, the third image quality learning unit 115 may update the plurality of third upscaling parameters by modifying or deleting at least one of the plurality of third upscaling parameters or adding one or more third upscaling parameters. .

Similar to the operation of the first and second image quality learning units 111 and 112 shown in FIGS. 4 and 5, the third image quality learning unit 115 learns any one for deriving a plurality of third upscaling parameters. In the process, based on the third image quality convolutional neural network 203 corresponding to a plurality of preset upscaling parameters, a learning output pixel corresponding to a learning input block including any one target pixel among the third training images is selected. To derive. Subsequently, the third image quality learning unit 115 derives a difference between the correct answer pixel corresponding to the target pixel and the learning output pixel among the third correct answer image, and performs a plurality of third upscaling based on the difference between the correct answer pixel and the learning output pixel. Update parameters.

If the difference between the correct answer pixel and the learning output pixel is maintained within the threshold of the same range and more than a predetermined threshold number of times while repeating the learning process, the third quality learning unit 115 stops repetition of the learning process, and A plurality of third upscaling parameters (a7, a8, a9, b7, b8, b9, c7, c8, c9) corresponding to the learning output pixels in which the difference is within the threshold of the same range may be finally output.

The image quality selection learning unit 113' includes first, second, and third image quality convolutional neural networks 201, 202, and 203 by the first, second, and third image quality learning units 111, 112, 115, and Corresponds to a plurality of third learning images each having an arbitrary image quality of the input resolution and a plurality of third learning images, and corresponding to the image quality selection convolutional neural network 210' through learning using a plurality of third correct answer images having a target resolution. A plurality of picture quality selection parameters are derived. Here, some of the plurality of third learning images may have image quality in a range similar to the first image quality, others may have image quality in a range similar to the second image quality, and others may have image quality in a range similar to the third image quality. have.

The quality selection learning unit 113' is a learning process using the first, second, and third quality convolutional neural networks 201, 202, and 203, each pixel of each third training image, and each pixel of each third correct answer image. Iteratively derives a plurality of quality selection parameters.

The image quality selection learning unit 113' presets a plurality of image quality selection parameters to predetermined initial values, and updates a plurality of image quality selection parameters through each learning process.

The learning data generation unit 114 ′ includes a plurality of first upscaling parameters by the first quality learning unit 111, a plurality of second upscaling parameters by the second quality learning unit 112, and a third quality learning unit. A plurality of third upscaling parameters by 115 and a plurality of picture quality selection parameters by the picture quality selection learning unit 113' are collected to generate learning data.

As shown in FIG. 15, in the image processing method according to another embodiment of the present invention, the learning module 110' selects the first, second, and third image quality convolutional neural networks 201, 202, and 203 and image quality. Except for the step (S10') of generating training data corresponding to the SR learning module 200' including the convolutional neural network 210', it is the same as the embodiment shown in FIG. Omit it.

In the step of generating training data (S10'), a plurality of first upscaling parameters corresponding to the first quality convolutional neural network 201 (a1, a2, a3, b1, b2, b3, c1, c2 in FIG. 13) Step of deriving c3) (S11), a plurality of second upscaling parameters corresponding to the second image quality convolutional neural network 202 (a4, a5, a6, b4, b5, b6, c4, c5, c6 in FIG. 13) ) Deriving (S12), a plurality of third upscaling parameters corresponding to the third image quality convolutional neural network 203 (a7, a8, a9, b7, b8, b9, c7, c8, c9 in FIG. 13) A step of deriving (S15), a step of deriving a plurality of image quality selection parameters (d4, d5, d6, e4, e5, e6 in FIG. 13) corresponding to the image quality selection convolutional neural network 210 (S13'), and A plurality of first upscaling parameters (a1, a2, a3, b1, b2, b3, c1, c2, c3), a plurality of second upscaling parameters (a4, a5, a6, b4, b5, b6, c4, c5) , c6), a plurality of third upscaling parameters (a7, a8, a9, b7, b8, b9, c7, c8, c9 in FIG. 13) and a plurality of image quality selection parameters (d1, d2, d3, e1, e2, and generating learning data (S14') based on e3).

In the step of deriving a plurality of third upscaling parameters (S15), the third image quality learning unit 115 of the learning module 110 ′ selects each pixel of each fourth learning image and each pixel of each fourth correct answer image. The used learning process is repeated to derive a plurality of third upscaling parameters.

The third image quality learning unit 115 learns including any one target pixel among the fourth training images based on a third image quality convolutional neural network corresponding to a plurality of third upscaling parameters preset in any one learning process. A learning output pixel, which is a result of performing SR processing on the input block, is derived, and a plurality of third upscaling parameters are updated based on a difference between the correct answer pixel corresponding to the target pixel and the learning output pixel of the fourth correct answer image.

In addition, the third image quality learning unit 115 stops the learning process and stops the learning process when a learning output pixel whose difference from the correct answer pixel is within the threshold within the same range is derived in the learning process of the threshold number of times, and A plurality of third upscaling parameters corresponding to the lusion neural network may be output.

In the step of deriving a plurality of image quality selection parameters (S13'), the image quality selection learning unit 113' of the learning module 110' includes first, second, and third image quality convolutional neural networks 201, 202, and 203. And the learning process using each pixel of each third learning image and each pixel of each third correct answer image is repeated to derive a plurality of quality selection parameters.

The image quality selection learning unit 113 ′ is based on the first, second, and third image quality convolutional neural networks and image quality selection convolutional neural networks corresponding to a plurality of preset image quality selection parameters in any one learning process. A learning output pixel, which is a result of performing SR processing on a learning input block including any one of the target pixels, is derived, and a plurality of pixels are obtained based on the difference between the correct answer pixel corresponding to the target pixel among the third correct answer image and the learning output pixel. Update the quality selection parameter.

And, in the learning process of the threshold number of times, the quality selection learning unit 113 ′ stops the learning process and stops the learning process, when a learning output pixel whose difference from the correct answer pixel is within the threshold within the same range is derived. A plurality of image quality selection parameters corresponding to the lution neural network may be output.

Thereafter, the learning data generation unit 114 ′ of the learning module 110 ′ collects a plurality of first upscaling parameters, a plurality of second upscaling parameters, a plurality of third upscaling parameters, and a plurality of image quality selection parameters. You can generate training data. (S14')

As described above, according to another embodiment of the present invention, the SR learning model 200 ′ further including the third image quality convolutional neural network 203 corresponding to the third image quality is the reactivity of the input resolution to the third image quality. Since is further improved, deterioration of the quality of the output image restored by SR processing of the input image of the third quality can be further prevented.

The present invention described above is not limited to the above-described embodiments and the accompanying drawings, and various substitutions, modifications and changes are possible within the scope of the technical spirit of the present invention. It will be obvious to those who have the knowledge of.

100: image processing system
200: SR circuit model
201, 202, 203: first, second, and third picture quality convolutional neural networks
210: quality selection convolutional neural network
300: display device
310: display panel
340: timing controller

Claims (20)

A first quality convolution neural network for converting an image of a first quality corresponding to an input resolution lower than a predetermined target resolution to the target resolution, and a first quality convolution neural network corresponding to the input resolution and lower than the first quality. 2 Picture quality selection for selecting a combination of a second quality convolutional neural network for converting an image of quality into the target resolution, and the first and second quality convolutional neural networks having the highest responsiveness to each quality of the input resolution A learning module for generating training data corresponding to a super resolution (SR) learning model including a convolutional neural network; And
An image for performing SR processing of constructing the SR (Super Resolution) learning model based on the training data, and converting an input image having an arbitrary image quality of the input resolution into the target resolution based on the constructed SR learning model An image processing system including a processing module.
The method of claim 1,
The learning module
The plurality of first training images having the first quality and the plurality of first training images corresponding to the first quality convolutional neural network through learning using a plurality of first correct answer images having the target resolution and corresponding to the plurality of first training images A first image quality learning unit for deriving a first upscaling parameter of;
The plurality of second learning images having the second image quality and the plurality of corresponding second image quality convolutional neural networks through learning using a plurality of second correct answer images corresponding to the plurality of second learning images and having the target resolution A second image quality learning unit that derives a second upscaling parameter of;
Using the first and second quality convolutional neural networks and a plurality of third training images each having an arbitrary quality of the input resolution, and a plurality of third correct answer images corresponding to the plurality of third training images and having the target resolution A picture quality selection learning unit for deriving a plurality of picture quality selection parameters corresponding to the picture quality selection convolutional neural network through learning; And
An image processing system comprising a learning data generator configured to generate the training data based on the plurality of first upscaling parameters, the plurality of second upscaling parameters, and the plurality of image quality selection parameters.
The method of claim 2,
The first quality learning unit derives the plurality of first upscaling parameters by repeating a learning process using each pixel of each of the first training images and each pixel of each of the first correct answer images,
The second image quality learning unit derives the plurality of second upscaling parameters by repeating a learning process using each pixel of each second learning image and each pixel of each second correct answer image,
The quality selection learning unit derives the plurality of quality selection parameters by repeating a learning process using the first and second quality convolutional neural networks, each pixel of each of the third training images, and each pixel of each of the third correct answer images. Image processing system.
The method of claim 3,
The image processing module converts the input image into the target resolution in pixel units based on the SR learning model.
The method of claim 3,
The first picture quality learning unit
In any one learning process for deriving the plurality of first upscaling parameters, any one of the first training images based on the first quality convolutional neural network corresponding to the preset plurality of first upscaling parameters A learning output pixel corresponding to a learning input block including a target pixel of is derived, and the plurality of first upscaling is performed based on a difference between a correct answer pixel corresponding to the target pixel and the learning output pixel of the first correct answer image. Image processing system to update parameters.
The method of claim 3,
The second image quality learning unit
In any one learning process for deriving the plurality of second upscaling parameters, any one of the second training images based on the second quality convolutional neural network implemented with the preset second upscaling parameters A learning output pixel corresponding to a learning input block including a target pixel of is derived, and the plurality of second upscaling is performed based on a difference between the correct answer pixel corresponding to the target pixel and the learning output pixel of the second correct answer image. Image processing system to update parameters.
The method of claim 3,
The quality selection learning unit
In any one learning process for implementing the plurality of image quality selection parameters, the first image quality convolutional neural network, the second image quality convolutional neural network, and the image quality selection convolutional neural network implemented with the plurality of preset image quality selection parameters On the basis of, a learning output pixel corresponding to a learning input block including any one of the target pixels of the third learning image is derived, and between the correct answer pixel corresponding to the target pixel of the third correct answer image and the learning output pixel An image processing system for updating the plurality of quality selection parameters based on a difference.
The method of claim 2,
The training data is
The first and second picture quality convolutional neural networks,
A third quality convolutional neural network for converting an image of a third quality corresponding to the input resolution and lower than the second quality to the target resolution,
An image processing system corresponding to an image quality selection convolutional neural network for selecting a combination of the first, second, and third image quality convolutional neural networks having the highest responsiveness to each image quality of the input resolution.
The method of claim 8,
The learning module provides the third quality convolutional neural network through learning using a plurality of fourth training images having the third image quality and a plurality of fourth correct answer images corresponding to the plurality of fourth training images and having the target resolution. Further comprising a third image quality learning unit for deriving a plurality of third upscaling parameters corresponding to,
The third image quality learning unit derives the plurality of third upscaling parameters by repeating a learning process using each pixel of each of the third learning images and each pixel of each of the third correct answer images,
In any one learning process for deriving the plurality of third upscaling parameters, the third image quality learning unit is based on the third image quality convolutional neural network corresponding to the plurality of third upscaling parameters. 3 A learning output pixel corresponding to a learning input block including any one of the target pixels is derived, and based on the difference between the correct answer pixel corresponding to the target pixel of the third correct answer image and the learning output pixel, the Update a plurality of third upscaling parameters,
The training data generation unit further generates the training data based on the plurality of third upscaling parameters.
The method of claim 1,
The image processing module
A memory unit for holding the learning data provided from the learning module; And
An image processing system comprising an SR engine unit for constructing the SR learning model based on the training data held in the memory unit and performing the SR processing on the input image based on the SR learning model.
The method of claim 1,
The image processing module is an image processing system implemented as a function of a timing controller of a display device.
A first quality convolution neural network for converting an image of a first quality corresponding to an input resolution lower than a predetermined target resolution to the target resolution, and a first quality convolution neural network corresponding to the input resolution and lower than the first quality. 2 Picture quality selection for selecting a combination of a second quality convolutional neural network for converting an image of quality into the target resolution, and the first and second quality convolutional neural networks having the highest responsiveness to each quality of the input resolution Generating training data corresponding to a super resolution (SR) training model including a convolutional neural network;
Building the SR (Super Resolution) learning model based on the training data; And
And performing SR processing of converting an input image having an arbitrary image quality of the input resolution into the target resolution based on the constructed SR learning model.
The method of claim 12,
The step of generating the training data
The plurality of first training images having the first quality and the plurality of first training images corresponding to the first quality convolutional neural network through learning using a plurality of first correct answer images having the target resolution and corresponding to the plurality of first training images Deriving a first upscaling parameter of;
The plurality of second learning images having the second image quality and the plurality of corresponding second image quality convolutional neural networks through learning using a plurality of second correct answer images corresponding to the plurality of second learning images and having the target resolution Deriving a second upscaling parameter of;
Using the first and second quality convolutional neural networks and a plurality of third training images each having an arbitrary quality of the input resolution, and a plurality of third correct answer images corresponding to the plurality of third training images and having the target resolution Deriving a plurality of picture quality selection parameters corresponding to the picture quality selection convolutional neural network through learning; And
And generating the training data based on the plurality of first upscaling parameters, the plurality of second upscaling parameters, and the plurality of image quality selection parameters.
The method of claim 13,
In the step of deriving the plurality of first upscaling parameters, the plurality of first upscaling parameters are derived by repeating a learning process using each pixel of each of the first learning images and each pixel of each of the first correct answer images. and,
In the step of deriving the plurality of first upscaling parameters, a learning process using each pixel of each second learning image and each pixel of each second correct answer image is repeated to derive the plurality of second upscaling parameters And
In the step of deriving the plurality of quality selection parameters, the learning process using the first and second quality convolutional neural networks, each pixel of each of the third training images, and each pixel of each of the third correct answer images is repeated, An image processing method for deriving a plurality of quality selection parameters.
The method of claim 14,
In the step of performing the SR processing, a training data image processing method for converting the input image into the target resolution in pixel units based on the SR learning model.
The method of claim 14,
In any one of the steps of deriving the plurality of first upscaling parameters, any one of the first training images based on the first quality convolutional neural network corresponding to the preset plurality of first upscaling parameters A learning output pixel corresponding to a learning input block including one target pixel is derived, and the plurality of first ups are performed based on a difference between a correct answer pixel corresponding to the target pixel and the learning output pixel among the first correct answer image. Image processing method for updating scaling parameters.
The method of claim 14,
In any one of the steps of deriving the plurality of second upscaling parameters, any one of the second training images based on the second image quality convolutional neural network implemented with the plurality of second upscaling parameters A learning output pixel corresponding to a learning input block including one target pixel is derived, and the plurality of second uploads are performed based on a difference between the correct answer pixel corresponding to the target pixel and the learning output pixel among the second correct answer image. Image processing method for updating scaling parameters.
The method of claim 14,
In any one of the steps of deriving the image quality selection parameter, the third image quality selection convolutional neural network implemented with the first and second image quality convolutional neural networks and the plurality of preset image quality selection parameters. A learning output pixel corresponding to a learning input block including any one of the target pixels of the training image is derived, and the plurality of the plurality of pixels are derived based on the difference between the correct answer pixel corresponding to the target pixel of the third correct answer image and the learning output pixel. Image processing method to update the quality selection parameter of the.
The method of claim 14,
In the step of generating the training data, the training data includes the first and second image quality convolutional neural networks, and a third image for converting an image of a third image quality corresponding to the input resolution and lower than the second image quality to the target resolution. 3 An image processing method corresponding to an image quality selection convolutional neural network for selecting a combination of an image quality convolutional neural network and the first, second, and third image quality convolutional neural networks having the highest responsiveness to each image quality of the input resolution.
The method of claim 19,
The step of generating the learning data includes the third image through learning using a plurality of fourth learning images having the third quality and a plurality of fourth correct answer images corresponding to the plurality of fourth learning images and having the target resolution. Further comprising the step of deriving a plurality of third upscaling parameters corresponding to the image quality convolutional neural network,
In the step of deriving the plurality of third upscaling parameters, the plurality of third upscaling parameters are derived by repeating the learning process using each pixel of each of the fourth learning images and each pixel of each of the fourth correct answer images. And
In any one learning process for deriving the plurality of third upscaling parameters, any one of the fourth training images based on the third quality convolutional neural network corresponding to the preset plurality of third upscaling parameters A learning output pixel corresponding to a learning input block including a target pixel of is derived, and the plurality of third upscaling is performed based on a difference between the correct answer pixel corresponding to the target pixel and the learning output pixel of the fourth correct answer image. Update the parameters,
In the step of deriving the plurality of image quality selection parameters, the plurality of image quality selection parameters are derived through further learning using the third image quality convolutional neural network,
In the step of generating the training data, the image processing method of generating the training data further based on a plurality of third upscaling parameters.

Images