KR20220144744A - Image processing circuit and system on chip including therof and method of enhancing image resolution - Google Patents

Abstract

Disclosed are an image processing circuit, a system on a chip including the same, and a method for improving image quality. The image processing circuit according to an aspect of the technical idea of the present disclosure comprises: a tuning circuit for receiving a segmentation map including class inference information for each pixel of an image and a confidence map including the confidence level of the class inference information for each pixel, determining a class for each pixel of the image, a correction effect for each pixel of the image, and a correction value indicating the strength of the correction effect based on the segmentation map and the confidence map, and generating a correction map based on the class for each pixel and the correction value for each pixel; and at least one correction circuit for generating an improved image by applying the correction effect depending on the correction value to each pixel of the image based on the correction map. Therefore, provided are an image processing circuit, a system on a chip including the same, and a method for improving image quality, wherein correction can be performed for each pixel of an image.

Description

IMAGE PROCESSING CIRCUIT AND SYSTEM ON CHIP INCLUDING THEROF AND METHOD OF EHANCING IMAGE RESOLUTION

The technical idea of the present disclosure relates to image processing, and more particularly, to an image processing circuit, a system-on-chip including the same, and an image quality improvement method.

Images can be acquired through various devices. For example, if you use the built-in camera of a smartphone, you can take an image and correct the captured image.

Algorithms for correcting the image may vary. As a representative example, a method of correcting an overall tone and/or color of an image may be used to determine one scene that may represent the image and to emphasize the scene.

However, there are cases in which subjects having various characteristics are included in one image, and in this case, each subject must be corrected to suit each characteristic. When uniform correction is performed on the entire image without distinguishing the characteristics of subjects included in the image, it is difficult to obtain a uniformly corrected image as a whole.

SUMMARY An object of the present disclosure is to provide an image processing circuit for performing correction in units of pixels of an image, a system-on-chip including the same, and a method for improving image quality.

In order to achieve the above object, an image processing circuit according to an aspect of the technical idea of the present disclosure, an image processing circuit according to an aspect of the technical idea of the present disclosure, includes segmentation including class inference information for each pixel of an image receiving a map and a confidence map including the reliability of the class inference information for each pixel, and based on the segmentation map and the confidence map, a class for each pixel of the image, a correction effect for each pixel of the image, and intensity of the correction effect A tuning circuit that determines a correction value representing and at least one correction circuit for generating the enhancement image.

In order to achieve the above object, the method of improving the image quality according to one aspect of the technical idea of the present disclosure includes the steps of receiving the image, and each pixel of the image belongs to using a learned neural network model. generating first class inference information by inferring a class, calculating a first reliability for the first class inference information, based on a table in which correction values are determined according to the class and reliability, to be applied to each pixel of the image and determining a first correction effect and a first correction value, and generating an enhanced image by applying the first correction effect to each pixel by the first correction value.

In order to achieve the above object, a system-on-a-chip for generating an enhanced image by correcting an image according to an aspect of the technical idea of the present disclosure is a first class for each pixel of the image using a learned neural network model. a first circuit for generating inference information and a first confidence level for the first class inference information, and based on the first class inference information and the first confidence level, determine a correction value for each pixel of the image; and a second circuit for generating the enhanced image by applying a correction effect corresponding to the correction value to each pixel of the image.

In order to achieve the above object, a system-on-chip for correcting an image according to an aspect of the technical idea of the present disclosure receives the image, and uses the learned neural network model to classify each pixel of the image By determining a correction effect to be applied to each pixel of the image based on a segmentation circuit generating a segmentation map including inference information and a confidence map including confidence in the class inference information, and the segmentation map and the confidence map and an image processing circuit that generates a correction map and applies the correction effect to the image based on the correction map.

According to the image processing circuit of the technical idea of the present disclosure, a system on a chip including the same, and an image quality improvement method, by enabling fine correction by lowering the correction unit to pixels, an image of improved image quality can be obtained, and images of different classes can be obtained. Even in an image in which pixels are intricately arranged, accurate correction corresponding to each pixel can be performed, thereby improving user satisfaction.

1 is a block diagram illustrating an image processing system according to an exemplary embodiment of the present disclosure.
2 is a block diagram illustrating a neural network model according to an exemplary embodiment of the present disclosure.
3 is a block diagram illustrating an upscaling circuit according to an exemplary embodiment of the present disclosure.
4 is an exemplary diagram illustrating a segmentation map and a confidence map according to an exemplary embodiment of the present disclosure.
5 is an exemplary diagram illustrating a segmentation map according to an exemplary embodiment of the present disclosure.
6 is an exemplary diagram illustrating an operation of a tuning circuit according to an exemplary embodiment of the present disclosure.
7 is an exemplary diagram illustrating class information according to an exemplary embodiment of the present disclosure.
8 is an exemplary diagram illustrating a part of a setting value table according to an exemplary embodiment of the present disclosure.
9 is a flowchart illustrating an operation of an image processing system according to an exemplary embodiment of the present disclosure.
10 is a block diagram illustrating an image processing system according to an exemplary embodiment of the present disclosure.
11 is a flowchart illustrating an operation of an image processing system according to an exemplary embodiment of the present disclosure.
12 is a block diagram illustrating an image processing system according to an exemplary embodiment of the present disclosure.
13 is a flowchart illustrating an operation of an image processing system according to an exemplary embodiment of the present disclosure.
14 is a block diagram illustrating a system according to an exemplary embodiment of the present disclosure.
15 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating an image processing system according to an exemplary embodiment of the present disclosure.

Referring to FIG. 1 , the image processing system 10 may include a plurality of circuits for outputting an enhanced image by correcting an image image_h. The image image_h may be generated by an image signal processor (ISP). For example, the image image_h may correspond to a still image or one frame of a moving picture including a plurality of frames.

In one embodiment, the image processing system 10 may include an image processing circuit 100 , a segmentation circuit 200 , and an upscaling circuit 250 . Each of the image processing circuit 100 , the segmentation circuit 200 , and the upscaling circuit 250 may be implemented as hardware, software (or firmware), or a combination of hardware and software for performing the spirit of the present disclosure. Although not shown, the image processing system 10 may further include an encoder for encrypting the enhancement image.

The segmentation circuit 200 may receive an image and perform a segmentation operation of dividing the image into a plurality of regions using the learned neural network model 210 . In more detail, the segmentation circuit 200 may infer a class corresponding to each of the plurality of pixels included in the image. As an embodiment, the segmentation circuit 200 may form a segmentation map by inferring a class corresponding to each pixel. As an embodiment, pixels with the same class inference information may form a region.

Also, the segmentation circuit 200 may calculate the reliability of the inferred class for each pixel. As an embodiment, the reliability may be formed in the form of a confidence map. As an embodiment, the size of the segmentation map and the size of the confidence map may each be the same as the size of the image.

As an embodiment, the segmentation circuit 200 may receive a low-quality image image_l in which an image image_h to be corrected is reduced. The low-resolution image image_l may have a smaller size than the image image_h. The segmentation circuit 200 may generate a low-quality segmentation map and a low-quality confidence map for the low-quality image image_l by using the neural network model 210 . In this case, the size of the low-quality segmentation map and the size of the low-quality confidence map may be the same as the size of the low-quality image image_l, respectively. Hereinafter, the neural network model 210 will be described in detail with reference to FIG. 2 .

The upscaling circuit 250 receives the low-quality segmentation map and the low-quality confidence map from the segmentation circuit 200, and interpolates and enlarges each to correspond to the size of the image (image_h), thereby having the same size as the segmentation map and the confidence map. can create

The image processing circuit 100 may receive the image image_h, perform a series of processes for improving the image quality of the image image_h or correct the image image_h, and generate the enhanced image as a result. The image processing circuit 100 may receive an image image_h, a segmentation map for the image image_h, and a confidence map.

The image processing circuit 100 may include a tuning circuit 140 and at least one correction circuit 101 .

The tuning circuit 140 may determine a correction effect to be applied to each pixel of the image image_h and the strength of the correction effect based on the segmentation map and the confidence map. Hereinafter, for convenience of description, the intensity of the correction effect may be referred to as a correction value. In order to apply at least one correction effect to each pixel, the tuning circuit 140 may generate a correction map corresponding to each correction effect and provide the correction map to the at least one correction circuit 101 .

The at least one correction circuit 101 may receive the correction map from the tuning circuit 140 and apply a correction effect to each pixel of the image image_h. As an embodiment, the at least one correction circuit 101 may apply a correction effect to a plurality of pixels to which the same correction effect is applied.

As an embodiment, the at least one correction circuit 101 may include a denoise circuit 110 , a color correction circuit 120 , and a sharpen circuit 130 . The denoise circuit 110 reduces noise of at least one pixel, the color correction circuit 120 adjusts a color value of at least one pixel, and the sharpen circuit 130 increases the sharpness of at least one pixel. can The at least one correction circuit 101 may further include a circuit providing various kinds of correction effects, and each of the at least one correction circuit 101 is implemented by hardware, software (or firmware), or a combination of hardware and software. can be

As an embodiment, the image processing circuit 100 may perform correction in units of regions including pixels classified into the same class. For example, in the image processing circuit 100 , the denoise circuit 110 reduces noise in the first region in order to apply the first correction effect to the first region including the first pixel classified into the first class. , at least one of the color correction circuit 120 for adjusting the color value of the first region and the sharpening circuit 130 for increasing the sharpness of the first region may be used. In this case, the image processing circuit 100 may use the confidence map to adjust the intensity of the correction effect of the first pixel or the first region including the first pixel.

The image processing circuit 100 according to an embodiment of the present disclosure may determine and apply a correction effect to be applied to each of a plurality of pixels included in the image image_h. That is, by reducing the unit of correction to pixels to enable fine correction, an image with improved image quality can be obtained.

In addition, by performing segmentation and correction in units of pixels on an image (image_h) in which pixels of different classes are complexly arranged, accurate correction corresponding to each pixel can be performed compared to performing correction in units of regions. can enhance

2 is a block diagram illustrating a neural network model according to an exemplary embodiment of the present disclosure.

Referring to FIG. 2 , the segmentation circuit 200 may receive a low-quality image image_l generated by reducing an image image_h from the image signal processor ISP. The segmentation circuit 200 may input a low-quality image image_l to the neural network model 210 and obtain a low-quality segmentation map map_seg_l and a low-quality confidence map map_conf_l as outputs. The segmentation circuit 200 may provide the low-quality segmentation map map_seg_l and the low-quality confidence map map_conf_l to the upscaling circuit 250 .

The low-resolution segmentation map map_seg_l may include class inference information for each pixel of the low-resolution image image_l. The low quality confidence map map_conf_l may include reliability of class inference information for each pixel of the low quality image image_l. For example, the second pixel of the low-quality segmentation map map_seg_l and the third pixel of the low-quality confidence map map_conf_l may correspond to the first pixel of the low-quality image image_l, respectively. In this case, the second pixel may indicate class inference information for the first pixel, and the third pixel may indicate probability and/or reliability of the corresponding class inference information.

The segmentation circuit 200 may include a neural network model 210 that has been trained. The neural network model 210 may learn a relationship between a plurality of classes classified to have different correction effects and a pixel. In this case, the plurality of classes may have different weights for at least one of the denoise effect, the color correction effect, and the sharpening effect.

The neural network model 210 may be trained offline. As an embodiment, the neural network model 210 may be generated by being trained in a learning device, for example, a server that trains a neural network based on a large amount of learning data. As an embodiment, the neural network model 210 may use an arbitrary pixel and a class labeled corresponding to the arbitrary pixel as training data. For example, a training image and a class labeled for each pixel of the training image can be used as training data.

Hereinafter, in the present specification, parameters (eg, network topology, bias, weight, etc.) of the neural network model 210 will be described on the assumption that they have already been determined through learning. However, the present invention is not limited thereto.

For example, the neural network model 210 is a Convolution Neural Network (CNN), a Region with Convolution Neural Network (R-CNN), a Region Proposal Network (RPN), a Recurrent Neural Network (RNN), and a Stacking-based (S-DNN). Deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restricted Boltzmann Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network It may include at least one of various types of neural network models.

3 is a block diagram illustrating an upscaling circuit according to an exemplary embodiment of the present disclosure.

Referring to FIG. 3 , the upscaling circuit 250 may generate a segmentation map map_seg_h and a confidence map map_conf_h by enlarging and/or interpolating the low quality segmentation map map_seg_l and the low quality confidence map map_conf_l. The size of each of the segmentation map map_seg_h and the confidence map map_conf_h generated by the upscaling circuit 250 may be the same as the size of the image.

The upscaling circuit 250 may use various algorithms to perform scaling and/or interpolation. As an example, the upscaling circuit 250 may convolv a low-resolution or low-resolution image into an interpolation kernel and resample it in a new grid. As an example, the upscaling circuit 250 may use a linear interpolation filter, and may apply a joint bilateral filter to keep the edges sharp. However, it is not necessarily limited thereto.

The upscaling circuit 250 may provide the segmentation map map_seg_h and the confidence map map_conf_h to the image processing circuit (eg, 100 of FIG. 1 ).

4 is an exemplary diagram illustrating a segmentation map and a confidence map according to an exemplary embodiment of the present disclosure.

1 and 4 together, the segmentation map map_seg_h may include a plurality of pixels, and each pixel px may include class inference information info_class. That is, each pixel px of the segmentation map map_seg_h may include class inference information info_class of a pixel of the corresponding image image_h. As an embodiment, the class inference information (info_class) may include n (eg, n is 3) bits, which is a natural number, but is not limited thereto. For example, when the type of class inference information (info_class) output from the neural network model increases, the number of bits representing the class inference information (info_class) may increase.

The confidence map map_conf_h may include a plurality of pixels, and each pixel px may include a reliability conf. Each pixel px of the confidence map map_conf_h may correspond to a pixel of the segmentation map map_seg_h and a pixel of the image image_h, respectively. Each pixel px of the confidence map map_conf_h may include the confidence level conf of the class inference information info_class of the pixel of the corresponding image image_h. As an embodiment, the reliability conf may include m (eg, m is 5) bits, which is a natural number greater than n, but is not limited thereto.

5 is an exemplary diagram illustrating a segmentation map according to an exemplary embodiment of the present disclosure.

Referring to FIG. 5 , the position of each pixel in the segmentation map map_seg_h may correspond to the position of the pixel in the image image_h, and the value of each pixel in the segmentation map map_seg_h is inferred from the pixel of the image image_h. class can be represented.

For example, in the image (image_h), the part corresponding to the branch is the first class (class1), the part corresponding to the sky is the second class (class2), the part corresponding to the human hair is the third class (class3), The part corresponding to the human face may be inferred as a fourth class (class4), and the part corresponding to the human skin may be inferred as a fifth class (class5).

According to an embodiment of the present disclosure, pixels may be segmented in consideration of a correction effect to be applied to each pixel of the image image_h. That is, pixels to which different correction effects are to be applied may be segmented into different classes.

In addition, according to an embodiment of the present disclosure, since segmentation is performed in units of pixels, it is possible to increase the accuracy of segmentation even in a detailed and complex image image_h such as hair and tree branches.

6 is an exemplary diagram illustrating an operation of a tuning circuit according to an exemplary embodiment of the present disclosure.

Referring to FIG. 6 , the tuning circuit 140 may receive an image image_h from the outside, and may receive a segmentation map map_seg_h and a confidence map map_conf_h from the upscaling circuit 250 in FIG. 1 . . As an example, the image image_h may be generated inside the image processing circuit (eg, 100 in FIG. 1 ) through the image sensor.

The tuning circuit 140 may include a set value table 141 , a selection circuit 142 , and a mixing circuit. The selection circuit 142 and the mixing circuit 143 may be implemented as hardware, software (or firmware), or a combination of hardware and software, respectively.

The set value table 141 may include class inference information meant by each pixel of the segmentation map map_seg_h and information on correction effects applied according to the reliability signified by each pixel of the confidence map map_conf_h. For example, it may include a correction value indicating the strength of the correction effect.

As an embodiment, a pixel of the segmentation map map_seg_h corresponding to an arbitrary pixel of the image image_h may represent the first class, and a pixel of the confidence map map_conf_h may represent the first value. In this case, when the first class has the first value and is inferred, the set value table 141 may include a correction effect to be applied to the pixel and a correction value to be applied. For example, the set value table 141 applies the first correction effect as much as the first correction value to the random pixel when the random pixel is determined as the first class having the first reliability, and applies the second correction effect to the random pixel. Information for applying as much as the second correction value may be included.

In the present specification, the set value table 141 is described as being predetermined to correspond to at least one correction circuit, but is not limited thereto, and information included in the set value table 141 may be changed.

The selection circuit 142 may determine the class of each pixel of the image image_h based on the segmentation map map_seg_h and the confidence map map_conf_h and a preset threshold value. As an example, the selection circuit 142 may determine the class of each pixel of the image image_h based on class inference information and reliability for each pixel. In this case, the selection circuit 142 may include class information and a preset threshold value. The class information may include types of classes from which each pixel may be determined.

A case in which an arbitrary pixel is inferred as the first class according to the segmentation map map_seg_h and the inference reliability of the arbitrary pixel is the first value according to the confidence map map_conf_h will be described as an example. The threshold required to be determined as the first class may be the first threshold. The selection circuit 142 may compare a first threshold value for determining the arbitrary pixel as the first class according to the inference result with a first value. The selection circuit 142 may determine the class of the arbitrary pixel as the first class as the first value exceeds (or exceeds) the first threshold.

A class of each of the plurality of pixels included in the image image_h may be determined by the selection circuit 142 . Meanwhile, although the class of any two pixels is the same, a correction effect to be applied may be different depending on the reliability of each pixel. In this case, the mixing circuit 143 may determine the strength of the correction effect to be applied to each of the plurality of pixels, that is, the correction value in consideration of reliability.

The mixing circuit 143 may generate the correction maps tmap1, tmap2, and tmap3 corresponding to each correction effect by considering the confidence map map_conf_h based on the class of each pixel determined by the selection circuit 142. . The size of the correction maps tmap1, tmap2, and tmap3 may be the same as the size of the image image_h. Each pixel of the correction maps tmap1, tmap2, and tmap3 may represent a correction value to be applied to each pixel in the image image_h.

According to an embodiment, the mixing circuit 143 generates a first correction map tmap1 including correction values for the denoise effect, and a second correction map tmap2 including correction values for the color correction effect. , and a third correction map tmap3 including correction values for the sharpening effect may be generated. The mixing circuit 143 may provide the first to third correction maps tmap1 , tmap2 , and tmap3 to the denoise circuit 110 , the color correction circuit 120 , and the sharpen circuit 130 , respectively.

Although it is illustrated in FIG. 6 that three correction maps tmap1, tmap2, and tmap3 are generated, the present invention is not limited thereto, and the mixing circuit 143 includes at least one correction circuit 110, 120, 130. can create

7 is an exemplary diagram illustrating class information according to an exemplary embodiment of the present disclosure.

Referring to FIG. 7 , class information info_class may be included in a tuning circuit (eg, 140 of FIG. 6 ). The class information info_class may include a plurality of classes into which pixels of an image may be classified.

The class and/or subclass may be generated based on the correction effect. That is, pixels to which the same correction effect is applied may be classified into one class, and pixels to which different correction effects will be applied may be classified into different classes. Without being limited thereto, the class and/or sub-class may be generated based on characteristics of a pixel, for example, color information.

According to an embodiment, one class may include at least one subclass. As an embodiment, the class information (info_class) may include first to seventh classes (class1 to class7), and the fourth class (class4) may include first to third subclasses (subclass1 to subclass3). can For example, the first to seventh classes (class1 to class7) may correspond to a face class, a skin class, a sky class, a detail class, an eye class, an eyebrow class, and a hair class, respectively. In this case, the detail class may include a grass subclass, a sand subclass, and a twig subclass. Meanwhile, the type and number of classes and subclasses are not limited thereto and may be changed.

8 is an exemplary diagram illustrating a part of a setting value table according to an exemplary embodiment of the present disclosure.

6 and 8 together, the set value table 141 may include correction values according to reliability for a plurality of correction effects. Hereinafter, one class will be described as an example, but may be similarly applied to other classes.

For example, the selection circuit 142 of the tuning circuit 140 may determine an arbitrary pixel in the image image_h as the first class (class1). In this case, the mixing circuit 143 may determine a correction value according to the reliability of the arbitrary pixel by referring to the confidence map map_conf_h for the image image_h.

For example, when the reliability of the first class (class1) is 0.25, the mixing circuit 143 may determine that the random pixel is given a denoise effect by d2, a color correction effect by c2, and a sharpening effect by s2. Accordingly, in the first correction map tmap1 for the denoise effect, the pixel value corresponding to the arbitrary pixel is d2, and in the second correction map tmap2 for the color correction effect, the pixel corresponding to the arbitrary pixel It is possible to determine the value as c2, and the pixel value corresponding to the arbitrary pixel in the third correction map tmap3 for the sharpening effect as s2. That is, the mixing circuit 143 generates a first correction map tmap1 including a d2 value, a second correction map tmap2 including a c2 value, and a third correction map (tmap2) including a s2 value. tmap3) can be created.

According to the exemplary embodiment of the present disclosure, since the correction effect and the correction value are determined in units of pixels, even in the image image_h including a complex shape, correction according to the characteristics of each pixel may be performed. Accordingly, it may be helpful to improve the image quality of the image (image_h) having a complex shape.

9 is a flowchart illustrating an operation of an image processing system according to an exemplary embodiment of the present disclosure.

1, 6, and 9 together, an operation of receiving an image image_h may be performed (S110). For example, the image processing system 10 may acquire an image image_h from an image sensor.

An operation of generating a low-quality image image_l based on the received image image_h may be performed (S120).

Using the learned neural network model 210, a low-quality segmentation map (map_seg_l) and a low-quality confidence map (map_conf_l) for the low-quality image image_l may be generated ( S130 ). As an embodiment, the neural network model 210 may be trained using the training image and the correct answer class labeled in each pixel of the training image as training data. In this case, the correct answer class may correspond to a correction effect to be applied to each pixel of the training image. As an embodiment, a class to which each pixel of the low-resolution image image_l belongs may be inferred by the neural network model 210 , and reliability of the inferred class may be calculated for each pixel.

By enlarging the low-quality segmentation map (map_seg_l) and the low-resolution confidence map (map_conf_l), a segmentation map (map_seg_h) and a confidence map (map_conf_h) for the image (image_h) may be generated ( S140 ).

Based on the segmentation map map_seg_h and the confidence map map_conf_h, the correction effect to be applied to each pixel of the image image_h and the intensity of the correction effect, ie, a correction value, may be determined ( S150 ). As an embodiment, the correction maps tmap1 to tmap3 may be generated for each correction effect. The correction maps tmap1 to tmap3 may include correction values and may have the same size as the image.

An enhanced image may be generated by applying a correction effect corresponding to the correction value to each pixel ( S160 ). As an embodiment, the correction circuits 110 , 120 , and 130 may apply a correction effect to each pixel of the image image_h based on the correction maps tmap1 to tmap3 . In this case, the enhanced image may be an improved image quality of the image image_h.

Thereafter, an operation of providing the enhancement image to the encoder and compressing it in various formats may be further performed.

10 is a block diagram illustrating an image processing system according to an exemplary embodiment of the present disclosure. Since the image processing system 20 of FIG. 10 is similar to the image processing system 10 of FIG. 1 , a redundant description will be omitted.

Unlike the image processing system 10 of FIG. 1 , in the image processing system 20 of FIG. 10 , the upscaling circuit 250 ( FIG. 1 ) may be omitted. Accordingly, the segmentation circuit 400 may receive the image image_h and generate a segmentation map map_seg_h and a confidence map map_conf_h for the image image_h using the learned neural network model 410 .

As an embodiment, the image processing circuit 300 may be referred to as a first circuit, and the segmentation circuit 400 may be referred to as a second circuit. Accordingly, the image processing system 10 of FIG. 1 may further include an upscaling circuit (250 of FIG. 1 , for example, a third circuit) compared to the image processing system 20 of FIG. 10 .

As an embodiment, the second circuit may generate the first class inference information for each pixel of the image image_h and the first confidence level for the first class inference information using the learned neural network model 410 . . The first circuit determines a correction value for each pixel of the image image_h based on the first class inference information and the first reliability information, and applies a correction effect corresponding to the correction value to each pixel of the image image_h. Enhanced images can be created. Here, as in the image processing system 10 of FIG. 1 , when a third circuit is further included, the second circuit receives the low-resolution image, and a second class for each pixel of the low-resolution image with the neural network model 410 . A second confidence level may be generated for the inference information and the second class inference information. The third circuit may generate the first class inference information and the first reliability based on the second class inference information and the second reliability.

The segmentation circuit 400 may provide the generated segmentation map map_seg_h and the confidence map map_conf_h to the tuning circuit 340 of the image processing circuit 300 . The image processing circuit 300 may perform a pixel-based correction operation of the image image_h based on the received segmentation map map_seg_h and the confidence map map_conf_h.

11 is a flowchart illustrating an operation of an image processing system according to an exemplary embodiment of the present disclosure.

10 and 11 together, an operation of receiving an image image_h may be performed (S210). For example, the image processing system 20 may acquire an image image_h from an image sensor. The image processing circuit 300 and the segmentation circuit 400 may receive an image image_h.

A segmentation map map_seg_h and a confidence map map_conf_h for the image image_h may be generated using the learned neural network model 410 ( S220 ). As an embodiment, the segmentation circuit 400 may infer a class to which each pixel of the image image_h belongs by using the neural network model 410 , and may calculate the reliability of the inferred class for each pixel.

Based on the segmentation map map_seg_h and the confidence map map_conf_h, a correction effect to be applied to each pixel of the image image_h and the intensity of the correction effect, that is, a correction value may be determined ( S230 ). An enhanced image may be generated by applying a correction effect corresponding to the correction value to each pixel ( S240 ).

Thereafter, the enhancement image may be provided to the encoder, and an operation of being encrypted in various formats by the encoder may be further performed.

12 is a block diagram illustrating an image processing system according to an exemplary embodiment of the present disclosure. Since the image processing system 30 of FIG. 12 is similar to the image processing system 10 of FIG. 1 , a redundant description will be omitted.

Unlike the image processing system 10 of FIG. 1 , the image processing system 30 of FIG. 12 may further include a scene detection circuit 710 . The scene detection circuit 710 according to an embodiment may determine a scene representing the image image_h. For example, the scene of the image can be determined as food, landscape, person, etc. The scene detection circuit 710 may use various algorithms to determine the scene of the image. For example, a feature point extraction algorithm, a face recognition algorithm, a color-based segmentation algorithm, etc. may be used.

The scene detection circuit 710 may generate a correction effect and a correction value according to the determined scene. Information including the correction effect and correction value generated by the scene detection circuit 710 may be referred to as sub-correction information. The scene detection circuit 710 may provide sub-correction information to the image processing circuit 720 . The image processing circuit 720 may apply a correction effect to the image image_h based on the sub correction information. Meanwhile, the scene detection circuit 710 may be implemented as hardware, software (or firmware), or a combination of hardware and software for carrying out the spirit of the present disclosure. In this case, the scene detection circuit 710 may be included in the image signal processor (ISP).

1 , a segmentation map map_seg_h and a confidence map map_conf_h may be generated through the segmentation circuit 730 and the upscaling circuit 740 . The confidence map map_conf_h may include a probability and/or reliability for each pixel of the segmentation map map_seg_h.

The image processing circuit 720 may receive sub correction information from the scene detection circuit 710 , and may receive a segmentation map map_seg_h and a confidence map map_conf_h from the upscaling circuit 740 .

The image processing circuit 720 may correct the image image_h based on the sub-correction information and correct the image image_h based on the segmentation map map_seg_h and the confidence map map_conf_h. The two correction operations may be performed in parallel or sequentially. An enhancement image may be produced as a result of the correction.

Specifically, the tuning circuit 724 may determine a class for each pixel of the image image_h based on the segmentation map map_seg_h and the confidence map map_conf_h, and determine a correction effect and correction value to be applied to each pixel. . The tuning circuit 724 may generate a correction map including a correction effect and a correction value.

The at least one correction circuit 721 , 722 , and 723 may correct the image image_h based on the sub correction information. In addition, the image image_h may be corrected based on the correction map generated by the tuning circuit 724 . For example, the at least one correction circuit 721 , 722 , and 723 may correct the entire image image_h using the sub-correction information, and correct each pixel of the image image_h based on the correction map.

13 is a flowchart illustrating an operation of an image processing system according to an exemplary embodiment of the present disclosure.

12 and 13 together, an operation of receiving an image image_h may be performed (S310). A low-resolution image image_l may be generated based on the image image_h (S315). A low-quality segmentation map (map_seg_l) and a low-quality confidence map (map_conf_l) for the low-quality image image_l may be generated using the learned neural network model 731 ( S320 ). By enlarging the low-quality segmentation map (map_seg_l) and the low-resolution confidence map (map_conf_l), a segmentation map (map_seg_h) and a confidence map (map_conf_h) for the image (image_h) may be generated ( S325 ). A correction effect and a correction value to be applied to each pixel of the image image_h may be determined based on the segmentation map map_seg_h and the confidence map map_conf_h ( S330 ). A correction effect corresponding to the correction value may be applied to each pixel (S335).

A scene representing the image image_h may be determined (S345). Based on the determined scene, sub-correction information for the image image_h may be generated ( S350 ). Based on the sub-correction information, a correction effect may be applied to the entire image image_h ( S355 ).

Meanwhile, steps S345 to S355 may be performed in parallel or sequentially with steps S315 to S335.

14 is a block diagram illustrating a system according to an exemplary embodiment of the present disclosure.

Referring to FIG. 14 , a system 40 includes a mobile phone, a smart phone, a tablet computer, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, and a digital video camera. (digital video camera), PMP (portable multimedia player), PDN (personal navigation device) or portable navigation device), handheld game console, or e-book. It may be implemented as a handheld device.

System 40 may include SoC 500 and memory device 600 . The SoC 500 includes a central processing unit (CPU) 510 , a graphic processing unit (GPU) 520 , a neural processing unit (NPU) 530 , an image signal processor (ISP) 540 , and a memory interface (MIF). 550 , a clock management unit (CMU) 560 , and a power management unit (PMU) 570 . The CPU 510 , GPU 520 , NPU 530 , and ISP 540 may be referred to as a master IP device, and the MIF 550 may be referred to as a slave IP device.

At least one of the CPU 510 , the GPU 520 , the NPU 530 , and the ISP 540 may include the image processing systems 10 , 20 , and 30 described above with reference to FIGS. 1 to 13 . Each component of the image processing system 10 , 20 , and 30 may be implemented in the same IP device, and at least one component may be implemented in another IP device. As an embodiment, the GPU 520 may include an upscaling circuit 521 , the NPU 530 may include a segmentation circuit 531 , and the IPS may include an image processing circuit 541 .

For example, the ISP 540 may generate an image to be corrected, generate a low-quality image for the image, and provide it to the NPU 530 . The segmentation circuit 531 of the NPU 530 may generate a low-quality segmentation map and a low-quality confidence map, and provide the low-quality segmentation map and the low-quality confidence map to the GPU 520 . The upscaling circuit 521 of the GPU 520 may generate a segmentation map and a confidence map, and may provide the segmentation map and the confidence map back to the ISP 540 . The image processing circuit 541 of the ISP 540 may generate an enhanced image by performing pixel-wise correction on the image based on the segmentation map and the confidence map.

The CPU 510 may process or execute instructions and/or data stored in the memory device 600 in response to a clock signal generated by the CMU 560 .

The GPU 520 may acquire image data stored in the memory device 600 in response to the clock signal generated by the CMU 560 . The GPU 520 may generate data for an image output through a display device (not shown) from image data provided from the MIF 550 , or may encode the image data.

NPU 530 may refer to any device executing a machine learning model. The NPU 530 may be a hardware block designed to execute a machine learning model. The machine learning model may be a model based on an artificial neural network, a decision tree, a support vector machine, a regression analysis, a Bayesian network, a genetic algorithm, or the like. Artificial neural networks include, as non-limiting examples, a convolution neural network (CNN), a region with convolution neural network (R-CNN), a region proposal network (RPN), a recurrent neural network (RNN), and a stacking-based deep neural (S-DNN). network), S-SDNN (state-space dynamic neural network), Deconvolution Network, DBN (deep belief network), RBM (restricted Boltzmann machine), Fully Convolutional Network, LSTM (long short-term memory) Network, Classification Network can do.

The ISP 540 may perform a signal processing operation on raw data received from an image sensor (not shown) located outside the SoC 500 and generate digital data having improved image quality.

The MIF 550 may provide an interface to the memory device 600 located outside the SoC 500 . The memory device 600 may be a dynamic random access memory (DRAM), a phase-change random access memory (PRAM), a resistive random access memory (ReRAM), or a flash memory.

The CMU 560 may generate a clock signal and provide the clock signal to components of the SoC 500 . The CMU 560 may include a clock generator such as a phase locked loop (PLL), a delayed locked loop (DLL), and a crystal. The PMU 570 may convert external power to internal power and supply power to components of the SoC 500 from the internal power.

The SoC 500 may further include a volatile memory. The volatile memory may be implemented as dynamic RAM (DRAM), SRAM, or the like. The volatile memory may store various programs and data for the operations of the image processing circuit 541 , the segmentation circuit 531 , and the upscaling circuit 521 , and the image processing circuit 541 , the segmentation circuit 531 and the upscaling circuit 541 . Data generated by the scaling circuit 521 may be stored. As an embodiment, the low-quality segmentation map and the low-quality confidence map may be stored, or the segmentation map and the confidence map may be stored.

In one embodiment, each of the at least one correction circuit of the image processing circuit 541 may access the volatile memory through direct memory access (DMA). To this end, the SoC 500 may further include an access device such as a DMA controller, memory DMA (MDMA), peripheral DMA (PDMA), remote DMA (RDMA), smart DMA (SDMA), and the like.

15 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure .

Referring to FIG. 15 , an electronic device 1000 according to an exemplary embodiment of the present disclosure includes an image sensor 1100 , an image signal processor 1200 , a display device 1300 , an application processor 1400 , and a working memory 1500 . ), a storage 1600 , a user interface 1700 , and a wireless transceiver 1800 , and the image signal processor 2000 may be implemented as an integrated circuit separate from the application processor 1400 .

As an embodiment, the image processing systems 10, 20, and 30 described above with reference to FIGS. 1 to 13 may be implemented in the ISP 1200 and/or the AP 1400 . For example, the image processing circuit may be implemented in the ISP 1200 , and the remaining components may be implemented in the AP 1400 .

The image sensor 1100 may generate image data, for example, raw image data, based on the received optical signal, and provide the binary data to the image signal processor 1200 . The application processor 1400 controls the overall operation of the electronic device 1000 and may be provided as a system-on-chip (SoC) that drives an application program, an operating system, and the like. The application processor 1400 may control the operation of the image signal processor 1200 , and may provide the converted image data generated by the image signal processor 1200 to the display device 1300 or store it in the storage 1600 . have.

The working memory 1500 may store programs and/or data processed or executed by the application processor 1400 . The storage 1600 may be implemented as a non-volatile memory device such as a NAND flash or a resistive memory. For example, the storage 1600 may be provided as a memory card (MMC, eMMC, SD, micro SD) or the like. The storage 1600 may store data and/or a program for an execution algorithm that controls the image processing operation of the image signal processor 1200 , and when the image processing operation is performed, the data and/or program are stored in the working memory 1500 . can be loaded with

The user interface 1700 may be implemented with various devices capable of receiving a user input, such as a keyboard, a curtain key panel, a touch panel, a fingerprint sensor, and a microphone. The user interface 1700 may receive a user input and provide a signal corresponding to the received user input to the application processor 1400 . The wireless transceiver 1800 may include a modem 1810 , a transceiver 1820 , and an antenna 1830 .

Exemplary embodiments have been disclosed in the drawings and specification as described above. Although the embodiments have been described using specific terms in the present specification, these are used only for the purpose of explaining the technical spirit of the present disclosure, and are not used to limit the meaning or the scope of the present disclosure described in the claims. Therefore, it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present disclosure should be defined by the technical spirit of the appended claims.

Claims (20)

Receive a segmentation map including class inference information for each pixel of an image and a confidence map including reliability of the class inference information for each pixel, based on the segmentation map and the confidence map, a class for each pixel of the image, the image a tuning circuit for determining a correction effect for each pixel and a correction value indicating the strength of the correction effect, and generating a correction map based on the class for each pixel and the correction value for each pixel; and
at least one correction circuit for generating an enhanced image by applying a correction effect according to a correction value to each pixel of the image based on the correction map;
An image processing circuit comprising a. According to claim 1,
The tuning circuit is
a selection circuit for determining a class of each pixel based on the segmentation map, the confidence map, and a preset threshold;
a set value table including at least one correction effect according to the class of each pixel and the reliability of the class of each pixel and the intensity of each of the at least one correction effect; and
a mixing circuit that determines a correction effect to be applied to each pixel and the correction value based on the set value table, and generates a correction map for the correction effect;
An image processing circuit comprising a. 3. The method of claim 2,
The correction circuit is
An image processing circuit comprising at least one of a denoise circuit, a color correction circuit, and a sharpen circuit. 4. The method of claim 3,
The mix circuit is
A first correction map for the denoise effect is generated and provided to the denoise circuit, a second correction map for the color correction effect is generated and provided to the color correction circuit, and a third correction map for the sharpen effect is generated to the image processing circuit, characterized in that provided to the sharpen circuit. According to claim 1,
The segmentation map and the confidence map are
An image processing circuit, characterized in that it is generated by a neural network model trained using an arbitrary pixel and a correct answer class labeled corresponding to the arbitrary pixel. According to claim 1,
The class for each pixel of the image is,
An image processing circuit comprising at least one of a face class, a skin class, a sky class, a detail class, an eye class, an eyebrow class, and a hair class. 7. The method of claim 6,
The detail class is
An image processing circuit comprising at least one of a grass class, a sand class and a tree class. A method of improving image quality, comprising:
receiving the image;
generating first class inference information by inferring a class to which each pixel of the image belongs by using the learned neural network model, and calculating a first confidence level for the first class inference information;
determining a first correction effect and a first correction value to be applied to each pixel of the image based on a table in which correction values are determined according to class and reliability; and
generating an enhanced image by applying the first correction effect to each pixel by the first correction value;
Image quality improvement method comprising a. 9. The method of claim 8,
training the neural network model by using the training image and the correct answer class labeled in each pixel of the training image as training data;
further comprising,
The correct answer class corresponds to a correction effect to be applied to each pixel of the training image. A system on a chip for generating an enhancement image by correcting the image, comprising:
a first circuit for generating first class inference information for each pixel of the image and a first confidence level for the first class inference information by using the learned neural network model; and
The enhanced image is generated by determining a correction value for each pixel of the image based on the first class inference information and the first reliability, and applying a correction effect corresponding to the correction value to each pixel of the image a second circuit to;
A system-on-a-chip comprising a. 11. The method of claim 10,
The first circuit is
Receive a low-quality image for the image, and generate second class inference information for each pixel of the low-resolution image with the neural network model and a second confidence level for the second class inference information,
The system on chip,
a third circuit configured to generate the first class inference information and the first reliability based on the second class inference information and the second reliability;
System-on-a-chip, characterized in that it further comprises. 12. The method of claim 11,
The third circuit is
and generating the first class inference information and the first reliability by interpolating the second class inference information and the second reliability. 12. The method of claim 11,
The second circuit is
and generating the low-resolution image by using the image, and providing the low-resolution image to the first circuit. 12. The method of claim 11,
and the third circuit is included in an image signal processor (ISP). 11. The method of claim 10,
The neural network model is
A system on a chip, configured to learn a relationship between a plurality of classes and pixels classified to have different correction effects. 16. The method of claim 15,
The plurality of classes are
A system on a chip, characterized in that at least one of a denoise effect, a color correction effect, and a sharpening effect has different weights. 11. The method of claim 10,
The first circuit is
and generating a two-dimensional segmentation map having the same size as the image including the first class inference information and a two-dimensional confidence map having the same size as the image and including the first reliability. 11. The method of claim 10,
The first class inference information includes n bits,
and the first reliability includes m bits greater than n. 11. The method of claim 10,
A third circuit for determining a scene representing the image and generating sub-correction information corresponding to the scene;
The second circuit is
and correcting the image based on the sub-correction information, and correcting each pixel of the image based on the correction value. 20. The method of claim 19,
and the second and third circuits are included in an image signal processor (ISP).

Images