WO2021107264A1 - Deep learning-based sparse color sensor image processing device, image processing method and computer-readable medium - Google Patents

Abstract

The present invention relates to a deep learning-based sparse color sensor image processing device, an image processing method and a computer-readable medium, and, more specifically, to a deep learning-based sparse color sensor image processing device, an image processing method and a computer-readable medium, the device acquiring images of superior quality by restoring colors through an artificial neural network from images acquired through an image sensor including a color filter array that comprises a very small percentage of color pixels.

Description

Deep learning-based sparse color sensor image processing apparatus, image processing method and computer-readable medium

The present invention relates to a deep learning-based sparse color sensor image processing apparatus, an image processing method, and a computer-readable medium, and more particularly, to an image sensor including a color filter array including a very small proportion of color pixels. It relates to a deep learning-based sparse color sensor image processing apparatus, an image processing method, and a computer-readable medium capable of obtaining a high-quality image by restoring color from an image through an artificial neural network.

In general, an image sensor is a semiconductor device that converts an optical image into an electrical signal, and a charge couple device (CCD) image sensor and a complementary metal-oxide semiconductor (CMOS) image sensor are widely used.

Since the photo-sensing device used in such an image sensor generally measures only the intensity of an optical signal and cannot detect the spectral characteristics, only optical signals of a certain frequency are passed through a color filter for a predetermined number of frequency domains. Each of the color filter and the light sensing element is provided to obtain the intensity of an optical signal for each frequency region, and color image data (R, G, B data) is obtained therefrom. As such a color filter, a color filter array (CFA) in which color filters such as red, green, and blue are regularly arranged is used.

1 is a diagram schematically illustrating an image sensor generally used. A typical image sensor is composed of a color filter array 20 and a sensor array 30 in which a light sensing element for detecting light passing through the color filter array 20 is arranged as shown in FIG. 1 .

A red filter (R), a green filter (G), and a blue filter (B) are regularly arranged in the color filter array 20 . One such color filter 20.1 corresponds to one photo-sensing element 30.1, and the photo-sensing element 30.1 detects the intensity of light passing through the color filter 20.1, The intensity of the light of that color is sensed.

The minimum repeating unit of the Bayer pattern shown in FIG. 1 includes in common two rows and two columns, that is, an array structure of 2 X 2

In the Bayer pattern, the red filter (R), green filter (G), and blue filter (B) have a ratio of 1:2:1. In this Bayer pattern, a red filter (R) and a blue filter (B) are arranged in a diagonal direction, and two green filters (G) are arranged in a diagonal direction that intersects the same.

Through such a configuration, light passing through each color filter is detected by each photo-sensing element, and an image is formed based on the intensity of the detected light. However, since only the intensity of light of one color can be read through the color filter, a demosaicing process of restoring the color based on information obtained through filters of other adjacent colors is required.

On the other hand, the light passing through the color filter of the color filter array as described above is lost by the color filter, and this characteristic is used to amplify a signal for a weak light intensity, especially when an image is sensed in a low-light environment. In the process, the influence of noise increases and the quality of the acquired image decreases.

In order to improve this, a method has been proposed in which a white filter that passes light of all frequencies, rather than a color filter, is disposed in the color filter array, thereby reducing the influence on noise and obtaining a high-quality image even in a low-light environment.

In this method, by using the white filter, light loss due to the color filter can be reduced even in a low-light environment, but color information cannot be obtained from the white filter, which is disadvantageous in color restoration in the demosaicing process. Therefore, in the current color filter array, the ratio of the white filter is very low to obtain color information, and when the white pattern is increased, a problem may occur in color restoration using the conventional method. Therefore, it is necessary to develop an apparatus and an image processing algorithm for obtaining a high-quality image by a new technique in a low light environment.

The present invention is a deep learning-based sparse color sensor image capable of obtaining excellent quality images by restoring colors through an artificial neural network from an image obtained through an image sensor including a color filter array containing a very small proportion of color pixels. An object of the present invention is to provide a processing apparatus, an image processing method and a computer-readable medium.

In order to solve the above problems, in one embodiment of the present invention, there is provided an image processing apparatus having one or more processors and one or more memories, receiving image data from an electrically connected image sensor and outputting a merged image, the image The processing apparatus may include: a first image input unit configured to receive a first image generated based on information sensed by a plurality of white pixels of the image sensor; a second image input unit for receiving a second image generated based on information sensed by a plurality of color pixels of the image sensor; and an image merging unit including one or more learned artificial neural networks and generating a merged image based on the first image and the second image; wherein the image merging unit includes: a luminance restoration unit for generating a luminance restored image by deriving the luminance of a region in which the color pixel is located based on the first image; and a color restoration unit for generating a merged image based on data including the second image and the luminance restored image. It provides an image processing apparatus comprising a.

In one embodiment of the present invention, the first image includes white pixel information of 90% or more of the total number of pixels of the merged image, and the second image contains less than 10% of the total number of pixels of the merged image. It may include color pixel information.

In an embodiment of the present invention, the luminance restoration unit and the color restoration unit may generate the luminance restoration image and the merged image based on each learned artificial neural network.

In an embodiment of the present invention, the image merging unit includes: a boundary extractor for generating a boundary image including color boundary information of the luminance restoration image through the luminance restoration unit; The method further includes, wherein the color restoration unit may generate a merged image based on the second image, the luminance restoration image, and the boundary image.

In an embodiment of the present invention, the color restoration unit may be configured with a generative adversarial network (GAN) to generate a merged image.

In an embodiment of the present invention, the color restoration unit includes: a color restoration module for generating a merged image based on data including the second image and the luminance restoration image; and an evaluation module for evaluating the merged image generated by the color restoration module. Including, the color restoration module and the evaluation module may perform mutual feedback.

In an embodiment of the present invention, the evaluation module receives a plurality of pre-stored learning merged images, evaluates the authenticity of the learned merged images, and performs feedback based on the evaluation results, so that learning can be performed.

In an embodiment of the present invention, the color restoration module receives a plurality of pre-stored learning second images, learning luminance restoration images, and learning boundary image sets to generate a learning merged image, and the evaluation module is generated. Learning can be performed by performing feedback based on the evaluation result of evaluating the merged image.

In an embodiment of the present invention, the color restoration module may be configured as a high-density U-net, and the evaluation module may be configured as a stacked convolutional neural network.

In an embodiment of the present invention, the luminance restoration unit may be configured as a stacked convolutional neural network.

In an embodiment of the present invention, the artificial neural network of each of the luminance restoration unit and the color restoration unit may be an artificial neural network in which learning is performed independently.

In order to solve the above problems, in an embodiment of the present invention, an image is performed in a computing device having one or more processors and one or more memories, and receives image data from an electrically connected image sensor and outputs a merged image A processing method, the image processing method comprising: a first image acquisition step of receiving a first image generated based on information sensed by a plurality of white pixels of the image sensor;

a second image acquisition step of receiving a second image generated based on information sensed by a plurality of color pixels of the image sensor; and an image merging step of generating a merged image based on the first image and the second image by using one or more learned artificial neural networks. wherein the image merging step comprises: a luminance restoration step of generating a luminance restored image by deriving the luminance of an area in which the color pixel is located based on the first image; and a color restoration step of generating a merged image based on the second image and the luminance restoration image. It provides an image processing method comprising a.

In one embodiment of the present invention, as a computer-readable medium for implementing an image processing method for outputting a merged image by receiving image data from an electrically connected image sensor, the computer-readable medium enables a computing device to Stores instructions for performing the following steps, the steps comprising: a first image acquisition step of receiving a first image generated based on information sensed by a plurality of white pixels of the image sensor; a second image acquisition step of receiving a second image generated based on information sensed by a plurality of color pixels of the image sensor; and an image merging step of generating a merged image based on the first image and the second image by using one or more learned artificial neural networks. wherein the image merging step comprises: a luminance restoration step of generating a luminance restored image by deriving the luminance of an area in which the color pixel is located based on the first image; and a color restoration step of generating a merged image based on the second image and the luminance restoration image. It provides a computer-readable medium comprising a.

In order to solve the above problems, the present invention provides an image processing apparatus having one or more processors and one or more memories, receiving image data from an electrically connected image sensor and outputting a merged image, the image processing apparatus comprising: a first image input unit receiving a first image generated based on information sensed by a plurality of white pixels of the image sensor; a second image input unit for receiving a second image generated based on information sensed by a plurality of color pixels of the image sensor; and an image merging unit for generating a merged image based on the first image and the second image by the learned artificial neural network. wherein the first image includes white pixel information of 70% or more of the total number of pixels of the merged image, and the second image includes color pixel information of 30% or less of the total number of pixels of the merged image. It provides an image processing apparatus, including.

In the present invention, each of the plurality of color pixels includes an R pixel that transmits red light; G pixels that transmit green light; and a B pixel that transmits blue light; any one of the above, and in the second image, a plurality of color pixels may be arranged according to a predetermined pattern.

In the present invention, the preset pattern may include a plurality of color pixel groups each including one or more of the R, G, and B pixels and arranged adjacently.

In the present invention, the plurality of color pixel groups may be arranged periodically according to a predetermined interval.

In the present invention, the learned artificial neural network may be learned based on the structural similarity (SSIM) of the image.

In the present invention, the learned artificial neural network includes a convolutional neural network, the first image is 1-channel data including only brightness information, the second image is 3-channel data including color information, and the convolution The neural network may use the data of the second image as a color hint.

In the present invention, the output of the first convolution block of the second image may be input to the first convolution block of the first image.

In order to solve the above problems, the present invention is an image processing method for outputting a merged image by receiving image data from an electrically connected image sensor and being performed in a computing device having one or more processors and one or more memories, The image processing method may include: a first image input step of receiving a first image generated based on information sensed by a plurality of white pixels of the image sensor; a second image input step of receiving a second image generated based on information sensed by a plurality of color pixels of the image sensor; and an image merging step of generating a merged image based on the first image and the second image by the learned artificial neural network. wherein the first image includes white pixel information of 70% or more of the total number of pixels of the merged image, and the second image includes color pixel information of 30% or less of the total number of pixels of the merged image. It provides an image processing method, including.

In the present invention, each of the plurality of color pixels includes an R pixel that transmits red light; G pixels that transmit green light; and a B pixel that transmits blue light; any one of the above, and in the second image, a plurality of color pixels may be arranged according to a predetermined pattern.

In the present invention, the preset pattern may include a plurality of color pixel groups each including one or more of the R, G, and B pixels and arranged adjacently.

In order to solve the above problems, the present invention provides a computer-readable medium for implementing an image processing method of receiving image data from an electrically connected image sensor and outputting a merged image, the computer-readable medium comprising: , a first image input step of receiving a first image generated based on information sensed by a plurality of white pixels of the image sensor. ; a second image input step of receiving a second image generated based on information sensed by a plurality of color pixels of the image sensor; and an image merging step of generating a merged image based on the first image and the second image by the learned artificial neural network, wherein the first image is white at least 70% of the total number of pixels of the merged image and pixel information, and wherein the second image includes color pixel information of no more than 30% of the total number of pixels of the merged image.

In the present invention, each of the plurality of color pixels includes an R pixel that transmits red light; G pixels that transmit green light; and a B pixel that transmits blue light; any one of the above, and in the second image, a plurality of color pixels may be arranged according to a predetermined pattern.

In the present invention, the preset pattern may include a plurality of color pixel groups each including one or more of the R, G, and B pixels and arranged adjacently.

In order to solve the above problems, the present invention provides an image sensing device, comprising: a color filter array including filters for a plurality of pixels; a sensor array in which a plurality of sensors for sensing light are arranged corresponding to the plurality of pixels of the color filter array; and an image sensor substrate for generating image data based on the data sensed from the sensor array; An image sensor comprising a; and an image processing device electrically connected to the image sensor, receiving the image data and outputting a merged image, wherein the number of white pixels in the color filter array is 70% or more of the total number of pixels, and the image processing The apparatus may include: a first image input unit for receiving a first image generated based on information sensed by a plurality of white pixels of the image sensor; a second image input unit for receiving a second image generated based on information sensed by a plurality of color pixels of the image sensor; and an image merging unit that generates a merged image based on the first image and the second image by means of a learned artificial neural network, wherein the first image is white or more than 70% of the total number of pixels of the merged image. and pixel information, and wherein the second image includes color pixel information of 30% or less of the total number of pixels of the merged image.

According to an embodiment of the present invention, by sensing light using a color filter array having a very high ratio of white pixels, it is possible to obtain an image of excellent quality in a low-light environment.

According to an embodiment of the present invention, color restoration through an artificial neural network based on color information obtained through a very small proportion of color pixels can exhibit the effect of performing high-quality color restoration.

According to an embodiment of the present invention, by generating a boundary image and using it for color restoration, it is possible to exhibit the effect of reducing color bleeding in the color restoration process.

According to an embodiment of the present invention, by restoring a color through a generative adversarial neural network, it is possible to prevent false colors from being reproduced.

1 is a diagram schematically illustrating the structure of a color filter array and a sensor array of a general image sensor.

2 is a diagram schematically illustrating a structure of an image sensing device according to an embodiment of the present invention.

3 is a diagram schematically illustrating an internal structure of an image processing apparatus according to an embodiment of the present invention.

4 is a diagram schematically illustrating a pattern of a color filter array according to an embodiment of the present invention.

5 is a diagram schematically illustrating a pattern of a color filter array according to an embodiment of the present invention.

6 is a diagram schematically illustrating operations of an image merging unit and a merged image output unit according to an embodiment of the present invention.

7 is a diagram schematically illustrating an operation of an artificial neural network of an image merging unit according to an embodiment of the present invention.

8 is a diagram schematically illustrating the structure of a convolutional neural network of an image merging unit according to an embodiment of the present invention.

9 is a diagram schematically illustrating each step of an image sensing method according to an embodiment of the present invention.

10 is a diagram illustrating an image obtained according to an embodiment of the present invention.

11 is a diagram illustrating an image obtained according to an embodiment of the present invention.

12 is a ground truth image for the images of FIGS. 10 and 11 .

13 is a diagram schematically illustrating an internal configuration of an image merging unit according to an embodiment of the present invention.

14 is a diagram schematically illustrating the structure of an artificial neural network of a luminance restoration unit, a color restoration module, and an evaluation module according to an embodiment of the present invention.

15 is a view exemplarily showing a first image, a second image, a luminance restored image, and a boundary image according to an embodiment of the present invention.

16 is a diagram schematically illustrating a learning step of an image merging unit according to an embodiment of the present invention.

17 is a diagram schematically illustrating each step of an image processing method according to an embodiment of the present invention.

18 is a diagram schematically illustrating detailed steps of an image merging step according to an embodiment of the present invention.

19 is a diagram schematically illustrating a pattern of a color filter array according to an embodiment of the present invention.

20 is a block diagram illustrating an example of an internal configuration of a computing device according to an embodiment of the present invention.

Hereinafter, various embodiments and/or aspects are disclosed with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of one or more aspects. However, it will also be recognized by one of ordinary skill in the art that such aspect(s) may be practiced without these specific details. The following description and accompanying drawings set forth in detail certain illustrative aspects of one or more aspects. These aspects are illustrative, however, and some of the various methods in principles of various aspects may be employed, and the descriptions set forth are intended to include all such aspects and their equivalents.

Further, various aspects and features will be presented by a system that may include a number of devices, components and/or modules, and the like. It is also noted that various systems may include additional devices, components, and/or modules, etc. and/or may not include all of the devices, components, modules, etc. discussed in connection with the figures. must be understood and recognized.

As used herein, “embodiment”, “example”, “aspect”, “exemplary”, etc. may not be construed as an advantage or advantage in any aspect or design described above over other aspects or designs. . The terms '~part', 'component', 'module', 'system', 'interface', etc. used below generally mean a computer-related entity, for example, hardware, hardware A combination of and software may mean software.

Also, the terms "comprises" and/or "comprising" mean that the feature and/or element is present, but excludes the presence or addition of one or more other features, elements, and/or groups thereof. should be understood as not

Also, terms including an ordinal number, such as first, second, etc., may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

In addition, in the embodiments of the present invention, unless otherwise defined, all terms used herein, including technical or scientific terms, are generally understood by those of ordinary skill in the art to which the present invention belongs. have the same meaning as Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the embodiment of the present invention, an ideal or excessively formal meaning is not interpreted as

2 is a diagram schematically illustrating a structure of an image sensing device according to an embodiment of the present invention.

Referring to FIG. 2 , an image sensing apparatus according to an embodiment of the present invention includes: a color filter array 2000 including filters for a plurality of pixels; a sensor array 3000 in which a plurality of sensors for sensing light are arranged corresponding to the plurality of pixels of the color filter array; and an image sensor substrate 4000 for generating image data based on the data sensed from the sensor array; An image sensor including a (5000); and an image processing apparatus 1000 electrically connected to the image sensor 5000, receiving the image data, and outputting a merged image.

The color filter array 2000 includes filters for a plurality of pixels, and the filters pass light so that each sensor of the sensor array 3000 can detect light. The color filter array may include a plurality of micro lenses to collect and transmit light to a corresponding sensor.

The sensor array 3000 has a plurality of sensors for detecting light corresponding to the filter of the color filter array 2000, and detects the light passing through the filter. In an embodiment of the present invention, the sensor may be configured as CCD or CMOS to detect light.

The image sensor substrate 4000 generates image data from information detected by each sensor of the sensor array 3000 . The image data includes information on the light detected by each sensor of the sensor array 3000 and information on the position of the sensor, so that a two-dimensional image can be generated.

The pixel includes a white pixel that transmits light of all colors; or color pixels that pass only light of a preset color; to be. The white pixel transmits light of all colors, and a sensor corresponding to the white pixel detects the intensity of light and provides brightness information, and the color pixel transmits light of a preset color (eg, red, green or blue), the sensor corresponding to the color pixel senses the intensity of light of the preset color and provides color information. The image processing apparatus 1000 generates a final output image based on the brightness information and the color information.

In an embodiment of the present invention, in the color filter array, the number of white pixels is 90% or more of the total number of pixels. In this embodiment, the number of color pixels is 10% or less of the total number of pixels.

In an embodiment of the present invention, in the color filter array, the number of white pixels is 95% or more of the total number of pixels. In this embodiment, the number of color pixels is 5% or less of the total number of pixels.

Preferably, the number of the white pixels is 99% or more of the total number of pixels. As described above, when the number of white pixels is greater than or equal to 99% of the total number of pixels, the number of color pixels is less than or equal to 1% of the total number of pixels.

As described above, when the number of color pixels is extremely small, a process of restoring color to the entire image based on color information obtained through the color pixels is required, and the image processing apparatus 1000 performs such an operation. At this time, in an embodiment of the present invention, the image processing apparatus 1000 may perform a process of restoring color through an artificial neural network.

3 is a diagram schematically illustrating an internal structure of an image processing apparatus according to an embodiment of the present invention.

3 is a block diagram illustrating internal components of the image processing apparatus 1000 according to an embodiment of the present invention.

The image processing apparatus 1000 according to the present embodiment may include a processor 100 , a bus 200 , a network interface 300 , and a memory 400 . The memory may include an operating system 410 , an image merging routine 420 , and sensor information 430 . The processor 100 may include a first image input unit 110 , a second image input unit 120 , an image merging unit 130 , and a merged image output unit 140 . In other embodiments, the image processing apparatus 1000 may include more components than those of FIG. 3 .

The memory is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. In addition, program codes for the operating system 410 , the image merging routine 420 , and the sensor information 430 may be stored in the memory. These software components may be loaded from a computer-readable recording medium separate from the memory using a drive mechanism (not shown). The separate computer-readable recording medium may include a computer-readable recording medium (not shown) such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, the software components may be loaded into the memory through the network interface unit 300 instead of a computer-readable recording medium.

The bus 200 may enable communication and data transmission between components of a computing device that controls the image processing apparatus 1000 . The bus may be configured using a high-speed serial bus, a parallel bus, a storage area network (SAN), and/or other suitable communication technology.

The network interface unit 300 may be a computer hardware component for connecting a computing device controlling the image processing apparatus 1000 to a computer network. The network interface 300 may connect a computing device controlling the image processing apparatus 1000 to a computer network through a wireless or wired connection. A computing device that controls the image processing apparatus 1000 through the network interface unit 300 may be wirelessly or wiredly connected to the image processing apparatus 1000 .

The processor may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations of the computing device controlling the image processing apparatus 1000 . The command may be provided to the processor 100 by the memory or network interface unit 300 and via the bus 200 . The processor may be configured to execute program codes for the first image input unit 110 , the second image input unit 120 , the image merging unit 130 , and the merged image output unit 140 . Such program code may be stored in a recording device such as a memory.

The first image input unit 110 , the second image input unit 120 , the image merging unit 130 , and the merged image output unit 140 are configured to perform an operation of the image processing apparatus 1000 , which will be described below. can be

In the processor, some components may be omitted, additional components not shown may be further included, or two or more components may be combined according to a method of controlling the image processing apparatus 1000 .

The image processing apparatus 1000 according to an embodiment of the present invention includes: a first image input unit 110 for receiving a first image generated based on information sensed by a plurality of white pixels of the image sensor; a second image input unit 120 for receiving a second image generated based on information sensed by a plurality of color pixels of the image sensor; an image merging unit 130 for generating a merged image based on the first image and the second image by the learned artificial neural network; and a merged image output unit 140 for outputting the merged image to the outside. includes

The first image input unit 110 receives the first image obtained from the sensor of the sensor array 3000 corresponding to the white pixel. The first image may be generated from information sensed by a sensor of the sensor array 3000 corresponding to the white pixel. In an embodiment of the present invention, the first image may include white pixel information of 95% or more of the total number of pixels of the merged image.

In an embodiment of the present invention, the first image may be generated by receiving only information sensed from a sensor of the sensor array 3000 corresponding to the white pixel, or obtained from all sensors of the sensor array 3000 The image may be generated by removing information on a pixel corresponding to a position of the sensor corresponding to the color pixel in addition to a pixel corresponding to a position of the sensor corresponding to the white pixel in an image.

The first image as described above is composed of information obtained by the sensor corresponding to the white pixel and may be generated as a mono image without color information.

The second image input unit 120 receives the second image obtained from the sensor of the sensor array 3000 corresponding to the color pixel. The second image may be generated from information sensed by a sensor of the sensor array 3000 corresponding to the color pixel. In an embodiment of the present invention, the second image may include color pixel information of 5% or less of the total number of pixels of the merged image.

In an embodiment of the present invention, the second image may be generated by receiving only information sensed from a sensor of the sensor array 3000 corresponding to the color pixel, or obtained from all sensors of the sensor array 3000 The image may be generated by removing information on a pixel corresponding to a location of the sensor corresponding to the white pixel in addition to a pixel corresponding to a location of the sensor corresponding to the color pixel in an image.

The second image may be generated as a color image by being composed of information obtained by the sensor corresponding to the color pixel.

The image merging unit 130 receives the first image and the second image from the first image input unit 110 and the second image input unit 120 , and based on the first image and the second image to create a merged image. According to an embodiment of the present invention, the first image is a mono image including only brightness information, and the second image is a color image including color information, and the image is obtained by merging the first image and the second image. The merging unit 130 generates a merged image completely equipped with brightness information and color information. Preferably, the image merging unit 130 generates a merged image by merging the first image and the second image through a learned artificial neural network. Preferably, the image merging unit 130 according to an embodiment of the present invention may merge the first image and the second image through two-step processing. In the first step, a luminance restored image may be generated by restoring the luminance of the missing pixel of the first image, and in the second step, a merged image may be generated based on the luminance restored image and the second image.

The merged image output unit 140 outputs the merged image generated by the image merging unit 130 to the outside. The merged image output unit 140 senses light through the image sensing device, processes the sensed light information, and transmits the generated merged image to a connected device for use by a user or the like. The merged image output unit 140 may transmit the merged image to an external computing device or the like through the network interface 300 .

4 is a diagram schematically illustrating a pattern of a color filter array according to an embodiment of the present invention.

Referring to FIG. 4 , the color filter array 2000 according to an embodiment of the present invention includes a plurality of pixels, wherein the pixels include white pixels that pass light of all colors; or color pixels that pass only light of a preset color; and the color pixel includes: an R pixel that transmits red light; G pixels that transmit green light; and a B pixel that transmits blue light; one of them, and are arranged according to a predetermined pattern on the color filter array.

Referring to FIG. 4 , the color filter array 2000 includes three types of color pixels (R, G, B) and white pixels (W), and the color pixels have a pattern preset in the color filter array 2000 . are arranged according to The preset pattern shown in FIG. 4 is only one embodiment of the present invention, and the color pixels of the present invention may be arranged in another pattern not shown in FIG. 4 .

Referring to FIG. 4 , the preset pattern according to an embodiment of the present invention is a repeating pattern having a size of 10×10 pixels. For convenience of description, the nth pixel from the left and the mth pixel from the top in the 10×10 size pattern are referred to as (n,m) pixels.

In the embodiment shown in Fig. 4, (1,1) pixel is B pixel, (10,1) pixel is G pixel, (5,5) pixel is R pixel, (6,5) pixel and (5,6) pixel A pixel is a G pixel, a (6,6) pixel is a B pixel, a (1,10) pixel is a G pixel, a (10,10) pixel is an R pixel, and all other pixels are white pixels 2200 .

By the white pixels 2200 and the color pixels 2100 arranged in this way, the sensors of the sensor array 3000 corresponding to the pixels can acquire brightness information and color information, respectively.

In this case, in an embodiment of the present invention, the preset pattern includes a color pixel group including one or more of the R pixel, the G pixel, and the B pixel and arranged adjacently, and the color pixel group includes the preset They are arranged regularly in the pattern. Preferably, the color pixel group includes one or more of the R, G, and B pixels, respectively, and is arranged adjacent to each other.

In the embodiment shown in FIG. 4, four pixels of (5,5) pixels, (6,5) pixels, (5,6) pixels, and (6,6) pixels are arranged adjacently to form a color pixel group 2100. a) to form This group of color pixels includes R pixels ((5,5) pixels), G pixels ((6,5) pixels, (5,6) pixels), and B pixels ((6,6) pixels). constitutes

In addition, (1,1) pixels, (10,1) pixels, (1,10) pixels, and (10,10) pixels are arranged adjacently at the corners of the 10 x 10 pattern to form a color pixel group 2100. b) form.

By forming a color pixel group each including one or more R pixels, G pixels, and B pixels in this way, intensity information of each of red, green, and blue in the color pixel group area can be obtained, and the three colors Based on each intensity information, color information in the color pixel group area can be obtained.

As such, in an embodiment of the present invention, the preset pattern includes a color pixel group in which at least one R pixel, G pixel, and B pixel are each included and arranged adjacently, thereby providing accurate color information required for color restoration and providing a higher value It can have the effect of performing color restoration of quality.

Information on such a preset pattern may be stored in the sensor information 430 of the memory 400 and utilized in the image processing apparatus 1000 . For example, the first image input unit 110 may acquire a first image from the image data received from the image sensor 500 based on the information on the preset pattern.

5 is a diagram schematically illustrating a pattern of a color filter array according to an embodiment of the present invention.

FIG. 5 shows a pattern of a color filter array having an arrangement different from that shown in FIG. 4 .

Referring to FIG. 5 , the color filter array 2000 includes three types of color pixels (R, G, B) and white pixels (W), and the color pixels have a pattern preset in the color filter array 2000 . are arranged according to The preset pattern shown in FIG. 5 is only one embodiment of the present invention, and the color pixels of the present invention may be arranged in another pattern not shown in FIG. 5 .

Referring to FIG. 5 , the preset pattern according to an embodiment of the present invention is a repeating pattern having a size of 18×18 pixels. For convenience of description, the n-th pixel from the left and the m-th pixel from the top in the 18×18 size pattern are referred to as (n,m) pixels.

In the embodiment shown in Fig. 5, (9,9) pixels are R pixels, (10,9) pixels and (9,10) pixels are G pixels, and (10,10) pixels are B pixels color pixels 2100 , and the remaining pixels are all white pixels 2100 . As described above, since four color pixels are included in the 18 x 18 pattern, the ratio of color pixels is 1.13% and the ratio of white pixels is 98.86%, which is a very low pattern of color pixels.

By the white pixels 2200 and the color pixels 2100 arranged in this way, the sensors of the sensor array 3000 corresponding to the pixels can acquire brightness information and color information, respectively.

6 is a diagram schematically illustrating operations of an image merging unit and a merged image output unit according to an embodiment of the present invention.

Referring to FIG. 6 , the image merging unit 130 according to an embodiment of the present invention receives a first image 1 and a second image ( 1 ) from the first image input unit 110 and the second image input unit 120 , respectively. 2) is input.

The first image 1 and the second image 2 shown in FIG. 6 are images obtained by the pattern of the color filter array 2000 as shown in FIG. 4 . The first image 1 is obtained from the sensor of the sensor array corresponding to the white pixel 2200 excluding the color pixel 2100 of the color filter array 2000, and the area except for the color pixel 2100 (white color) In the display), brightness information is included by detecting the light intensity of each pixel. As shown in FIG. 6 , the first image 1 includes brightness information on the remaining areas of the color pixel groups 2100.a and 2100.b arranged by a preset pattern, leaving holes. .

The second image 2 is obtained from the sensor of the sensor array corresponding to the color pixel 2100 of the color filter array 2000, and the color set in each pixel in the area (displayed in white) of the color pixel 2100 Color information is included by sensing the intensity of light. As shown in FIG. 6 , the second image 2 includes color information for the area of the color pixel group 2100.a and 2100.b.

In an embodiment of the present invention, the image merging unit 130 may generate a merged image by merging the first image 1 and the second image 2 based on the learned artificial neural network. Preferably, the artificial neural network is a convolutional neural network. The convolutional neural network is an algorithm that has been optimized so that the artificial neural network can acquire two-dimensional images well by merging the filter technology with the artificial neural network.

By merging the first image and the second image through the artificial neural network to generate a merged image, a higher quality merged image can be generated.

The merged image generated through the image merging unit 130 is output to an external device or the like through the merged image output unit 140 . For example, the merged image output unit 140 may store the merged image in a memory card connected through the network interface 300 or transmit and store the merged image to a cloud service connected through the Internet network. .

Hereinafter, a detailed process of merging images acquired through the color filter array 2000 as described above will be described through embodiments.

Merge of images acquired through color filter array containing color pixel group

The merging of images described below merges the images acquired through the color filter array 2000 as shown in FIG. 4 .

7 is a diagram schematically illustrating an operation of an artificial neural network of an image merging unit according to an embodiment of the present invention.

7A illustrates a process in which the artificial neural network 135 of the image merging unit 130 performs learning from learning data according to an embodiment of the present invention. Referring to (a) of FIG. 7 , the artificial neural network 135 may acquire first learning data corresponding to a first image obtainable according to an embodiment of the present invention and an embodiment of the present invention. The second learning data corresponding to the second image is received and the first learning data and the second learning data are merged to generate a learning merged image. The first learning data is a mono image including brightness information for a region excluding the color pixel 2100 like the first image, and the second learning data is for the region of the color pixel 2100 like the second image. It is a color image containing color information. Such first and second learning data correspond to an educational data set capable of performing supervised learning on the artificial neural network 135 .

In addition, the artificial neural network 135 also receives the third learning data. The third learning data is a color image generated by the first learning data and the second learning data, and is a ground truth for a learning merged image generated by merging the first learning data and the second learning data. Truth) image. The third learning data constitutes an education data set for supervised learning of the artificial neural network 135 together with the first learning data and the second learning data.

The artificial neural network 135 receives the third learning data as shown in (a) of FIG. 7 . After that, the artificial neural network 135 compares the learning merged image with the third learning data, and the learning merged image generated by the artificial neural network 135 from the first learning data and the second learning data is the second learning data. 3 Perform learning to get closer to the learning data. This may be performed through a process of changing the internal parameters of the artificial neural network 135 through a backpropagation algorithm or the like.

7B illustrates a process in which the learned artificial neural network 135 generates a merged image. The artificial neural network 135 learned through the same process as in FIG. 7A receives the first image and the second image and generates a merged image. The artificial neural network 135, which has performed learning through an education data set composed of the first learning data, the second learning data, and the third learning data, receives the first image and the second image as input and merges the image. will create

8 is a diagram schematically illustrating the structure of a convolutional neural network of an image merging unit according to an embodiment of the present invention.

Referring to FIG. 8 , the first image is 1-channel data including only brightness information, and the second image is 3-channel data including color information. The first image and the second image may be input to a convolutional neural network, respectively.

In an embodiment of the present invention, the convolutional neural network may be configured as a cross-channel autoencoder using the second image as a color hint.

In more detail, referring to FIG. 8 , the convolutional neural network includes ten convolution blocks from a first convolution block to a tenth convolution block. Each of the convolution blocks includes two to three pairs of convolution and activation functions, the first convolution block and the tenth convolution block have a height H and a width W of 64 channels, and the second convolution block and the ninth convolution block has a size of 128 channels in a height H/2 and a width W/2, and the third convolution block and the eighth convolution block have a size of 256 channels in a height H/4 and a width W/4. , and the fourth to seventh convolution blocks have a height of H/8, a width of W/8, and a size of 512 channels. Preferably, the activation function is a Rectified Linear Unit (ReLU).

When the spatial resolution is reduced as in from the first convolution block to the second convolution block, from the second convolution block to the third convolution block, and from the third convolution block to the fourth convolution block, by downsampling Reduce the spatial resolution, and increase the spatial resolution as in the case from the 7th convolution block to the 8th convolution block, from the 8th convolution block to the 9th convolution block, and from the 9th convolution block to the 10th convolution block. In this case, the spatial resolution is increased by upsampling.

The output of the first convolutional block of the second image is connected to the first convolutional block of the first image.

In addition, the output of the first convolution block is connected to the tenth convolution block, the output of the second convolution block is connected to the ninth convolution block, and the output of the third convolution block is connected to the eighth convolution block. .

The last layer of the tenth convolution block is a 1 x 1 kernel and generates output colors of 3 channels. In an embodiment of the present invention, the output color of the three channels may be an RGB color value of an RGB color space.

Such a convolutional neural network is trained in the following way.

The mapping function of the convolutional neural network may be expressed as follows.

F(M, C; θ)

In this case, M is the first image, C is the second image, and θ is the parameter of the convolutional neural network.

At this time, the loss function representing the degree of correspondence between the output value Y' and the actual data Y may be expressed as follows.

L(Y', Y)

In this case, the convolutional neural network learns the convolutional neural network by deriving the parameter θ that satisfies the following equation.

In an embodiment of the present invention, a convolutional neural network is trained using a structural similarity (SSIM) of an image in order to prevent color bleeding in the color restoration process through the convolutional neural network.

9 is a diagram schematically illustrating each step of an image processing method according to an embodiment of the present invention.

Referring to FIG. 9 , an image processing method according to an embodiment of the present invention is an image processing method that receives image data from an electrically connected image sensor and outputs a merged image, and is sensed by a plurality of white pixels of the image sensor. a first image input step (S110) of receiving a first image generated on the basis of the received information; a second image input step (S120) of receiving a second image generated based on information sensed by a plurality of color pixels of the image sensor; an image merging step (S200) of generating a merged image based on the first image and the second image by the learned artificial neural network; and a merged image output step (S300) of outputting the merged image to the outside; includes

The first image acquisition step S110 performs the same operation as that of the first image input unit 110 described above. A first image is obtained by receiving information sensed from a sensor corresponding to the white pixel 2100 among the sensors of the connected sensor array 3000 . Such a first image may be configured by receiving only information sensed from a sensor corresponding to the white pixel 2100, or after receiving information sensed from all sensors of the sensor array 3000, the white pixel It may be configured by selecting information sensed from a sensor corresponding to 2100 .

The second image acquisition step S120 performs the same operation as that of the second image input unit 120 described above. A second image is obtained by receiving information sensed from a sensor corresponding to the color pixel 2200 among the sensors of the connected sensor array 3000 . Such a second image may be configured by receiving only information sensed from a sensor corresponding to the color pixel 2200, or after receiving information sensed from all sensors of the sensor array 3000, the color pixel It may be configured by selecting information sensed from a sensor corresponding to 2200 .

In an embodiment of the present invention, the first image acquisition step S110 and the second image acquisition step S120 may be performed simultaneously.

The image merging step (S200) generates a merged image based on the first image and the second image. In an embodiment of the present invention, in the image merging step (S200), the first image and the second image obtained in the first image acquisition step (S110) and the second image acquisition step (S120) are transmitted and received, A merged image is generated by merging the first image and the second image. According to an embodiment of the present invention, the first image is a mono image including only brightness information, and the second image is a color image including color information, and the image is obtained by merging the first image and the second image. In the merging step ( S200 ), a merged image fully equipped with brightness information and color information is generated. Preferably, in the image merging step (S200), the first image and the second image are merged through a learned artificial neural network to generate a merged image.

The merged image output step (S300) outputs the merged image to the outside. In the merged image output step (S300), light is sensed through the image sensing device, and the merged image generated by processing the sensed light information is transmitted to a connected device so that the user can use it. In the step of outputting the merged image ( S300 ), the merged image may be transmitted to an external computing device or the like through the network interface 300 .

10 and 11 are diagrams illustrating images obtained according to an embodiment of the present invention.

10 is a first image obtained from an image sensing device according to an embodiment of the present invention. The first image is a monochromatic image obtained from white pixels excluding the color pixel group of the color filter array 2000, and as shown in the upper part of FIG. 10, image information at the color pixel group position is processed as a black-and-white image. .

11 is a merged image generated by an image sensing device according to an embodiment of the present invention. By merging the first image and the second image including color information obtained from the color pixel group through an artificial neural network, a color image including color information may be obtained as shown in FIG. 11 .

As shown in FIG. 11 , the merged image merged through an artificial neural network according to an embodiment of the present invention is color restored based on information obtained through the color filter array 2000 including a very low ratio of color pixels. It reproduces colors very well.

12 is a ground truth image for the images of FIGS. 10 and 11 .

12 and 11, it can be confirmed that the merged image merged through an artificial neural network according to an embodiment of the present invention faithfully restores the ground truth image.

Merge of images acquired through color filter array containing sparse color pixels

The image merging described below merges the images acquired through the color filter array 2000 as shown in FIG. 5 .

13 is a diagram schematically illustrating an internal configuration of an image merging unit according to an embodiment of the present invention.

As described above, the image merging unit 130 according to an embodiment of the present invention merges the first image 1 and the second image 2 through a two-step process to form a merged image 3 . can create To this end, the image merging unit 130 generates a luminance restored image 4 by deriving the luminance of a region in which the color pixel 2100 is located based on the first image 1 . ); a boundary extraction unit 132 for generating a boundary image 5 including color boundary information of the luminance restoration image 4 generated through the luminance restoration unit; and a color restoration unit 133 for generating a merged image 3 based on the second image 2, the luminance restoration image 4, and the boundary image 5; may include.

As described above, according to an embodiment of the present invention, the luminance restoration unit 131 and the boundary extraction unit 132 primarily perform processing on the first image 1 to obtain the luminance restoration image 4 and the boundary image. (5) is generated, and the second image (2), the luminance restored image (4), and the boundary image (5) are secondarily processed by the color restoration unit 133 to process the merged image (3) ) will be created.

As shown in FIG. 6 , the first image 1 includes brightness information on the remaining area in which the pixels in which the color pixels 2100 are located, which are arranged in a predetermined pattern, are left as holes. The luminance restoration unit 131 may generate the luminance restoration image 4 by receiving the first image 1 and deriving and filling the brightness information on the pixel in which the color pixel 2100 is located.

Meanwhile, the color restoration unit 133 may generate the merged image 2 by merging the luminance information of the luminance restoration image 4 with the color information of the second image 2 .

At this time, in an embodiment of the present invention, the luminance restoration unit 131 and the color restoration unit 133 generate the luminance restoration image 4 and the merged image 3 based on each artificial neural network. . That is, the luminance restoration unit 131 includes an artificial neural network for generating the luminance restoration image 4 , and the color restoration unit 133 uses an artificial neural network for generating the merged image 3 . consists of including The boundary extractor 132 may generate the boundary image 5 based on edge information extracted during the process in which the artificial neural network of the luminance restoration unit 131 generates the luminance restoration image 4 . The boundary image 5 generated in this way is used as information for generating the merged image 3 by the color restoration unit 133, thereby reducing color bleeding that may occur in the process of generating the merged image 3 have.

In another embodiment of the present invention, the color restoration unit 133 generates the merged image 3 by an artificial neural network, but the luminance restoration unit 131 uses an artificial neural network such as a known pixel interpolation algorithm. It is also possible to generate the luminance restored image 4 in a way that does not use .

In an embodiment of the present invention, the color restoration unit 133 may be configured with a generative adversarial network (GAN) to generate the merged image 3 . The generative adversarial neural network is an artificial neural network structure that includes two neural networks, a generator and a discriminator, and improves the performance of the generator by competitively learning the generator and the discriminator.

In the generative adversarial neural network, the generator generates data, and the discriminator determines whether input data is real data or generated false data. Accordingly, the discriminator improves the ability to discriminate the real data through learning, and the generator improves the ability to generate false data close to the real data to the extent that the discriminator cannot discriminate it through learning.

In an embodiment of the present invention, the color restoration unit 133 includes a color restoration module for generating a merged image based on the second image 2, the luminance restoration image 4, and the boundary image 5 ( 133a); and an evaluation module (133b) for evaluating the merged image generated by the color restoration module; Including, the color restoration module and the evaluation module may perform mutual feedback. In this embodiment, the color restoration module 133a serves as a generator of the generative adversarial neural network, and the evaluation module 133b acts as a discriminator of the generative adversarial neural network.

14 is a diagram schematically illustrating the structure of an artificial neural network of the luminance restoration unit 131, the color restoration module 133a, and the evaluation module 133b according to an embodiment of the present invention.

14 (a) shows the structure of the artificial neural network of the luminance restoration unit 131 according to an embodiment of the present invention.

In an embodiment of the present invention, the luminance restoration unit 131 may be configured as a stacked convolutional neural network. In more detail, it may be composed of a stacked 5-layer convolutional neural network. The first three layers of the convolutional neural network have a depth of 64, and the next two layers have a depth of 128. Each layer has a kernel size of 3 x 3, and ReLU can be used as an activation function. The final layer of the convolutional neural network may have an output of H x W x 1. In this case, H and W mean the height and width of the first image 1, respectively.

14 (b) shows the structure of the artificial neural network of the luminance restoration module 133a according to an embodiment of the present invention.

In an embodiment of the present invention, the color restoration module 133a may be configured as a high-density U-net. High-density U-net is a combination of high-density convolutional neural network and U-net, which utilizes high-density blocks to extract and propagate features through densely connected layers while maintaining the overall architectural structure of U-net. The high-density U-net takes the second image 2 and the luminance restored image 4 as inputs. The second image 2 and the luminance restored image 4 are processed through a first convolutional layer Conv1 having a kernel size of 3 x 3, a depth of 15, and using ReLU as an activation function. Also, the boundary image 5 is connected to the output of the first convolutional layer. After the first convolutional layer, three high-density blocks (Dense1-3) and three transition blocks (Tran1-3) are alternately located. The output of the first convolutional layer Conv1 is supplied as continuous high-density blocks Dense1-3 in a feed-forward manner, and each high-density block Dense1-3 has a plurality of kernel sizes of 3 x 3 and a stride of 1. has a convolutional layer of Each convolutional layer is normalized by batch normalization, and ReLU is used as the activation function. The output sizes of the three high-density blocks are 64, 128 and 256, respectively. The output of the high-density block passes through each conversion block Dense1-3. Each conversion block Dense1-3 is composed of a convolution layer and a downsampling layer. The kernel size of the convolutional layer of the conversion block (Dense1-3) is 3 x 3, the stride is 1, and ReLU is used as the activation function. The downsampling layer performs a convolution operation with a stride of 2 at the same kernel size. The third conversion block Dense3 is followed by three convolution blocks Conv2-4. The second convolutional block (Conv2) has two more convolutional layers, and the third convolutional block (Conv3) consists of five extended convolutional layers (extension = 2) for high-level feature extraction, and the fourth convolutional block (Conv3) A convolution block has a single convolutional layer. Each layer of the three convolution blocks Conv2-4 has a kernel size of 3 x 3, a stride of 1, and a depth of 512. Also, this layer is normalized by batch normalization and uses ReLU as the activation function. The fourth convolution block (Conv4) is followed by three upsampling blocks (Conv5-7). Each upsampling block Conv5 - 7 is composed of a 2x2 upsampling layer and a convolutional layer having a kernel size of 3x3 following the upsampling layer. The upsampling block (Conv5-7) is also normalized by batch normalization and uses ReLU as an activation function. The output sizes of the three upsampling blocks Conv5-7 are 256, 128, and 64, respectively. Each of the three upsampling blocks Conv5-7 has a separate skip connection with the high-density block Dense1-3, respectively. By such a skip connection, the network can propagate low-level features, thereby contributing to the reconstruction of the final image. Finally, the eighth convolutional layer Conv8 is positioned. The output size of the eighth convolutional layer Conv8 is H x W x 3. The final merged image uses tanh as the activation function.

14 (c) shows the structure of the artificial neural network of the evaluation module 133b according to an embodiment of the present invention.

In an embodiment of the present invention, the evaluation module 133b may be configured as a stacked convolutional neural network. In more detail, the stacked convolutional neural network receives a second image, a luminance restored image, a boundary image, and a merged image. The first three convolutional layers of the stacked convolutional neural network have a depth of 64, and the next convolutional layer has a depth of 128. The kernel of each convolutional layer has a size of 3 x 3 and a stride of 1. All convolutional layers are normalized by batch normalization, and ReLU is used as the activation function. The last layer of the stacked convolutional neural network performs flattening and uses a sigmoid function as an activation function.

In an embodiment of the present invention, each artificial neural network of the luminance restoration unit 131 and the color restoration unit 133 is independently trained. As described above, the artificial neural network of the luminance restoration unit and the artificial neural network of the color restoration unit are each trained, so that learning can be efficiently performed using learning data suitable for each artificial neural network.

15 is a view exemplarily showing a first image, a second image, a luminance restored image, and a boundary image according to an embodiment of the present invention.

15A shows a first image 1 according to an embodiment of the present invention. The first image 1 does not include color information but only brightness information, and the pixel corresponding to the color pixel 2100 does not have brightness information, so it is expressed like a black hole.

15 (b) shows a second image 2 according to an embodiment of the present invention. In this second image 2, color information is included only in pixels corresponding to the color pixels 2100, and color information is not present in the pixels corresponding to the white pixels 2200, and thus is expressed as black.

Figure 15 (c) shows a luminance restored image 4 according to an embodiment of the present invention. The luminance restoration image 4 is filled by the luminance restoration unit 131 deriving and filling the brightness information of the pixel corresponding to the color pixel 2100 of the first image 1 shown in FIG. 15A. can be created by

15 (d) shows a boundary image 5 according to an embodiment of the present invention. The boundary image 5 may be generated by processing the luminance restoration image 4 in the boundary extraction unit 132 .

16 is a diagram schematically illustrating a learning step of an image merging unit according to an embodiment of the present invention.

FIG. 16A illustrates a process in which learning of the evaluation module 133b is performed. Referring to (a) of FIG. 16 , the evaluation module 133b according to an embodiment of the present invention receives a plurality of pre-stored learning merged images, evaluates the authenticity of the learning merged images, and based on the evaluation results, Learning takes place through feedback. The learning merged image may include a learning truth merged image and a learning false merged image. The evaluation module 133b receives such a learning merged image, evaluates the authenticity of the learning merged image, derives an evaluation result, and feeds back the evaluation result to learn the artificial neural network of the evaluation module 133b. The evaluation capability of the evaluation module 133b may be improved.

16 (b) shows a process in which learning of the color restoration module 133a is performed. Referring to FIG. 16 (b), the color restoration module 133a according to an embodiment of the present invention receives a plurality of pre-stored learning second images, learning luminance restoration images, and learning boundary image sets, and learning merged images. , and the evaluation module 133b evaluates the generated learning merged image, and feedback is performed based on the evaluation result to perform learning. That is, the color restoration module 133a receives an image set for learning and generates a learning merged image, and the evaluation module 133b feeds back an evaluation result of evaluating whether the learning merged image is artificial to restore the color. The generation capability of the module 133a may be improved. At this time, the color restoration module 133a performs learning in a direction in which the learned merged image can be determined as a true merged image in the evaluation module 133b to generate a merged image close to the authentic and false merged image. ability can be improved.

17 is a diagram schematically illustrating each step of an image processing method according to an embodiment of the present invention.

Referring to FIG. 17 , an image processing method according to an embodiment of the present invention is an image processing method that receives image data from an electrically connected image sensor and outputs a merged image, and is sensed by a plurality of white pixels of the image sensor. A first image acquisition step (S110) of receiving a first image generated on the basis of the received information; a second image acquisition step (S120) of receiving a second image generated based on information sensed by a plurality of color pixels of the image sensor; an image merging step (S200) of generating a merged image based on the first image and the second image by the learned artificial neural network; and a merged image output step (S300) of outputting the merged image to the outside; includes

The first image acquisition step S110 performs the same operation as that of the first image input unit 110 described above. A first image is obtained by receiving information sensed from a sensor corresponding to the white pixel 2100 among the sensors of the connected sensor array 3000 . Such a first image may be configured by receiving only information sensed from a sensor corresponding to the white pixel 2100, or after receiving information sensed from all sensors of the sensor array 3000, the white pixel It may be configured by selecting information sensed from a sensor corresponding to 2100 .

The second image acquisition step S120 performs the same operation as that of the second image input unit 120 described above. A second image is obtained by receiving information sensed from a sensor corresponding to the color pixel 2200 among the sensors of the connected sensor array 3000 . Such a second image may be configured by receiving only information sensed from a sensor corresponding to the color pixel 2200, or after receiving information sensed from all sensors of the sensor array 3000, the color pixel It may be configured by selecting information sensed from a sensor corresponding to 2200 .

In an embodiment of the present invention, the first image acquisition step S110 and the second image acquisition step S120 may be performed simultaneously.

The image merging step (S200) generates a merged image based on the first image and the second image. In an embodiment of the present invention, in the image merging step (S200), the first image and the second image obtained in the first image acquisition step (S110) and the second image acquisition step (S120) are transmitted and received, A merged image is generated by merging the first image and the second image. According to an embodiment of the present invention, the first image is a mono image including only brightness information, and the second image is a color image including color information, and the image is obtained by merging the first image and the second image. In the merging step ( S200 ), a merged image fully equipped with brightness information and color information is generated. Preferably, in the image merging step (S200), the first image and the second image are merged through a learned artificial neural network to generate a merged image.

The merged image output step (S300) outputs the merged image to the outside. In the merged image output step (S300), light is sensed through the image sensing device, and the merged image generated by processing the sensed light information is transmitted to a connected device so that the user can use it. In the step of outputting the merged image ( S300 ), the merged image may be transmitted to an external computing device or the like through the network interface 300 .

18 is a diagram schematically illustrating detailed steps of an image merging step according to an embodiment of the present invention.

Referring to FIG. 18 , the image merging step ( S200 ) according to an embodiment of the present invention is a luminance restoration step of generating a luminance restored image by deriving the luminance of an area in which the color pixel is located based on the first image. (S210); a boundary extraction step of generating a boundary image including color boundary information of the luminance restoration image through the luminance restoration unit (S220); a color restoration step of generating a merged image based on the second image, the luminance restored image, and the boundary image (S230); and an evaluation step of evaluating the merged image generated in the color restoration step (S240); may include.

In the luminance restoration step (S210), a luminance restoration image is generated by deriving the luminance of a region in which the color pixel is located based on the first image. In the luminance restoration step ( S210 ), the luminance restoration image 4 may be generated by receiving the first image 1 , deriving brightness information about the pixel in which the color pixel 2100 is located, and filling the information. Preferably, in the luminance restoration step (S210), the luminance restoration image is generated based on an artificial neural network.

In the boundary extraction step (S220), a boundary image including color boundary information of the luminance restoration image generated through the luminance restoration operation (S210) is generated. Such a boundary image may be obtained through filter processing on the luminance restored image.

In the color restoration step S230, a merged image is generated based on the second image, the luminance restored image, and the boundary image. Preferably, in the color restoration step ( S230 ), the merged image may be generated by restoring color information to the luminance restored image based on the brightness information of the luminance restored image and the color information of the second image. In this case, the color can be more accurately restored by using the boundary image as a guide for restoring color information. Preferably, in the color restoration step (S230), the merged image is generated based on an artificial neural network.

The evaluation step (S240) evaluates the merged image generated in the color restoration step (S230). The evaluation step (S240) evaluates the merged image to determine whether the merged image is an actual image or a generated false image. The evaluation result is fed back to the color restoration step S230 to generate a merged image closer to the actual image in the color restoration step S230. Preferably, in the evaluation step (S240), the merged image is evaluated based on the artificial neural network.

19 is a diagram schematically illustrating a pattern of a color filter array according to an embodiment of the present invention.

In an embodiment of the present invention, the image merging unit 130 suppresses color bleeding through the boundary image 5 and restores the color through a generative adversarial neural network, thereby exhibiting very good color reproduction in the merged image 3 . Accordingly, the color filter array 2000 for obtaining the first image and the second image according to an embodiment of the present invention may have a very low ratio of color pixels 2200 .

19 shows an example of a color filter array 2000 having such a very low ratio of color pixels 2200 .

If pixels are displayed in the same way as in FIG. 5, in the embodiment shown in FIG. 19, (4,4) pixels are R pixels, (17,6) pixels and (5,15) pixels are G pixels, (14, 19) A pixel is a color pixel 2200 of a B pixel, and all other pixels are white pixels 2100 . That is, the color filter array 2000 shown in FIG. 19 includes four color pixels in the color filter array 2000 having a size of 20 x 20 pixels, so that the ratio of color pixels is 1%, and the ratio of white pixels is 99% reaches to

In particular, in the embodiment shown in FIG. 19 , the R pixels, G pixels, and B pixels are not arranged in a predetermined pattern but are randomly arranged in a color filter array 2000 having a size of 20 x 20 pixels.

In one embodiment of the present invention, the color pixels 2200 having a ratio of 1% as described above are also excellent from the first image 1 and the second image 2 obtained by the color filter array 2000 randomly arranged. A quality merged image (3) can be created.

20 is a block diagram illustrating an example of an internal configuration of a computing device according to an embodiment of the present invention.

20, the computing device 11000 includes at least one processor 11100, a memory 11200, a peripheral interface 11300, an input/output subsystem ( It may include at least an I/O subsystem) 11400 , a power circuit 11500 , and a communication circuit 11600 . In this case, the computing device 11000 may correspond to a user terminal or a consultation matching service providing system.

The memory 11200 may include, for example, a high-speed random access memory, a magnetic disk, an SRAM, a DRAM, a ROM, a flash memory, or a non-volatile memory. have. The memory 11200 may include a software module, an instruction set, or other various data necessary for the operation of the computing device 11000 .

In this case, access to the memory 11200 from other components such as the processor 11100 or the peripheral interface 11300 may be controlled by the processor 11100 .

Peripheral interface 11300 may couple input and/or output peripherals of computing device 11000 to processor 11100 and memory 11200 . The processor 11100 may execute a software module or an instruction set stored in the memory 11200 to perform various functions for the computing device 11000 and process data.

The input/output subsystem 11400 may couple various input/output peripherals to the peripheral interface 11300 . For example, the input/output subsystem 11400 may include a controller for coupling peripheral devices such as a monitor, keyboard, mouse, printer, or a touch screen or sensor as needed to the peripheral interface 11300 . According to another aspect, input/output peripherals may be coupled to peripheral interface 11300 without going through input/output subsystem 11400 .

The power circuit 11500 may supply power to all or some of the components of the terminal. For example, the power circuit 11500 may include a power management system, one or more power sources such as batteries or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or a power source. It may include any other components for creation, management, and distribution.

The communication circuit 11600 may enable communication with another computing device using at least one external port.

Alternatively, as described above, if necessary, the communication circuit 11600 may include an RF circuit to transmit and receive an RF signal, also known as an electromagnetic signal, to enable communication with other computing devices.

This embodiment of FIG. 20 is only an example of the computing device 11000 , and the computing device 11000 may omit some components shown in FIG. 20 , or further include additional components not shown in FIG. 20 , or 2 It may have a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor in addition to the components shown in FIG. 20 , and various communication methods (WiFi, 3G, LTE) are provided in the communication circuit 1160 . , Bluetooth, NFC, Zigbee, etc.) may include a circuit for RF communication. Components that may be included in the computing device 11000 may be implemented in hardware, software, or a combination of both hardware and software including an integrated circuit specialized for one or more signal processing or applications.

Methods according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded in a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. The application to which the present invention is applied may be installed in the user terminal through a file provided by the file distribution system. As an example, the file distribution system may include a file transmission unit (not shown) that transmits the file according to a request of the user terminal.

According to an embodiment of the present invention, by sensing light using a color filter array having a very high ratio of white pixels, it is possible to obtain an image of excellent quality in a low-light environment.

According to an embodiment of the present invention, color restoration through an artificial neural network based on color information obtained through a very small proportion of color pixels can exhibit the effect of performing high-quality color restoration.

According to an embodiment of the present invention, by generating a boundary image and using it for color restoration, it is possible to exhibit the effect of reducing color bleeding in the color restoration process.

According to an embodiment of the present invention, by restoring a color through a generative adversarial neural network, it is possible to prevent false colors from being reproduced.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result. Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (14)

An image processing apparatus having one or more processors and one or more memories, receiving image data from an electrically connected image sensor and outputting a merged image,

The image processing device,

a first image input unit receiving a first image generated based on information sensed by a plurality of white pixels of the image sensor;

a second image input unit for receiving a second image generated based on information sensed by a plurality of color pixels of the image sensor; and

an image merging unit including one or more learned artificial neural networks and generating a merged image based on the first image and the second image; including,

The image merging unit,

a luminance restoration unit generating a luminance restoration image by deriving the luminance of a region in which the color pixels are located based on the first image; and

a color restoration unit for generating a merged image based on data including the second image and the luminance restored image; Including, an image processing device.

The method according to claim 1,

The first image includes white pixel information of 90% or more of the total number of pixels of the merged image,

The second image includes color pixel information of 10% or less of the total number of pixels of the merged image.

The method according to claim 1,

The color restoration unit generates the merged image based on the learned artificial neural network, an image processing apparatus.

The method according to claim 1,

The luminance restoration unit and the color restoration unit generate the luminance restoration image and the merged image based on each learned artificial neural network.

The method according to claim 1,

The image merging unit,

a boundary extraction unit generating a boundary image including color boundary information of the luminance restoration image through the luminance restoration unit; further comprising,

The color restoration unit,

An image processing apparatus for generating a merged image based on the second image, the luminance restored image, and the boundary image.

The method according to claim 1,

The color restoration unit,

An image processing device that generates a merged image composed of a generative adversarial network (GAN).

The method according to claim 1,

The color restoration unit,

a color restoration module for generating a merged image based on data including the second image and the luminance restored image; and

an evaluation module for evaluating the merged image generated by the color restoration module; including,

wherein the color restoration module and the evaluation module perform mutual feedback.

8. The method of claim 7,

The evaluation module is

An image processing apparatus in which learning is performed by receiving a plurality of pre-stored learning merged images, evaluating the authenticity of the learning merged images, and performing feedback based on the evaluation results.

8. The method of claim 7,

The color restoration module,

A plurality of pre-stored learning second images, learning luminance restoration images, and learning boundary image sets are input to generate a learning merged image, and the evaluation module performs feedback based on the evaluation result of evaluating the generated learning merged image. Learning is made, an image processing device.

8. The method of claim 7,

The color restoration module,

It is composed of high-density U-net,

The evaluation module is

An image processing device composed of a stacked convolutional neural network.

The method according to claim 1,

The luminance restoration unit,

An image processing device composed of a stacked convolutional neural network.

The method according to claim 1,

The artificial neural network of each of the luminance restoration unit and the color restoration unit is an artificial neural network in which learning is performed independently.

An image processing method that is performed in a computing device having one or more processors and one or more memories, receiving image data from an electrically connected image sensor and outputting a merged image,

The image processing method is

a first image acquisition step of receiving a first image generated based on information sensed by a plurality of white pixels of the image sensor;

a second image acquisition step of receiving a second image generated based on information sensed by a plurality of color pixels of the image sensor; and

an image merging step of generating a merged image based on the first image and the second image by one or more learned artificial neural networks; including,

The image merging step is

a luminance restoration step of generating a luminance restored image by deriving the luminance of a region in which the color pixel is located based on the first image; and

a color restoration step of generating a merged image based on the second image and the luminance restoration image; Including, an image processing method.

A computer-readable medium for implementing an image processing method of receiving image data from an electrically connected image sensor and outputting a merged image, the computer-readable medium comprising instructions for causing a computing device to perform the following steps , the steps of which are:

a first image acquisition step of receiving a first image generated based on information sensed by a plurality of white pixels of the image sensor;

a second image acquisition step of receiving a second image generated based on information sensed by a plurality of color pixels of the image sensor; and

an image merging step of generating a merged image based on the first image and the second image by one or more learned artificial neural networks; including,

The image merging step is

a luminance restoration step of generating a luminance restored image by deriving the luminance of a region in which the color pixel is located based on the first image; and

a color restoration step of generating a merged image based on the second image and the luminance restoration image; A computer-readable medium comprising:

Images