KR20230105295A - Image acquisition apparatus and electronic apparatus including the same - Google Patents

Abstract

The image acquisition device according to the embodiment includes a multispectral image sensor that acquires images of at least four channels based on a second wavelength band of 10 nm to 1000 nm and inputs the images of at least four channels to a pretrained deep learning network to illuminate the image. and a processor for estimating information and performing color conversion on images acquired using the estimated lighting information.

Description

Image acquisition apparatus and electronic apparatus including the same {Image acquisition apparatus and electronic apparatus including the same}

The disclosed embodiments relate to an image acquisition device and an electronic device including the same.

An image sensor is a device that receives light incident from a subject and photoelectrically converts the received light to generate an electrical signal.

For color expression, the image sensor usually uses a color filter consisting of an array of filter elements that selectively transmits red light, green light, and blue light, and after sensing the amount of light transmitted through each filter element, image processing is performed. A color image of the subject is formed.

Since the value sensed by the image sensor is affected by lighting, the color of an image photographed by a camera is also affected by lighting. White balance is a technology that eliminates this effect and allows you to capture the color of an object as much as possible.

In the conventional white balance technology, white balance is performed by taking an RGB image and then analyzing information in the image. Since this method has Gray World Assumption, that is, an assumption that the average value of each R, G, and B channel of the image is the same, or other restrictions, it may not operate properly in situations where such restrictions are not satisfied.

It relates to an image acquisition device and an electronic device including the same.

An image acquisition device according to an embodiment includes a multispectral image sensor that acquires images of at least four channels based on a second wavelength band of 10 nm to 1000 nm; and a processor for estimating lighting information by inputting the images of the at least four channels to a pretrained deep learning network, and performing color conversion on the acquired images using the estimated lighting information.

An electronic device according to another embodiment includes the image capture device.

A control method of an image acquisition device according to another embodiment includes acquiring images of at least four channels based on a second wavelength band of 10 nm to 1000 nm; estimating lighting information by inputting the images of the at least four channels to a pre-learned deep learning network; and performing color conversion on the acquired images using the estimated illumination information.

The image capturing device according to the embodiment may perform AWB more effective than using an RGB color image by using a multispectral image sensor to minimize an overlapping effect between wavelengths.

In addition, an AWB algorithm that reflects not only human visual characteristics but also perception characteristics can be derived.

The above-described image capture device may be employed in various electronic devices.

1A and 1B are block diagrams showing a schematic structure of an image capture device according to an embodiment.
2 is a detailed block diagram of the image capture device shown in FIG. 1A.
3A and 3B are exemplary diagrams of images acquired by a multispectral image sensor according to an exemplary embodiment.
4 is a schematic diagram of a deep learning network for lighting estimation according to another embodiment.
5 is a schematic diagram of a deep learning network for estimating light reflectivity according to another embodiment.
6 is a conceptual diagram showing a schematic structure of the image capture device shown in FIG. 1B.
FIG. 7 is a diagram showing circuit configurations of a multispectral image sensor and an image sensor included in the image capture device shown in FIG. 1B.
8 is a diagram illustrating a wavelength spectrum by an image sensor included in an image capture device according to an embodiment.
9 to 11 are views illustrating exemplary pixel arrangements of an image sensor included in an image capture device according to an embodiment.
12 is a diagram illustrating a wavelength spectrum by a multispectral image sensor included in an image capture device according to an embodiment.
13 to 15 are diagrams illustrating exemplary pixel arrangements of a multispectral image sensor included in an image capture device according to an embodiment.
16 is a flowchart schematically illustrating a control method of an image capture device according to another embodiment.
17 is a block diagram showing a schematic structure of an electronic device according to an embodiment.
FIG. 18 is a block diagram schematically illustrating a camera module included in the electronic device of FIG. 15 .
19 to 28 show various examples of an electronic device to which an image capture device according to an embodiment is applied.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The described embodiments are merely illustrative, and various modifications are possible from these embodiments. In the following drawings, the same reference numerals denote the same components, and the size of each component in the drawings may be exaggerated for clarity and convenience of description.

Hereinafter, what is described as "above" or "above" may include not only what is directly on top of contact but also what is on top of non-contact.

The terms first, second, etc. may be used to describe various components, but are used only for the purpose of distinguishing one component from another. These terms are not intended to limit the difference in material or structure of the components.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In addition, when a part "includes" a certain component, it means that it may further include other components without excluding other components unless otherwise stated.

In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. .

The use of the term “above” and similar denoting terms may correspond to both singular and plural.

Steps comprising a method may be performed in any suitable order, unless expressly stated that they must be performed in the order described. In addition, the use of all exemplary terms (for example, etc.) is simply for explaining technical ideas in detail, and the scope of rights is not limited due to these terms unless limited by claims.

In general, a value sensed by a camera can be expressed as a product of illumination, color of an object, and camera response as shown in Equation 1 below.

[Equation 1]

는 센싱된 값,

,

,

은 각각 조명, 물체의 표면반사(surface reflectance) 및 카메라 반응에 대한 스펙트럼

의 함수이다. 센싱된 값은 조명에 의해 영향을 받기 때문에 카메라로 촬영된 영상의 컬러도 조명에 의해 영향을 받게 된다. 이러한 영향을 없애고 최대한 물체 고유의 색을 촬영할 수 있도록 하는 것이 화이트밸런스이다.here

is the sensed value,

,

,

is the spectrum for illumination, object surface reflectance, and camera response, respectively.

is a function of Since the sensed value is affected by lighting, the color of an image photographed by a camera is also affected by lighting. White balance is to eliminate this effect and to capture the color of an object as much as possible.

AWB is applied to most camera images for preprocessing and renders object-specific colors in images, which is useful from a photographic point of view and can be applied to image processing such as object recognition, scene understanding, color enhancement, and computer vision research. In a recent study, an incorrect WB setting works as an adversarial attack model for deep neural networks, so effective AWB is important for stable recognition performance in post-processing and applications using deep learning. It is also applied to color adjustment and aesthetic image processing, which are easily recognized by humans, such as human skin color, and have a considerably large effect on the preference of the entire image.

The basic principle of the method for realizing AWB of digital images is to remove the effect of lighting by applying a linear transformation using a 3x3 lighting correction matrix in the camera's RGB color space after estimating lighting components. However, it is difficult to decompose the reflectance and illumination components in the RGB 3-dimensional color space due to overlapping wavelength information in the RGB sensor, and when the illumination is corrected using a diagonal matrix, there is a limit to perfect color correction for colors other than achromatic colors. there is

The image acquisition device according to the embodiment can minimize the overlapping effect between wavelengths by using a multispectral image to represent lighting components in a diagonal matrix, thereby reducing the amount of AWB calculation and more accurately expressing the unique color of an object. there is. In particular, a method for modeling and learning base space modeling based on a low-dimensional determinant, which was not effectively performed in the existing model-based lighting compensation technique, is provided through deep learning.

1A is a block diagram showing a schematic structure of an image capture device according to an embodiment.

Referring to FIG. 1A , an image capture device includes a multispectral image sensor 100 and a processor 500 .

The multispectral image sensor 100 acquires images of at least four channels based on a wavelength band of 10 nm to 1000 nm. Also, the multispectral image sensor 100 may generate images of 16 channels within a wavelength range of 10 nm to 1000 nm, or may generate images of 31 channels by interpolating images of 16 channels. However, the number of channels that can be acquired or generated by the multispectral image sensor 100 is not limited to 4, 16, or 31, of course.

The processor 500 estimates lighting information by inputting images of at least four channels obtained from the multispectral image sensor 100 to a pre-learned deep learning network, and uses the estimated lighting information to output the obtained images. color conversion is performed. The input image input to the deep learning network, learning, output, and image restoration of the deep learning network will be described later.

Figure 1b is a block diagram showing a schematic structure of an image capture device according to an embodiment.

Referring to FIG. 1B , the image capture device includes a multispectral image sensor 100 , an image sensor 200 , and a processor 500 . An image capture device according to an embodiment performs white balance on an image captured using a plurality of image sensors. The image sensor 200 acquires a first image of a first wavelength band. The second wavelength band may include and be wider than the first wavelength band. The multispectral image sensor 100 acquires a second image of a second wavelength band. Here, the image sensor 200 may be an RGB image sensor, and the multispectral image sensor 100 may be a multispectral image (MSI) sensor. An RGB image sensor has an R channel, a G channel and a B channel. The MSI sensor has a larger number of channels than the RGB sensor, so it senses light in more wavelength bands. In an embodiment, the multispectral image sensor 100 may obtain images of four or more channels based on the second wavelength band of 10 nm to 1000 nm. Also, the multispectral image sensor 100 may generate images of 16 channels within a wavelength range of 10 nm to 1000 nm, or may generate images of 31 channels by interpolating images of 16 channels.

The image sensor 200 is a sensor employed in a general RGB camera, and may be a CMOS image sensor using a Bayer color filter array. The first image IM1 acquired by the image sensor 200 may be an RGB image based on red, green, and blue.

The multispectral image sensor 100 is a sensor that senses light of more types of wavelengths than the image sensor 200 . The multispectral image sensor 100 may use, for example, 16 channels, or may use 31 channels, or any other number of channels. The bandwidth of each channel is set to be narrower than the R, G, and B bands, and the total bandwidth of the sum of the bandwidths of all channels includes the RGB bandwidth, that is, the visible light bandwidth, and may be wider than this. For example, it may have a bandwidth of 10 nm to 1000 nm. The second image IM2 obtained by the multispectral image sensor 100 may be a multispectral or hyperspectral image, and may be a wavelength band wider than the RGB wavelength band, for example, the visible ray band. It may be a wavelength-based image obtained by dividing an ultraviolet or infrared wavelength band, which is a wider wavelength band, into 16 or more channels. The second image IM2 may be an image obtained by utilizing all available channels of the multispectral image sensor 100 or may be an image obtained by selecting a specific channel. The spatial resolution of the second image IM2 may be lower than that of the first image IM1, but is not limited thereto.

In an embodiment, the image sensor 200 may be an RGB sensor. In this case, the RGB sensor may be a CMOS image sensor. The RGB sensor may generate images of three channels by sensing spectrums representing R, G, and B using a Bayer color filter array. Also, it goes without saying that other types of color filter arrays may be used. The MSI sensor senses and displays light of a different wavelength than the RGB sensor. The MSI sensor is characterized by sensing light of more types of wavelengths by having a larger number of channels. In a specific example, the number of channels may use 16 channels. In another example, 31 channels may be used. Each channel can adjust a band through which light passes, a transmission amount, and a bandwidth so as to sense light of a desired band. A total bandwidth formed by summing the bandwidths of all channels includes the bandwidth of a conventional RGB sensor and may be wider than that. Sensing spectrums or wavelength bands of the RGB sensor and the MSI sensor will be described later with reference to FIGS. 8 and 12 .

The processor 500 estimates lighting information by inputting images of four or more channels obtained from the multispectral image sensor 100 to a pre-trained deep learning network. The processor 500 performs color conversion on an image using lighting information output through a deep learning network. Here, the lighting information may be a lighting vector corresponding to the intensity of a light spectrum, an XYZ vector representing a color of lighting, an RGB vector representing a color of lighting, a color temperature of lighting, or an index value representing previously stored lighting information.

Also, the processor 500 may input images of 16 channels or images of 31 channels obtained from the multispectral image sensor 100 to the deep learning network.

Referring to FIGS. 3A and 3B , a RAW image acquired by a multispectral image sensor and an image after demosaicing are shown. In the RAW image shown in FIG. 3A, one small square represents one pixel. Numbers in squares represent channel numbers. As shown, in the RAW image, pixels corresponding to different channels, for example, 16 channels are mixed. As shown in FIG. 3B , an image of each channel is generated by collecting pixels of the same channel through demosaicing. As shown in FIGS. 3A and 3B , the processor 500 may estimate lighting information by inputting the generated images to a deep learning network.

The processor 500 may perform color conversion, AWB, or color correction using the estimated illumination. In this case, an object of AWB or color correction may be an image obtained from the image sensor 200 . The image sensor 200 and the multispectral image sensor 100 capture the same scene and correct the image captured by the image sensor 200 using lighting information estimated from the multispectral image sensor 100. can Also, as shown in FIG. 1A , an image captured by the multispectral image sensor 100 may be corrected. For example, among images captured by the multispectral image sensor 100, images corresponding to channels R, G, and B are set as original images, and images of 4 or more channels, 16 channels, or 31 channels are used as original images. Lighting information may be estimated using the image, and the estimated lighting information may be reflected in the original image.

2 is a detailed block diagram of the image capture device shown in FIG. 1A. Referring to FIG. 2 , for explaining a function of estimating lighting information, a multispectral image sensor 100 and an lighting information estimating device 500 are included. The lighting information estimator 500 may be some functional modules of the processor 500 shown in FIGS. 1A and 1B. Descriptions of the same parts as those of FIGS. 1A and 1B will be omitted, and the lighting information estimating device 500 will be mainly described.

The lighting information estimator 500 includes an input unit 510, a deep learning network 520, and an output unit 530. The illumination information estimator 500 may receive images of four or more channels in a wavelength range of 10 nm to 1000 nm from the multispectral image sensor 100 and output an illumination spectrum for each channel.

The input unit 510 inputs images of 4 or more channels to the deep learning network. Here, the input may consist of one or more combinations of the following. The input may be a multi-channel image acquired by the multispectral image sensor 100 . For example, if 16 channels are acquired, all of the 16 channels or a partial channel of the 16 channels can be input. Alternatively, a larger number of channel information may be interpolated and generated using 16 channels, and then input. For example, 31 channels may be generated by interpolating 16 channel images obtained from the multispectral image sensor 100, and images of the 31 channels may be input.

When inputting a channel image, the input unit 510 may sample and input the wavelength band at a specific interval. Here, the interval may be a uniform or uniform interval. Also, when inputting a channel image, the input unit 510 may normalize and input the channel image.

The input image may use part or all of the captured image. Also, the input image may be scaled. Alternatively, both an image without scaling and an image with scaling may be used. In addition, it goes without saying that images undergoing other conversions may be used.

The deep learning network 520 estimates lighting information from an input image. The deep learning network may include a convolution layer, a ReLU layer as an activation function, a maxpooling layer, and a fully-connected layer. The deep learning network can learn by comparing lighting information estimated for each of the input images with angular error values of ground truth lighting. The deep learning network 520 will be described later with reference to FIG. 4 .

The output unit 530 may output the lighting information estimated from the deep learning network 520 as a vector having an intensity value of the lighting spectrum of each channel. For example, a 31-dimensional vector can be output. In this case, the element value of the vector may record the intensity value corresponding to each channel. The output of the network may be an index value representing already stored lighting information. The output of the network may be an illumination vector. In this case, the lighting vector may be a multi-dimensional vector. For example, in the case of a 16-channel multispectral image sensor, a 16-dimensional vector can be output. Alternatively, it may be a 3D vector representing XYZ values on the XYZ color coordinates. Alternatively, it may be a 3D vector representing RGB values on RGB color coordinates. Alternatively, it may be a color temperature value or an index value representing the color temperature of lighting.

The output unit 530 may output an image instead of the above-described output. In this case, the (x, y) position of the image may record lighting information at each pixel position.

In an embodiment, the lighting information estimator 500 or the processor 500 may output a corrected image by reflecting lighting information using a deep learning network or a generative network. In this case, the image to be corrected may be an image captured using the multispectral image sensor 100 . Also, it may be an image selectively photographed using the image sensor 200 . If the output of the deep learning network is the light spectrum, this information is used to calculate the light vector in the color space of the image to be corrected, and AWB can be performed by dividing the value of each color pixel of the image by the value of each element of the vector. . For example, when AWB is performed on an RGB image, an RGB vector [r, g, b] is calculated from the light spectrum, and each pixel value [Ri, Gi, Bi] of the image is calculated as [Ri/r, Gi /g, Bi/b]. Alternatively, [Ri', Gi', Bi']' = M [Ri, Gi, Bi]', M corrects the image by using a 3x3 transformation matrix using a conversion equation considering cross-channel transformation can do. In addition, when the input is normalized, it may be restored by multiplying the scale value. When the wavelength bands of the channel constituting the input signal and the channel of the output signal are different, interpolation may be performed from the channel of the output signal to restore the signal.

4 is a schematic diagram of a deep learning network for lighting estimation according to another embodiment.

Referring to FIG. 4 , the deep learning network 410 receives multispectral images 400 and outputs illumination information 430 (Lestimation) through a fully connected layer 420. Referring to FIG. 4, this is a process of calculating a 31-dimensional output vector for 31 input channels.

을 구한 뒤에 관측 영상으로부터

을 구한다. 그리고

에 표준광원스펙트럼을 곱하여 AWB 영상을 복원할 수 있다. The deep learning network 410 (hereinafter referred to as CCNet) shown in FIG. 4 may be composed of three convolution layers followed by a Rectified Linear Unit function (hereinafter ReLU) as an activation function and Max pooling. The output of the AWB image is estimated from CCNet as shown in Equation 2 below.

After obtaining , from the observation image

save and

The AWB image can be restored by multiplying by the standard light source spectrum.

[Equation 2]

Using the supervised learning method of the deep learning network, the parameters of the network can be optimized through evaluation between the correct lighting spectrum vector for the input multi-spectral image and the predicted lighting spectrum vector output from the network. At this time, the network aims to minimize the difference (error) between the actual value and the predicted value of the network. Regression can be used to infer the lighting spectrum vector.

The loss function used for learning evaluation may use one or more of Angular Loss, L1 Loss (MAE Loss), and L2 Loss (MSE Loss).

Using the angular loss (Angular Loss, Cosine Similarity Loss(π)), we aim to make the network better learn the schematic shape of the lighting spectrum. In light estimation, since the size of light indicates relative intensity, it is important to learn the relative difference between the light values for each channel rather than the absolute size of light. Since each loss considers only a difference in relative orientation between vectors, not the magnitude of the vectors themselves, it is suitable for considering the above-described characteristics of lighting. Therefore, the learning efficiency of the deep learning network can be increased by using each loss, which considers only the direction rather than the size of the vector, for backpropagation.

In an embodiment, the deep learning network uses a loss function including the aforementioned angular error, L1 loss, and L2 loss. In addition to angular loss, L1 Loss (MAE) and L2 Loss (MSE), which are generally one of the loss functions commonly used in regression models, can be used together.

The angular loss can be defined as in Equation 3 below.

[Equation 3]

The L1 loss can be defined as in Equation 4 below.

[Equation 4]

The L2 loss can be defined as in Equation 5 below.

[Equation 5]

는 입력 영상의 Ground Truth인 조명 정보이고,

은 출력 영상의 조명 정보이다. here,

is the lighting information that is the ground truth of the input image,

is the illumination information of the output image.

In an embodiment, synthetic data samples may be used to increase the size of a dataset for training. The synthetic data sample is configured as follows.

The reflectivity map R can be obtained by one or more of the following methods. Get the ground-truth value. A reflectivity map (R) is obtained by estimating from an image acquired under controlled lighting or sunlight in a daytime environment.

The synthetic light is a random light spectrum, and can be calculated as shown in Equation 6 below.

[Equation 6]

,

,

,

는 각각 백열조명, 주간조명, LED조명, 형광조명의 조명 스페트럼이다. 또한, 일정 랜덤 노이즈를 추가할 수 있다. Here, a1, a2, a3, a4 are randomly generated weight values,

,

,

,

is the lighting spectrum of incandescent lighting, daytime lighting, LED lighting, and fluorescent lighting, respectively. In addition, a certain random noise can be added.

A composite image may be obtained by multiplying the composite illumination calculated using Equation 6 and the reflectivity map R.

An output illumination vector can be obtained after convolution operation by inputting an input image to CCNet shown in FIG. 4 . In this case, one network estimates the lighting by looking at all the input images, so this case can be referred to as global lighting estimation. In addition to global illumination estimation, a local illumination estimation method can also be used as follows.

The local illumination estimation method first cuts and inputs the input image in patch units. Selectively input patches when cutting and inputting in patch units. For example, a specific object, color histogram, can only be input into a lighting estimation network if it falls into a patch. All of the locally obtained light spectra are added together to form the final resultant light. Alternatively, the resulting illumination can be constructed by selectively summing locally obtained illumination spectra. Alternatively, a weighted sum of locally obtained light spectra can be used to construct the final resultant light. Alternatively, band-pass filtering may be performed so that only a part of the locally obtained light spectrum remains, and then the final resultant light may be configured by summing the remaining light spectra.

Also, optionally, the global illumination estimation result and the local illumination estimation result may be used together.

The global lighting estimation result is added together with the locally obtained lighting spectrum to form the final lighting result. Alternatively, the final lighting can be constructed by selectively summing the global lighting estimation result with the locally obtained lighting spectrum. Alternatively, the final result lighting can be constructed by weighting the global lighting estimation result with the lighting spectrum obtained locally. Alternatively, the global lighting estimation result is band-pass filtered so that only the locally acquired lighting spectrum and a partial spectrum remain. And it can also be combined to configure the final result lighting.

In addition, both the global area and the local area of the image can be viewed using the kernel size of multiple networks and the downscaling factor of the input image, and the final lighting estimation result can be calculated.

5 is a schematic diagram of a deep learning network for estimating light reflectivity according to another embodiment. In an embodiment, since the observed spectrum of an image can be modeled as a combination of an illumination spectrum and a reflectivity map, AWB may be performed by estimating the reflectivity map together.

Referring to FIG. 5 , by adding a separate deep learning network for estimating the lower albedo map to the CCNET shown in FIG. 4 , the albedo map component can be estimated together.

At this time, the network can be configured to estimate the reflectivity map (R), but instead, the coefficient (C) is calculated from the network output from the relationship of R = BC using the transformation matrix (B) obtained in advance, and then the estimated reflectivity map (BC") is obtained and the estimated illumination vector (L") is multiplied to output an image (D"). Referring to FIG. 5, B can be obtained from principal component analysis or learned from a separate network.

The deep learning network shown in FIG. 5 can be trained by backpropagation to minimize the difference between the original image (D) and the estimated image (D"). As shown in FIGS. 4 and 5, the illumination component and the reflectance component are two Although a neural network composed of different branches is used, the final output image may be obtained by exchanging or using feature information for each layer so that mutual information can be used in the estimation process.

The reflectivity component shown in FIG. 5 outputs a final value through successive convolution layers, but at this time, the result may be output using additional information such as image segmentation or semantic segmentation, object detection, or a histogram.

6 and 7 relate to a specific configuration of the image capture device shown in FIG. 1B, FIG. 6 is a conceptual diagram showing a schematic structure of the image capture device according to an embodiment, and FIG. 7 is a view of the image capture device according to the embodiment. The image sensor and the circuit configuration of the multispectral image sensor are shown.

The image acquisition device 1000 includes a multispectral image sensor 100 that acquires a first image IM1 based on a first wavelength band of 10 nm to 1000 nm, and a multispectral image sensor 100 that acquires a second image IM2 based on a second wavelength band. The processor 500 generates a third image by signal processing the image sensor 200 and the first image IM1 and the second image IM2. The image capture device 1000 further includes a first memory 300 storing data related to the first image IM1 and a second memory 310 storing data related to the second image IM2. and may further include an image output unit 700 that outputs an image.

The image acquisition device 1000 also includes a first imaging optical system 190 that forms an optical image of the object OBJ in the multispectral image sensor 100 and the object OBJ in the image sensor 200. It may include a second imaging optical system 290 forming an optical image of. The first imaging optical system 190 and the second imaging optical system 290 are illustrated as including one lens, but this is exemplary and not limited thereto. The first imaging optical system 190 and the second imaging optical system 290 may be configured to have the same focal length and the same angle of view. In this case, the first image IM1 and the second imaging optical system 290 are formed to form the third image IM3. A process of matching the two images IM2 may be easier. However, it is not limited thereto.

The multispectral image sensor 100 includes a first pixel array PA1 , wherein the first pixel array PA1 includes a first sensor layer 110 in which a plurality of first sensing elements are arrayed, and the first sensor layer and a spectroscopic filter 120 disposed on 110. The spectral filter 120 includes a plurality of filter groups, and each of the plurality of filter groups may include a plurality of unit filters having different transmission wavelength bands. The spectral filter 120 may be configured to subdivide and filter a wavelength band wider than the color filter 220 , eg, a wavelength range of ultraviolet to infrared wavelengths than the color filter 220 . A first micro-lens array 130 may be disposed on the first pixel array PA1. An example of a pixel arrangement applied to the first pixel array PA1 will be described later with reference to FIGS. 13 to 15 .

The image sensor 200 includes a second pixel array PA2 , and the second pixel array PA2 includes a second sensor layer 210 in which a plurality of second sensing elements are arrayed, and the second sensor layer 210 and a color filter 220 disposed thereon. The color filter 220 may include red filters, green filters, and blue filters that are alternately arranged. A second micro lens array 230 may be disposed on the second pixel array PA2 . Various examples of pixel arrangements applied to the second pixel array PA2 will be described later with reference to FIGS. 8-11 .

The first sensor layer 110 and the second sensor layer 210 may include, but are not limited to, a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The first pixel array PA1 and the second pixel array PA2 may be disposed horizontally, eg, spaced apart from each other in the X direction, on the same circuit board SU.

The circuit board SU may include first circuit elements that process signals from the first sensor layer 110 and second circuit elements that process signals from the second sensor layer 210 . However, it is not limited thereto, and the first circuit elements and the second circuit elements may be provided on separate substrates.

The memory 300 in which data for the first image IM1 and the second image IM2 are stored is shown separately from the circuit board SU, but this is exemplary, and circuit elements are included in the circuit board SU. It may be arranged in the same layer or divided into separate layers. The memory 300 may be a line memory that stores an image in units of lines, or may be a frame buffer that stores an entire image. A static random access memory (SRAM) or a dynamic random access memory (DRAM) may be used for the memory 300 .

Various circuit elements necessary for the image capture device 1000 may be integrated and disposed on the circuit board SU. For example, a logic layer including various analog circuits and digital circuits may be provided, and a memory layer storing data may be provided. The logic layer and the memory layer may be composed of different layers or the same layer.

Referring to FIG. 7 , a row decoder 102 , an output circuit 103 , and a timing controller (TC) 101 are connected to the first pixel array PA1 . The row decoder 102 selects one row of the first pixel array PA1 in response to the row address signal output from the timing controller 101 . The output circuit 103 outputs a photo-sensing signal in units of columns from a plurality of pixels arranged along the selected row. To this end, the output circuit 103 may include a column decoder and an analog to digital converter (ADC). For example, the output circuit 103 may include a plurality of ADCs respectively disposed for each column between the column decoder and the first pixel array PA1, or one ADC disposed at an output terminal of the column decoder. The timing controller 101, the row decoder 102, and the output circuit 103 may be implemented as one chip or separate chips. A processor for processing the first image IM1 output through the output circuit 103 may be implemented as a single chip together with the timing controller 101 , the row decoder 102 , and the output circuit 103 .

The row decoder 202, the output circuit 203, and the timing controller (TC) 201 are also connected to the second pixel array PA2, and signals from the second pixel array PA2 are processed similarly to the above. It can be. In addition, a processor for processing the second image IM2 output through the output circuit 203 may be implemented as a single chip together with the timing controller 201, the row decoder 202, and the output circuit 203. there is.

Although the first pixel array PA1 and the second pixel array PA2 have the same size and number of pixels, this is an example for convenience and is not limited thereto.

In operating two different types of sensors, timing control may be required according to different resolutions and output speeds, and the size of a region required for image matching. For example, when reading one image column based on the image sensor 200, the image column of the multispectral image sensor 100 corresponding to that area may be already stored in a buffer or may need to be newly read. there is. Alternatively, operations of the image sensor 200 and the multispectral image sensor 100 may be synchronized using the same synchronization signal. For example, the timing controller 400 may be further provided to transmit a sync signal (sync.) to the image sensor 200 and the multispectral image sensor 100 .

8 shows a wavelength spectrum by an image sensor included in an image capture device according to an embodiment, and FIGS. 9 to 11 show exemplary pixel arrangements of an image sensor included in an image capture device according to an embodiment.

Referring to FIG. 9 , the color filter 220 provided in the second pixel array PA2 includes filters filtering red (R), green (G), and blue (B) wavelength bands in a Bayer pattern. are arranged That is, one unit pixel includes sub-pixels arranged in a 2x2 array, and a plurality of unit pixels are repeatedly arranged two-dimensionally. A red filter and a green filter are disposed on one row of unit pixels, and a green filter and a blue filter are disposed on a second row of unit pixels. The pixel array may be arranged in a manner other than the Bayer pattern.

For example, referring to FIG. 10 , a magenta pixel (M), a cyan pixel (C), a yellow pixel (Y), and a green pixel (G) constitute one unit pixel. Arrangements of the CYGM method of construction are also possible. Also, referring to FIG. 11 , an RGBW method arrangement in which a green pixel (G), a red pixel (R), a blue pixel (B), and a white pixel (W) constitute one unit pixel is also possible. Also, although not shown, unit pixels may have a 3x2 array shape. In addition, pixels of the second pixel array PA2 may be arranged in various ways according to color characteristics of the image sensor 200 .

12 shows a wavelength spectrum by a multispectral image sensor provided in an image capture device according to an embodiment, and FIGS. 13 to 15 show exemplary pixel arrangements of a multispectral image sensor of an image capture device according to another embodiment. see.

Referring to FIG. 13 , the spectral filter 120 provided in the first pixel array PA1 may include a plurality of filter groups 121 arranged in a two-dimensional form. Here, each filter group 121 may include 16 unit filters F1 to F16 arranged in a 4×4 array form.

The first and second unit filters F1 and F2 may have central wavelengths UV1 and UV2 in the ultraviolet region, and the third to fifth unit filters F3 to F5 may have central wavelengths B1 in the blue light region. ~B3). The sixth to eleventh unit filters F6 to F11 may have center wavelengths G1 to G6 of the green light region, and the twelfth to fourteenth unit filters F12 to F14 may have center wavelengths R1 of the red light region. ~R3). Also, the fifteenth and sixteenth unit filters F15 and F16 may have center wavelengths NIR1 and NIR2 in the near infrared region.

14 is a plan view of one filter group 122 provided in the spectral filter 120 according to another example. Referring to FIG. 14 , the filter group 122 may include nine unit filters F1 to F9 arranged in a 3x3 array form. Here, the first and second unit filters F1 and F2 may have central wavelengths UV1 and UV2 of the ultraviolet region, and the fourth, fifth and seventh unit filters F4, F5 and F7 may have blue light. It may have center wavelengths B1 to B3 of the region. The third and sixth unit filters F3 and F6 may have center wavelengths G1 and G2 of the green light region, and the eighth and ninth unit filters F8 and F9 may have center wavelengths R1 of the red light region. , R2).

15 is a plan view of one filter group 123 provided in the spectral filter 120 according to another example. Referring to FIG. 15 , the filter group 123 may include 25 unit filters F1 to F25 arranged in a 5×5 array form. Here, the first to third unit filters F1 to F3 may have central wavelengths UV1 to UV3 in the ultraviolet region, and the sixth, seventh, eighth, eleventh, and twelfth unit filters F6, F7, F8, F11, and F12) may have center wavelengths B1 to B5 of the blue light region. The fourth, fifth, and ninth unit filters F4, F5, and F9 may have center wavelengths G1 to G3 of the green light region, and may have tenth, thirteenth, fourteenth, fifteenth, 18th, and 18th unit filters. The 19 unit filters F10, F13, F14, F15, F18, and F19 may have central wavelengths R1 to R6 of the red light region. The 20th, 23rd, 24th, and 25th unit filters F20 , F23 , F24 , and F25 may have center wavelengths NIR1 to NIR4 in the near infrared region.

The above-described unit filters provided in the spectral filter 120 may have a resonance structure having two reflectors, and a transmitted wavelength band may be determined according to characteristics of the resonance structure. The transmission wavelength band may be adjusted according to the material of the reflector, the material of the dielectric material in the cavity, and the thickness of the cavity. In addition to this, a structure using a grating, a structure using a DBR (distributed bragg reflector), and the like may be applied to the unit filter.

In addition to this, the pixels of the first pixel array PA1 may be arranged in various ways according to color characteristics of the multispectral image sensor 100 .

16 is a flowchart schematically illustrating a control method of an image capture device according to another embodiment.

Referring to FIG. 16 , in step 1600, images of at least four channels based on a second wavelength band of 10 nm to 1000 nm are acquired. Here, the 4 channel images are images obtained from the MSI sensor.

In step 1602, images of at least four channels are input to a pretrained deep learning network. Here, 4 channel images, 16 channel images, or 31 channel images may be input to the deep learning network.

In step 1604, lighting information is estimated. Here, the lighting information may be one of a lighting vector corresponding to the intensity of the lighting spectrum, an XYZ vector representing the color of the lighting, an RGB vector representing the color of the lighting, a color temperature of the lighting, and an index value representing previously stored lighting information. .

In step 1606, color conversion is performed on the obtained image using the estimated lighting information. Here, the obtained image is a 4-channel image generated from the MSI sensor. Also, the obtained image may be an image obtained from an additional image sensor, for example, an RGB image obtained from an RGB image sensor.

The control method of the image capture device according to the embodiment can perform AWB more effectively than using an RGB color image by minimizing an overlapping effect between wavelengths by using a multi-spectral image. In addition, an AWB algorithm that reflects not only human visual characteristics but also perception characteristics can be derived. In addition, when it is difficult to acquire image samples, a learning technique for learning a neural network can be applied. In addition, it is possible to identify preferred image characteristics by deriving a relationship with subjective characteristics that evaluate AWB as well as objective evaluation of image quality. In addition, it can be used as a precise measurement tool by finding out the actual reflector characteristics of an input image under a complex light source. In addition, aesthetic characteristics can be better revealed in colors that are sensitive to human vision in an image, for example, skin color or natural color.

The above-described image capture device 1000 may be employed in various high-performance optical devices or high-performance electronic devices. Such electronic devices include, for example, smart phones, mobile phones, cell phones, personal digital assistants (PDAs), laptops, PCs, various portable devices, home appliances, security cameras, medical cameras, automobiles, and the Internet of Things (IoT). It may be an Internet of Things (IoT) device or other mobile or non-mobile computing device, but is not limited thereto.

In addition to the image capture device 1000, the electronic device may further include a processor that controls image sensors included therein, for example, an application processor (AP), and drives an operating system or an application program through the processor. Thus, a plurality of hardware or software components can be controlled, and various data processing and calculations can be performed. The processor may further include a graphic processing unit (GPU) and/or an image signal processor. When the processor includes an image signal processor, the image (or video) obtained by the image sensor may be stored and/or output using the processor.

17 is a block diagram showing a schematic structure of an electronic device according to an embodiment. Referring to FIG. 17 , in a network environment ED00, an electronic device ED01 communicates with another electronic device ED02 through a first network ED98 (such as a short-distance wireless communication network) or a second network ED99. It is possible to communicate with another electronic device ED04 and/or server ED08 via (a long-distance wireless communication network, etc.). The electronic device ED01 may communicate with the electronic device ED04 through the server ED08. The electronic device (ED01) includes a processor (ED20), a memory (ED30), an input device (ED50), an audio output device (ED55), a display device (ED60), an audio module (ED70), a sensor module (ED76), and an interface (ED77). ), haptic module (ED79), camera module (ED80), power management module (ED88), battery (ED89), communication module (ED90), subscriber identification module (ED96), and/or antenna module (ED97). can In the electronic device ED01, some of these components (such as the display device ED60) may be omitted or other components may be added. Some of these components can be implemented as a single integrated circuit. For example, the sensor module ED76 (fingerprint sensor, iris sensor, illuminance sensor, etc.) may be implemented by being embedded in the display device ED60 (display, etc.). In addition, when the image sensor 1000 includes a spectral function, some functions (color sensor, illuminance sensor) of the sensor module may be implemented in the image sensor 1000 itself instead of a separate sensor module.

The processor ED20 may execute software (program ED40, etc.) to control one or a plurality of other components (hardware, software components, etc.) of the electronic device ED01 connected to the processor ED20, and , various data processing or calculations can be performed. As part of data processing or calculation, processor ED20 loads commands and/or data received from other components (sensor module ED76, communication module ED90, etc.) into volatile memory ED32 and The command and/or data stored in ED32) may be processed, and the resulting data may be stored in non-volatile memory ED34. The processor (ED20) includes a main processor (ED21) (central processing unit, application processor, etc.) and a co-processor (ED23) (graphics processing unit, image signal processor, sensor hub processor, communication processor, etc.) that can operate independently or together with it. can include The auxiliary processor ED23 may use less power than the main processor ED21 and perform specialized functions.

The auxiliary processor ED23 takes the place of the main processor ED21 while the main processor ED21 is inactive (sleep state), or the main processor ED21 is active (application execution state). Together with the processor ED21, functions and/or states related to some of the components of the electronic device ED01 (display device ED60, sensor module ED76, communication module ED90, etc.) may be controlled. can The auxiliary processor ED23 (image signal processor, communication processor, etc.) may be implemented as part of other functionally related components (camera module ED80, communication module ED90, etc.).

The memory ED30 may store various data required by components (processor ED20, sensor module ED76, etc.) of the electronic device ED01. The data may include, for example, input data and/or output data for software (such as the program ED40) and commands related thereto. The memory ED30 may include a volatile memory ED32 and/or a non-volatile memory ED34. The non-volatile memory ED32 may include a built-in memory ED36 fixedly mounted in the electronic device ED01 and a removable external memory ED38.

The program ED40 may be stored as software in the memory ED30 and may include an operating system ED42, middleware ED44, and/or an application ED46.

The input device ED50 may receive a command and/or data to be used by a component (such as the processor ED20) of the electronic device ED01 from an external device (such as a user) of the electronic device ED01. The input device ED50 may include a microphone, mouse, keyboard, and/or a digital pen (stylus pen, etc.).

The sound output device ED55 may output sound signals to the outside of the electronic device ED01. The audio output device ED55 may include a speaker and/or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback, and the receiver can be used to receive an incoming call. The receiver may be incorporated as a part of the speaker or implemented as an independent separate device.

The display device ED60 may visually provide information to the outside of the electronic device ED01. The display device ED60 may include a display, a hologram device, or a projector and a control circuit for controlling the device. The display device ED60 may include a touch circuitry set to detect a touch and/or a sensor circuit (such as a pressure sensor) set to measure the intensity of force generated by the touch.

The audio module ED70 may convert sound into an electrical signal or vice versa. The audio module ED70 acquires sound through the input device ED50, the sound output device ED55, and/or other electronic devices directly or wirelessly connected to the electronic device ED01 (such as the electronic device ED02). ) may output sound through a speaker and/or a headphone.

The sensor module ED76 detects the operating state (power, temperature, etc.) of the electronic device ED01 or the external environmental state (user state, etc.), and generates electrical signals and/or data values corresponding to the detected state. can do. The sensor module ED76 includes a gesture sensor, a gyro sensor, a pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (Infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, and/or an illuminance sensor. May contain sensors.

The interface ED77 may support one or a plurality of specified protocols that may be used to directly or wirelessly connect the electronic device ED01 to another electronic device (such as the electronic device ED02). The interface ED77 may include a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, and/or an audio interface.

The connection terminal ED78 may include a connector through which the electronic device ED01 may be physically connected to another electronic device (such as the electronic device ED02). The connection terminal ED78 may include an HDMI connector, a USB connector, an SD card connector, and/or an audio connector (such as a headphone connector).

The haptic module ED79 can convert electrical signals into mechanical stimuli (vibration, movement, etc.) or electrical stimuli that the user can perceive through tactile or kinesthetic senses. The haptic module ED79 may include a motor, a piezoelectric element, and/or an electrical stimulation device.

The camera module ED80 may capture still images and moving images. The camera module ED80 may include the image capture device 1000 described above, and may include additional lens assembly image signal processors and/or flashes. A lens assembly included in the camera module ED80 may collect light emitted from a subject that is an image capture target.

The power management module ED88 may manage power supplied to the electronic device ED01. The power management module ED88 may be implemented as part of a Power Management Integrated Circuit (PMIC).

The battery ED89 may supply power to components of the electronic device ED01. The battery ED89 may include a non-rechargeable primary cell, a rechargeable secondary cell, and/or a fuel cell.

The communication module ED90 establishes a direct (wired) communication channel and/or a wireless communication channel between the electronic device ED01 and other electronic devices (electronic device ED02, electronic device ED04, server ED08, etc.); And it can support communication through the established communication channel. The communication module ED90 may include one or a plurality of communication processors that operate independently of the processor ED20 (application processor, etc.) and support direct communication and/or wireless communication. The communication module (ED90) includes a wireless communication module (ED92) (cellular communication module, short-range wireless communication module, GNSS (Global Navigation Satellite System, etc.) communication module) and/or a wired communication module (ED94) (LAN (Local Area Network) communication). module, power line communication module, etc.). Among these communication modules, the corresponding communication module is a first network (ED98) (a local area communication network such as Bluetooth, WiFi Direct, or IrDA (Infrared Data Association)) or a second network (ED99) (cellular network, Internet, or computer network (LAN). , WAN, etc.) to communicate with other electronic devices. These various types of communication modules may be integrated into one component (single chip, etc.) or implemented as a plurality of separate components (multiple chips). The wireless communication module ED92 uses the subscriber information (International Mobile Subscriber Identifier (IMSI), etc.) stored in the subscriber identification module ED96 within a communication network such as the first network ED98 and/or the second network ED99. The electronic device (ED01) can be identified and authenticated in .

The antenna module ED97 can transmit or receive signals and/or power to the outside (other electronic devices, etc.). The antenna may include a radiator made of a conductive pattern formed on a substrate (PCB, etc.). The antenna module ED97 may include one or a plurality of antennas. When a plurality of antennas are included, an antenna suitable for a communication method used in a communication network such as the first network ED98 and/or the second network ED99 is selected from among the plurality of antennas by the communication module ED90. can Signals and/or power may be transmitted or received between the communication module ED90 and other electronic devices through the selected antenna. In addition to the antenna, other parts (RFIC, etc.) may be included as part of the antenna module (ED97).

Some of the components are connected to each other through communication methods (bus, GPIO (General Purpose Input and Output), SPI (Serial Peripheral Interface), MIPI (Mobile Industry Processor Interface), etc.) and signal (command, data, etc.) ) are interchangeable.

Commands or data may be transmitted or received between the electronic device ED01 and the external electronic device ED04 through the server ED08 connected to the second network ED99. The other electronic devices ED02 and ED04 may be of the same or different type as the electronic device ED01. All or part of the operations executed in the electronic device ED01 may be executed in one or a plurality of other electronic devices ED02 , ED04 , and ED08 . For example, when the electronic device ED01 needs to perform a certain function or service, instead of executing the function or service itself, one or more other electronic devices perform some or all of the function or service. You can ask to do it. One or a plurality of other electronic devices receiving the request may execute additional functions or services related to the request and deliver the result of the execution to the electronic device ED01. To this end, cloud computing, distributed computing, and/or client-server computing technologies may be used.

FIG. 18 is a block diagram schematically illustrating a camera module ED80 included in the electronic device of FIG. 17 . The camera module ED80 may include the above-described image capture device 1000 or may have a structure modified therefrom. Referring to FIG. 18, the camera module ED80 includes a lens assembly CM10, a flash CM20, an image sensor CM30, an image stabilizer CM40, a memory CM50 (buffer memory, etc.), and/or an image signal. A processor CM60 may be included.

The image sensor CM30 may include the multispectral image sensor 100 and the image sensor 200 provided in the image capture device 1000 described above. The multispectral image sensor 100 and the image sensor 200 may obtain an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly CM10 into an electrical signal. The image sensor 200 may obtain an RGB image, and the multispectral image sensor 100 may acquire a hyperspectral image in a wavelength range of ultraviolet to infrared.

In addition to the image sensor 200 and the multispectral image sensor 100 described above, the image sensor CM30 may include image sensors having different properties, such as other RGB sensors, black and white (BW) sensors, IR sensors, or UV sensors. It may further include one or a plurality of sensors selected from among. Each of the sensors included in the image sensor CM30 may be implemented as a CCD (Charged Coupled Device) sensor and/or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

The lens assembly CM10 may collect light emitted from a subject, which is an image capturing target. The camera module ED80 may include a plurality of lens assemblies CM10. In this case, the camera module ED80 may be a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies CM10 may have the same lens properties (angle of view, focal length, auto focus, F number, optical zoom, etc.) or different lens properties. The lens assembly CM10 may include a wide-angle lens or a telephoto lens.

The lens assembly CM10 may be configured so that two image sensors included in the image sensor CM30 may form an optical image of a subject at the same position and/or focus may be controlled.

The flash CM20 may emit light used to enhance light emitted or reflected from a subject. The flash CM20 may include one or a plurality of light emitting diodes (Red-Green-Blue (RGB) LED, White LED, Infrared LED, Ultraviolet LED, etc.), and/or a Xenon Lamp.

The image stabilizer CM40 moves one or a plurality of lenses or the image sensor 1000 included in the lens assembly CM10 in a specific direction in response to the movement of the camera module ED80 or the electronic device CM01 including the same. Alternatively, the operation characteristics of the image sensor 1000 may be controlled (adjustment of read-out timing, etc.) so that negative effects caused by motion may be compensated for. The image stabilizer CM40 detects the movement of the camera module ED80 or the electronic device ED01 by using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module ED80. can The image stabilizer CM40 may be implemented optically.

The memory CM50 may store some or all data of an image acquired through the image sensor 1000 for a next image processing task. For example, when a plurality of images are acquired at high speed, the acquired original data (Bayer-patterned data, high-resolution data, etc.) is stored in the memory (CM50), only low-resolution images are displayed, and then selected (user selection, etc.) It can be used to cause the original data of the image to be passed to the Image Signal Processor (CM60). The memory CM50 may be integrated into the memory ED30 of the electronic device ED01 or configured as a separate memory operated independently.

The image signal processor CM60 may perform image processing on images acquired through the image sensor CM30 or image data stored in the memory CM50. 1 to 17, by processing a first image (eg, an RGB image) and a second image (eg, an MSI image) obtained by two image sensors included in the image sensor CM30. , a third image on which white balance has been performed may be generated. The configuration of the processor 500 for this may be included in the image signal processor CM60.

Image processing, in addition, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, and/or image compensation (noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening ( sharpening, softening, etc.). The image signal processor CM60 may perform control (exposure time control, read-out timing control, etc.) for components included in the camera module ED80 (image sensor CM30, etc.). The images processed by the image signal processor (CM60) are stored back in the memory (CM50) for further processing or the external components of the camera module (ED80) (memory (ED30), display unit (ED60), electronic unit (ED02)) , the electronic device ED04, the server ED08, etc.). The image signal processor CM60 may be integrated into the processor ED20 or configured as a separate processor that operates independently of the processor ED20. When the image signal processor CM60 is configured as a separate processor from the processor ED20, the image processed by the image signal processor CM60 undergoes additional image processing by the processor ED20 and then displays the display device ED60. can be displayed through

The electronic device ED01 may include a plurality of camera modules ED80 each having a different property or function. In this case, one of the plurality of camera modules ED80 may be a wide-angle camera and the other may be a telephoto camera. Similarly, one of the plurality of camera modules ED80 may be a front camera and the other may be a rear camera.

19 to 28 show various examples of electronic devices to which an image capture device according to an embodiment is applied.

An image acquisition device according to embodiments may include a mobile phone or smart phone 5100m shown in FIG. 19, a tablet or smart tablet 5200 shown in FIG. 20, a digital camera or camcorder 5300 shown in FIG. 21, It can be applied to the notebook computer 5400 shown in FIG. 22 or the television or smart TV 5500 shown in FIG. 23 . For example, the smart phone 5100m or the smart tablet 5200 may include a plurality of high resolution cameras each having a high resolution image sensor. Depth information of subjects in an image may be extracted using high-resolution cameras, out-focusing of the image may be adjusted, or subjects in the image may be automatically identified.

In addition, the image capture device 1000 includes a smart refrigerator 5600 shown in FIG. 24, a security camera 5700 shown in FIG. 25, a robot 5800 shown in FIG. 26, and a medical camera 5900 shown in FIG. ), etc. can be applied. For example, the smart refrigerator 5600 automatically recognizes food in the refrigerator using the image acquisition device 1000, and informs the user of the presence or absence of specific food, the type of food that has been stored or shipped out, etc. through a smartphone. can tell you The security camera 5700 can provide ultra-high resolution images and can recognize objects or people in the images even in a dark environment by using high sensitivity. The robot 5800 may provide a high-resolution image when deployed at a disaster or industrial site to which a person cannot directly approach. The medical camera 5900 can provide high resolution images for diagnosis or surgery and can dynamically adjust the field of view.

Also, the image capture device 1000 may be applied to the vehicle 6000 as shown in FIG. 28 . The vehicle 6000 may include a plurality of vehicle cameras 6010, 6020, 6030, and 6040 disposed in various locations. Each of the vehicle cameras 6010, 6020, 6030, and 6040 may include an image capture device according to an embodiment. The vehicle 6000 may use a plurality of vehicle cameras 6010, 6020, 6030, and 6040 to provide the driver with various information about the interior or surroundings of the vehicle 6000, and automatically recognize objects or people in the image to Information necessary for autonomous driving can be provided.

Although the above-described image capture device and electronic device including the same have been described with reference to the embodiment shown in the drawings, this is only an example, and those skilled in the art can make various modifications and equivalent other implementations therefrom. It will be appreciated that examples are possible. Therefore, the disclosed embodiments should be considered from an illustrative rather than a limiting point of view. The scope of rights is shown in the claims rather than the foregoing description, and all differences within an equivalent scope should be construed as being included in the scope of rights.

1000: image acquisition device
100: multispectral image sensor
110: first sensor layer
120: spectral filter
130: first micro lens array
190: first imaging optical system
200: image sensor
210: second sensor layer
220: color filter
230; 2nd Micro Lens Array
290: second imaging optical system

Claims (20)

a multispectral image sensor that acquires images of at least four channels based on a wavelength band of 10 nm to 1000 nm; and
Image acquisition including a processor for estimating illumination information by inputting the images of the at least four channels to a pre-trained deep learning network, and performing color conversion on the obtained images using the estimated illumination information Device. According to claim 1
The multispectral image sensor,
Acquiring images of 16 channels,
the processor,
An image acquisition device for inputting images of 16 channels obtained from the multispectral image sensor to the deep learning network. According to claim 2,
the processor,
Interpolating the acquired images of 16 channels to generate images of 31 channels, and inputting the generated images of 31 channels to the deep learning network. According to claim 1,
The lighting information,
An image acquisition device that is one of a light vector corresponding to the intensity of a light spectrum, an XYZ vector indicating a color of light, an RGB vector indicating a color of light, a color temperature of light, and an index value indicating previously stored light information. According to claim 1,
the processor,
Images obtained by sampling wavelength bands corresponding to the images of the at least four channels at predetermined intervals or images obtained by normalizing the images of the at least four channels are input to the deep learning network. According to claim 1,
The deep learning network,
An image acquisition device comprising at least three convolutional layers, a rectified linear unit function as an activation function, and a max pooling layer. According to claim 1,
The deep learning network,
A pre-learned, image acquisition device using a loss function including angular error, L1 loss and L2 loss. According to claim 7,
The angular loss is calculated according to Equation 3 below,
[Equation 3]

The L1 loss is calculated according to Equation 4 below,
[Equation 4]

The L2 loss is calculated according to Equation 5 below.
[Equation 5]

(here,

is the lighting information that is the ground truth of the input image,

is the lighting information of the output image) According to claim 1,
the processor,
In order to increase the number of training images of the deep learning network, an image acquisition device for generating a synthesized image obtained by multiplying a pre-calculated composite lighting and a reflectance map. According to claim 9,
The composite illumination is a random illumination spectrum, and is calculated according to Equation 6 below.
[Equation 6]

(Where a1, a2, a3, a4 are randomly generated weight values,

,

,

,

is the lighting spectrum of incandescent lighting, daytime lighting, LED lighting, and fluorescent lighting, respectively) According to claim 9,
The reflectivity map,
An image acquisition device that is calculated based on illumination information in a controlled environment or illumination information in an image acquired under sunlight in a daytime environment. According to claim 1,
Further comprising an image sensor for obtaining an image based on the first wavelength band,
the processor,
Using the estimated lighting information, color conversion is performed on the image obtained from the image sensor. According to claim 1,
The estimated lighting information,
A global lighting estimation device for estimating lighting information based on the entire image input to the deep learning network. According to claim 1,
The estimated lighting information,
Local illumination estimation for cutting an image to be input to the deep learning network in patch units and estimating illumination information based on the input patch. 15. The method of claim 14,
An image acquisition device that selectively inputs an object to the deep learning network based on an object and a color histogram among a plurality of patches. According to claim 1,
The estimated lighting information,
An image acquisition device that is one of a final result value obtained by summing the global illumination estimation result value and the local illumination estimation result value, a final result value obtained by selectively summing the final result value, a final result value obtained by weighted summing, and a result value obtained by performing band pass filtering. According to claim 1,
Further comprising a separate deep learning network for estimating the reflectivity map, the image acquisition device. 18. The method of claim 17,
Acquiring the color-converted resultant image using the estimated lighting information and the estimated reflectivity map. An electronic device comprising the image acquisition device of any one of claims 1 to 18. In the control method of the image acquisition device,
obtaining images of at least four channels based on a second wavelength band of 10 nm to 1000 nm;
estimating lighting information by inputting the images of the at least four channels to a pre-learned deep learning network; and
and performing color conversion on the obtained images using the estimated illumination information.

Images