KR20230174127A - Image sensor correcting crosstalk, operating method of image sensor, and electronic device including image sensor - Google Patents

Abstract

This description relates to image sensors. The image sensor of the present disclosure includes a plurality of pixels, a pixel unit configured to generate an analog signal using the plurality of pixels, and a data conversion unit that receives the analog signal and converts the analog signal into a digital signal; And a processing unit configured to generate image data by performing crosstalk correction and remosaic on the digital signal, and output the image data to an external device, and the processing unit receives image data information from the external device. It is configured to receive, calculate the saturation ratio based on the image data information, and adjust the remosaic settings based on the saturation ratio.

Description

An image sensor that corrects crosstalk, a method of operating the image sensor, and an electronic device including the image sensor

This disclosure relates to electronic devices, and more specifically, to an image sensor that adaptively corrects crosstalk, a method of operating the image sensor, and an electronic device including the image sensor.

An image sensor includes a plurality of pixels, and can generate image data using the plurality of pixels. Image data generated by an image sensor may include various noises and distortions. The image sensor may perform calibration to remove noises and distortions from image data, and output calibrated image data.

Noises and distortions in image data may include crosstalk. Crosstalk may occur due to uneven detection of incident light across the pixels of the image sensor or leakage of detection results between pixels. To correct crosstalk, the image sensor can perform various corrections.

The purpose of the present disclosure is to provide an image sensor capable of correcting crosstalk even when pixel values detected by pixels of the image sensor are saturated, a method of operating the image sensor, and an electronic device including the image sensor. there is.

The pixel unit according to an embodiment of the present disclosure includes a plurality of pixels, and is configured to generate an analog signal using the plurality of pixels, receiving the analog signal, and converting the analog signal into a digital signal. and a processing unit configured to generate image data by performing crosstalk correction and remosaic on the digital signal, and output the image data to an external device, wherein the processing unit receives image data from the external device. It is configured to receive information, calculate a saturation ratio based on the image data information, and adjust the settings of the remosaic based on the saturation ratio.

A method of operating an image sensor including a plurality of pixels includes the steps of the image sensor receiving image data information from an external device, and the image sensor calculating a saturation ratio of image data acquired by the image sensor from the image data information. , and a step of the image sensor adjusting remosaic settings based on the saturation ratio, and the image sensor is configured to perform remosaic based on the remosaic settings for the image acquired by the image sensor.

An electronic device according to an embodiment of the present disclosure includes an image sensor that generates image data, and a processor that generates image data information based on the image data and transmits the image data information to the image sensor, and the image sensor includes a plurality of image sensors. pixels, and configured to calculate a saturation ratio of the plurality of pixels from the image data information, and correct crosstalk of the plurality of pixels based on the saturation ratio based on one of the first mode and the second mode, In a first mode, the image sensor is configured to correct crosstalk based on gain and offset, and in a second mode, the image sensor is configured to correct crosstalk based on gain and offset and remosaic settings. It is configured to additionally correct crosstalk by adjusting .

According to embodiments of the present disclosure, as pixel values detected by pixels become saturated, the image sensor may adjust remosaic settings. Accordingly, an image sensor capable of correcting crosstalk even when pixel values are saturated, a method of operating the image sensor, and an electronic device including the image sensor are provided.

1 shows an electronic device according to an embodiment of the present disclosure.
FIG. 2 shows an example of a method of operating the electronic device of FIG. 1.
Figure 3 shows an image sensor according to an embodiment of the present invention.
Figure 4 shows an example of arrangement of pixels of an image sensor.
Figure 5 shows an example of a cross-sectional view of some of the pixels of a pixel array.
Figure 6 shows an example of pixel values on which remosaic was performed.
Figure 7 shows an example of how the crosstalk correction unit of the processing circuit of the image sensor corrects crosstalk.
FIG. 8 is a diagram for explaining an example in which an application processor generates image data information (IDI) from second image data.
Figure 9 shows an example of a process in which an application processor transmits image data information to an image sensor.
Figure 10 shows an example of how an image sensor's processing circuit suppresses crosstalk based on the saturation ratio.
Figure 11 shows an example of how the image sensor's processing circuitry calculates the saturation ratio.
Figure 12 shows an example of a method for adjusting remosaic settings according to an embodiment of the present disclosure.
Figure 13 shows an example of a trend in which the image sensor's processing circuit adjusts the remosaic settings according to the saturation ratio.
Figure 14 shows an example of how the processing circuit of the image sensor performs remosaic based on the remosaic setting.
Figure 15 shows another example of how the image sensor's processing circuitry calculates the saturation ratio.
16 is a block diagram of an electronic device including a multi-image sensor.
FIG. 17 is a detailed block diagram of the image sensor of FIG. 16.

Hereinafter, embodiments of the present invention will be described clearly and in detail so that a person skilled in the art can easily practice the present invention. Below, the term 'and/or' is interpreted to include any one of the items listed in connection with the term, and a combination of some or all of the items listed in connection with the term.

Figure 1 shows an electronic device 10 according to an embodiment of the present disclosure. Referring to FIG. 1 , the electronic device 10 may include an image sensor 100 and an application processor 200. By way of example, the electronic device 10 may further include various peripheral devices such as a display device, memory, storage device, and user interface device.

The image sensor 100 may include a pixel unit 110, a data conversion unit 120, and a processing unit 130. The pixel unit 110 may include a plurality of pixels. The pixel unit 110 may generate an analog signal AS using a plurality of pixels. The analog signal AS may include a plurality of signals having different levels (eg, voltage levels or current levels) depending on the intensity of light incident on the pixels of the pixel unit 110.

The data conversion unit 120 may convert the analog signal AS generated by the pixel unit 110 into a digital signal DS. The data conversion unit 120 may output the converted digital signal DS to the processing unit 130. The digital signal DS may include a plurality of digital values converted from the levels of a plurality of signals of the analog signal AS.

The processing unit 130 may receive the digital signal DS from the data conversion unit 120. The processing unit 130 may generate image data (ID) by performing calibration to remove noises and/or distortions from the digital signal (DS). The processing unit 130 may output image data (ID) to the application processor 200. For example, the processing unit 130 may output image data (ID) to the application processor 200 based on MIPI (Mobile Industry Processor Interface) C-PHY or D-PHY.

For example, the processing unit 130 may include a memory (eg, static random access memory) for storing the digital signal DS for calibration. The memory may store the digital signal DS corresponding to pixels in one or more rows of pixels of the pixel unit 110. The capacity of the memory may correspond to the unit of pixels required to perform calibration.

The processing unit 130 may receive image data information (IDI) from the application processor 200. For example, the processing unit 130 may receive image data information (IDI) from the application processor 200 through MIPI C-PHY or D-PHY, or an Inter-Integrated Circuit (I2C) interface. Image data information (IDI) may include information about image data (ID) of the previous frame. The processing unit 130 may select a mode or method for performing calibration based on image data information (IDI).

The processing unit 130 may include a remosaic unit 131, a binning unit 132, a crosstalk correction unit 133, and a saturation management unit 134. The processing unit 130 may selectively activate one of the remosaic unit 131 and the binning unit 132 according to the brightness (or illuminance) corresponding to the digital values of the digital signal DS.

When the brightness (or illuminance) corresponding to the digital values of the digital signal DS is greater than or equal to the reference value, the processing unit 130 may activate the remosaic unit 131. The remosaic unit 131 generates image data (e.g., intermediate image data processed into image data ID) corresponding to a Bayer pattern from the digital signal DS corresponding to the pixel pattern of the pixel unit 110. can do. When the brightness (or illuminance) corresponding to the digital values of the digital signal DS is less than a reference value (eg, a brightness reference value), the processing unit 130 may activate the binning unit 132. The binning unit 132 sums the digital values of the same channel (e.g., the same color) of the digital signal DS corresponding to the pixel patterns of the pixel unit 110 to obtain Bayer pattern image data (e.g., image data). Intermediate image data that is processed into data (ID) can be created.

When the remosaic unit 131 is activated, the crosstalk correction unit 133 may be activated. The crosstalk correction unit 133 can perform crosstalk correction by applying gain and offset to the pixel values of the digital signal (DS) or the image data remosaiced by the remosaic unit 131. there is. For example, the crosstalk correction unit 133 may store the gain and offset in the form of a table, and perform crosstalk correction by referring to the table.

When the remosaic unit 131 is activated, the saturation management unit 134 may be activated. The saturation management unit 134 determines the degree to which the pixel values of the pixels of the pixel unit 110 (for example, the digital signal values of the digital signal DS corresponding to the pixels) are saturated based on the image data information (IDI). The saturation ratio indicated can be calculated. The saturation management unit 134 can adjust the settings for the remosaic unit 131 to perform remosaic based on the saturation ratio. The saturation management unit 134 adjusts the remosaic setting, Crosstalk that occurs in an environment where pixel values of pixels of the pixel unit 110 are saturated can be suppressed.

In Figure 1, the image sensor 100 and the application processor 200 are shown as if in direct communication with each other. However, this is to facilitate the transfer of embodiments of the present disclosure, and other components that mediate or assist in the transfer of image data (ID) between the image sensor 100 and the application processor 200 may be provided.

FIG. 2 shows an example of a method of operating the electronic device 10 of FIG. 1 . Referring to FIGS. 1 and 2 , in step S11, the image sensor 100 may transmit image data (eg, image data (ID) of the first frame) to the application processor 200. In step S12, the application processor 200 may generate image data information (IDI) from the received image data. By way of example, the application processor 200 may display the received image data through a display device or store it in a memory.

In step S13, the application processor 200 may transmit image data information (IDI) to the application processor 200. In step S14, the processing unit 130 of the image sensor 100 may calculate the saturation ratio based on image data information (IDI). In step S15, the processing unit 130 of the image sensor 100 may determine the crosstalk correction mode based on the saturation ratio. Illustratively, the image data information (IDI) may include average value(s) of pixel values or number(s) of saturated pixels.

For example, when the saturation ratio (e.g., the saturation ratio of the image data (ID) of the first frame) is lower than the reference value (e.g., the saturation reference value) or is '0', the image sensor 100 The processing unit 130 corrects the crosstalk of the digital signal DS using only the crosstalk correction unit 133 to produce image data ID (e.g., image data of the second frame following the first frame). (ID)) can be created. When the saturation ratio is greater than or equal to a reference value (e.g., a saturation reference value), the processing unit 130 of the image sensor 100 corrects the crosstalk of the digital signal DS based on the crosstalk correction unit 133. , crosstalk can be additionally corrected by adjusting the remosaic settings of the remosaic unit 131 to generate image data (ID) (for example, image data of the second frame following the first frame).

Exemplarily, the crosstalk correction unit 133 may perform crosstalk correction on pixel values of pixels that are not saturated. The remosaic unit 131 may perform crosstalk correction (or suppression) on all or part of the pixel values of the digital signal DS. The remosaic unit 131's selection of an area to correct (or suppress) crosstalk on the digital signal corresponding to the pixels may be performed regardless of saturation or saturation ratio.

Figure 3 shows a device 300 according to an embodiment of the present invention. By way of example, the device 300 may correspond to the pixel unit 110 and the data conversion unit 120 of the image sensor 100 of FIG. 1 . 1 and 3, the device 300 includes a pixel array 310, a row driver 320, a ramp signal generator 330 (RSG), an analog-to-digital conversion circuit 340, and a memory circuit 350. , an interface circuit 360, and a timing generator 370 (TG).

By way of example, the pixel array 310 may correspond to the pixel unit 110 . The row driver 320, the ramp signal generator 330, the analog-to-digital conversion circuit 340, the memory circuit 350, the interface circuit 360, and the timing generator 370 may correspond to the data conversion unit 120. there is.

The pixel array 310 may include a plurality of pixels (PX) arranged in a matrix form along rows and columns. Each of the plurality of pixels PX may include photo detectors. For example, photo detectors may include photo diodes, photo transistors, photo gates, pinned photo diodes, etc. Each of the plurality of pixels PX may detect light using a photodetector and convert the amount of detected light into an electrical signal, for example, voltage or current.

A color filter array (CFA) and a lens may be stacked on the pixel array 310 . The color filter array may include red (R), green (G), and blue (B) filters. Two or more different color filters may be disposed in the plurality of pixels PX. For example, at least one blue color filter, at least one red color filter, and at least two green color filters may be disposed in the plurality of pixels PX.

The row driver 320 may be connected to each row of pixels PX of the pixel array 310 through the first to mth row lines RL1 to RLm (m is a positive integer). The row driver 320 decodes the address and/or control signal generated by the timing generator 370 to sequentially select the first to mth row lines RL1 to RLm of the pixel array 310, And the selected row line can be driven with a specific voltage. For example, the row driver 320 may drive the selected row line to a voltage suitable for detecting light.

Each of the first to mth row lines RL1 to RLm connected to rows of pixels PX may include two or more lines. Two or more lines, for example, a signal to select (or activate) a pixel's photo detectors, a signal to reset a floating diffusion node, a signal to select a column line, and a signal to control the conversion gain (CG). Various signals, including signals for

The ramp signal generator 330 may generate a ramp signal RS. The ramp signal generator 330 may operate under the control of the timing generator 370. For example, the ramp signal generator 330 may operate under a control signal such as a ramp enable signal or mode signal. In response to the ramp enable signal being activated, the ramp signal generator 330 may generate a ramp signal with a slope set based on the mode signal. For example, the ramp signal generator 330 may generate a ramp signal RS that continuously decreases or increases from the initial level over time.

The analog-to-digital conversion circuit 340 may be connected to each of the columns of pixels PX of the pixel array 310 through the first to nth column lines CL1 to CLn (n is a positive integer). The analog-to-digital conversion circuit 340 may include first to nth analog-to-digital converters (AD1 to ADn) respectively connected to the first to nth column lines (CL1 to CLn). The first to nth analog-to-digital converters AD1 to ADn may commonly receive the ramp signal RS from the ramp signal generator 330.

The first to nth analog-to-digital converters AD1 to ADn may compare the voltages (or currents) of the first to nth column lines CL1 to CLn with the ramp signal RS. A ramp signal is a signal that decreases (or increases) at a constant rate. When the ramp signal RS becomes smaller than the pixel voltages (or pixel currents) of the first to nth column lines CL1 to CLn ( Alternatively, the count value can be latched until it becomes large, and the latched count value can be converted to a digital value and output. That is, the first to nth analog-to-digital converters AD1 to ADn are the magnitudes of voltages (or currents) output from the pixels PX to the first to nth column lines CL1 to CLn ( or quantity) can be output.

Each of the first to nth analog-to-digital converters (AD1 to ADn) may include at least two sub-converters. The sub-converters are commonly connected to the corresponding column line, and can commonly receive the ramp signal (RS). The sub-converters may have the same resolutions or different resolutions. The sub-converters may be activated at different timings to convert the voltage (or current) of the corresponding column line into digital values (or digital signals).

The memory circuit 350 may include first to nth memories M1 to Mn respectively corresponding to the first to nth analog-to-digital converters AD1 to ADn. The first to nth memories (M1 to Mn) store digital values (or digital signals) received from the first to nth analog-to-digital converters (AD1 to ADn), and the stored values (or signals) ) can be transmitted to the processing unit 130 of FIG. 1.

The timing generator 370 (TG) may control the timings at which the device 300 operates. The timing generator 370 controls the timings at which the row driver 320 sequentially selects the first to mth row lines RL1 to RLm, and among the first to mth row lines RL1 to RLm. The timings at which signals are transmitted through two or more lines included in the selected row line can be controlled.

The timing generator 370 may control the timings at which the ramp signal generator 330 generates the ramp signal RS and initializes the ramp signal. The timing generator 370 generates timings at which the first to nth analog-to-digital converters (AD1 to ADn) start counting and comparison and timings to initialize the first to nth analog-to-digital converters (AD1 to ADn). You can control them.

Figure 4 shows an example of arrangement of pixels of the pixel unit 110. Referring to FIGS. 1, 3, and 4, the pixel unit 110 includes a first pixel (PX1) at the top left, a second pixel (PX2) at the top right, a third pixel (PX3) at the bottom left, and a pixel (PX3) at the bottom right. The fourth pixel PX4 may have a pattern that is repeatedly arranged in the form of a two-dimensional plane.

Each of the first pixel (PX1), the second pixel (PX2), the third pixel (PX3), and the fourth pixel (PX4) may include a plurality of subpixels. Exemplarily, each of the first pixel (PX1), the second pixel (PX2), the third pixel (PX3), and the fourth pixel (PX4) is shown as including 9 sub-pixels, but the first pixel (PX1) ), the number of subpixels included in each of the second pixel (PX2), third pixel (PX3), and fourth pixel (PX4) is not limited.

As an example, subpixels of the first pixel PX1 may share a red (R) color filter. Subpixels of the second pixel PX2 may share a green (G) (eg, Gr) color filter. Subpixels of the third pixel PX3 may share a green (G) (eg, Gb) color filter. Subpixels of the fourth pixel PX4 may share a blue (B) color filter.

Figure 5 shows an example of a cross-sectional view of some of the pixels of pixel array 400. Illustratively, the pixel array 400 may correspond to the pixel array 310 of the device 300. Referring to FIGS. 3, 4, and 5, the pixel array 400 is

The pixel array 400 includes micro lenses 410 that respectively correspond to sub-pixels and are configured to focus incident light on the corresponding pixels, a color filter area 420 located below the micro lens 410, and a color filter. An anti-reflection film 425 located below the area 420 and preventing reflection of incident light, photoelectric conversion areas 430 that generate charges based on incident light, and photoelectric conversion areas 430 located below the photoelectric conversion areas 430 and preventing reflection of incident light. It may include sensing circuits 440 that detect charges generated by the conversion regions 430 . The photoelectric conversion regions 430 may be electrically insulated from each other.

In the color filter area 420, the first color filter CF1 may be provided on top of the subpixels of the first pixel PX1. The first color filter CF1 may pass light with a frequency corresponding to red. In the color filter area 420, the second color filter CF2 may be provided on top of the subpixels of the second pixel PX2. The second color filter CF2 may pass light with a frequency corresponding to green. Between the color filters in the color filter area 420, for example, the first color filter CF1 and the second color filter CF2, grids including a metal component and preventing crosstalk of incident light ( GD) may be provided.

Although not shown in FIG. 4, like the first pixel PX1 and the second pixel PX2, a color filter is provided at the top of the subpixels of the third pixel PX3 to pass light with a frequency corresponding to green. , and a color filter that passes light with a frequency corresponding to blue may be provided on top of the subpixels of the fourth pixel PX4.

Between the photoelectric conversion areas 430 of the first pixel (PX1), the second pixel (PX2), the third pixel (PX3), and the fourth pixel (PX4), and the first pixel (PX1) and the second pixel Between the photoelectric conversion areas 430 of each subpixel of (PX2), the third pixel (PX3), and the fourth pixel (PX4), light is blocked from leaking to other adjacent subpixels or pixels. Reflective materials may be provided.

The sensing circuit 440 includes transistors connected to a corresponding column line among the first to nth column lines CL1 to CLn and a corresponding row line among the first to mth row lines RL1 to RLm. can do.

As shown in FIG. 4, the thickness of the color filter CF1 or CF2 may not be uniform. The thickness of the first color filter CF1 corresponding to the subpixels at the center of the first pixel PX1 is greater than the thicknesses of the first color filter CF1 corresponding to the subpixels at the edges of the first pixel PX1. It can be thick. Therefore, even if light of the same brightness (e.g., illuminance) is incident in the same direction (e.g., vertical direction), the pixel value detected by the central subpixel of the first pixel (PX1) is the same as that of the first pixel (PX1). ) may be lower than the detected pixel value of the subpixels at the edge. Additionally, the central subpixel of the first pixel PX1 has neighboring subpixels having color filters of the same color. However, a subpixel at the edge of the first pixel PX1 may have neighboring subpixels having a color filter of the same color and neighboring subpixels having a color filter of a different color. That is, the effect of crosstalk may vary depending on the environment of the subpixels.

When the brightness (eg, illuminance) is lower than the reference value (eg, illuminance reference value), the binning unit 132 of the processing unit 130 (see FIG. 1) may perform binning. For example, the binning unit 132 sums (or averages) the pixel values of each subpixel of the first pixel (PX1), the second pixel (PX2), the third pixel (PX3), and the fourth pixel (PX4). can be calculated). That is, the processing unit 130 generates image data (ID) including pixel values of the pixels of the first pixel (PX1), the second pixel (PX2), the third pixel (PX3), and the fourth pixel (PX4). can do. Since the image data (ID) on which the binning has been performed is based on a Bayer pattern, a separate conversion operation is not required.

When the brightness (eg, illuminance) is greater than or equal to a reference value (eg, illuminance reference value), the remosaic unit 131 of the processing unit 130 (see FIG. 1 ) may perform remosaic. For example, the remosaic unit 131 maintains pixel values in sub-pixel units and can convert the analog signal AS into a Bayer pattern.

Figure 6 shows an example of pixel values on which remosaic was performed. Referring to FIGS. 3, 4, 5, and 6, pixel values corresponding to subpixels may be converted to be based on a Bayer pattern. For example, the pixel value corresponding to the subpixel in the first row and first column of the first pixel PX1 may be maintained to correspond to a red pixel value. The pixel value corresponding to the subpixel in the first row and second column of the first pixel PX1 may be converted (eg, based on interpolation) to correspond to a green pixel value. The pixel value corresponding to the subpixel in the first row and third column of the first pixel PX1 may be maintained to correspond to a red pixel value.

The pixel value corresponding to the subpixel in the second row and first column of the first pixel PX1 may be converted (eg, based on interpolation) to correspond to a green pixel value. The pixel value corresponding to the subpixel in the second row and second column of the first pixel PX1 may be converted (eg, based on interpolation) to correspond to a blue pixel value. Pixel values corresponding to subpixels in the second row and third column of the first pixel PX1 may be converted (eg, based on interpolation) to correspond to green pixel values.

The pixel value corresponding to the subpixel in the third row and first column of the first pixel PX1 may be maintained to correspond to a red pixel value. The pixel value corresponding to the subpixel in the third row and second column of the first pixel PX1 may be converted (eg, based on interpolation) to correspond to a green pixel value. The pixel value corresponding to the subpixel in the third row and third column of the first pixel PX1 may be maintained to correspond to a red pixel value.

The pixel value corresponding to the subpixel in the first row and first column of the second pixel PX2 may be maintained to correspond to a green pixel value. Pixel values corresponding to subpixels in the first row and second column of the second pixel PX2 may be converted (eg, based on interpolation) to correspond to red pixel values. The pixel value corresponding to the subpixel in the first row and third column of the second pixel PX2 may be maintained to correspond to a green pixel value.

The pixel value corresponding to the subpixel in the second row and first column of the second pixel PX2 may be converted (eg, based on interpolation) to correspond to a blue pixel value. The pixel value corresponding to the subpixel in the second row and second column of the second pixel PX2 may be maintained to correspond to a green pixel value. The pixel value corresponding to the subpixel in the second row and third column of the second pixel PX2 may be converted (eg, based on interpolation) to correspond to a blue pixel value.

The pixel value corresponding to the subpixel in the third row and first column of the second pixel PX2 may be maintained to correspond to a green pixel value. The pixel value corresponding to the subpixel in the third row and second column of the second pixel PX2 may be converted (eg, based on interpolation) to correspond to a red pixel value. The pixel value corresponding to the subpixel in the third row and third column of the second pixel PX2 may be maintained to correspond to the green pixel value.

Likewise, pixel values of subpixels of the third pixel PX3 and the fourth pixel PX4 may also be converted to be based on the Bayer pattern. For example, in FIGS. 5 and 6 , a pixel value corresponding to a subpixel filled with lines going from the upper right to the lower left may be a pixel value corresponding to red. The pixel value corresponding to the subpixel filled with lines going from the upper left to the lower right may be a pixel value corresponding to blue. The pixel value corresponding to the subpixel filled with dots may be a pixel value corresponding to green.

As explained with reference to FIG. 5, the subpixel detected by the central subpixel among the subpixels of the first pixel (PX1), the second pixel (PX2), the third pixel (PX3), and the fourth pixel (PX4) The pixel value may be lower than the pixel values sensed by the edge sub-pixels. Therefore, in the pixel values corresponding to the subpixels of FIG. 6, crosstalk may occur in which the pixel values corresponding to the central subpixels are lower than the pixel values corresponding to the edge subpixels.

FIG. 7 shows an example of how the crosstalk correction unit 133 of the processing unit 130 of the image sensor 100 corrects crosstalk. Referring to FIGS. 1, 6, and 7, in step S21, the crosstalk correction unit 133 uses a first pixel (PX1), a second pixel (PX2), a third pixel (PX3), and a fourth pixel (PX4). ) can be applied to each of the pixel values of the subpixels. Based on the gain, the crosstalk correction unit 133 may adjust pixel values corresponding to each subpixel by ratio.

In step S22, the crosstalk correction unit 133 applies an offset to the pixel values of the subpixels of the first pixel (PX1), the second pixel (PX2), the third pixel (PX3), and the fourth pixel (PX4). You can. Based on the offset, the crosstalk correction unit 133 may adjust (for example, add or subtract) pixel values corresponding to each subpixel by the offset value.

When light of the maximum brightness (or maximum illuminance) that can be detected by the subpixels of the first pixel (PX1), the second pixel (PX2), the third pixel (PX3), and the fourth pixel (PX4) is incident, the subpixels can be saturated and output the maximum pixel values. When the subpixels output the maximum pixel values, if the crosstalk correction unit 133 performs crosstalk correction, artifacts may occur in the image data (ID) due to gain and offset. Accordingly, the crosstalk correction unit 133 may be configured to omit (or not perform) crosstalk correction for saturated pixel values.

In addition, as explained with reference to FIG. 5, since the thickness of the color filter (CF1 or CF2) is not uniform, the first pixel (PX1), the second pixel (PX2), the third pixel (PX3), and the fourth pixel ( The subpixels at each edge of the PX4) may be saturated, and the subpixels in the center may not be saturated. In this case, artifacts due to crosstalk may occur on the image data (ID).

To prevent this problem, crosstalk correction is performed on the pixel values of subpixels, and a crosstalk correction algorithm is required to correct crosstalk in a situation where some subpixels are saturated. The processing unit 130 of the image sensor 100 according to an embodiment of the present disclosure calculates the saturation ratio from the image data information (IDI) and adjusts the remosaic setting based on the saturation ratio, so that some subpixels are saturated. In situations where this occurs, artifacts caused by crosstalk can be removed (or suppressed).

FIG. 8 is a diagram for explaining an example in which the application processor 200 generates image data information (IDI) from image data (ID). Referring to FIGS. 1 and 8 , the application processor 200 may divide image data (ID) into patches (PT). Each of the patches PT may include pixel values corresponding to a group of two or more pixels (or subpixels). Patches (PT) can divide image data (ID) in a grid form. The application processor 200 may generate image data information (IDI) based on the patches (PT).

For example, in the pixel values of FIG. 6, pixel values corresponding to red subpixels may be divided into patches (PT) of channels corresponding to red. Pixel values corresponding to blue subpixels may be divided into patches (PT) of channels corresponding to blue. Pixel values corresponding to green subpixels may be divided into patches (PT) of channels corresponding to green. Alternatively, pixel values corresponding to subpixels of a first type of green (e.g., Gr) are divided into patches (PT) of the channel corresponding to the first type of green, and For example, the pixel values corresponding to the subpixels of Gb) may be divided into patches (PT) of the channel corresponding to the second type of green. FIG. 9 shows that the application processor 200 stores image data information (PT). An example of the process of transmitting IDI) to the image sensor 100 is shown. Referring to FIGS. 1 and 9 , in step S31, the application processor 200 may receive image data (ID) from the image sensor 100. In step S22, the application processor 200 may generate image data information (IDI) for each patch (PT). In step S33, the application processor 200 may output the generated image data information (IDI) to the image sensor 100.

For example, the image data information IDI may include information on each pixel value of the patches PT, for example, an average pixel value. Image data information (IDI) includes a plurality of channels (e.g., red (R) channel, green (G) channel (or Gr channel and Gb channel), and blue (B) channel of each of the patches (PT). ) may include average pixel values.

As another example, the image data information (IDI) may include the number of saturated pixel values in each of the patches (PT). Image data information (IDI) includes a plurality of channels (e.g., red (R) channel, green (G) channel (or Gr channel and Gb channel), and blue (B) channel of each of the patches (PT). ) may include the number of saturated pixel values in each.

Figure 10 shows an example of how the processing unit 130 of the image sensor 100 suppresses crosstalk based on the saturation ratio. Referring to FIGS. 1 and 10 , in step S110, the saturation management unit 134 of the processing unit 130 may receive image data information (IDI).

In step S130, the saturation management unit 134 of the processing unit 130 may calculate the saturation ratio of the image data (ID) transmitted in the previous frame based on the image data information (IDI).

In step S140, the saturation management unit 134 of the processing unit 130 may adjust the remosaic setting of the remosaic unit 131 based on the calculated saturation ratio. For example, the saturation management unit 134 may adjust the remosaic setting to remove crosstalk by removing more noise from the image data (ID) of the current frame.

FIG. 11 shows an example of how the processing unit 130 of the image sensor 100 calculates the saturation ratio. Referring to FIGS. 1, 8, and 11, in step S210, the saturation management unit 134 of the processing unit 130 may detect N-channel average pixel values from the image data information (IDI). The average pixel values of N channels (N is a positive integer) are three channels (e.g., R, G, B), four channels (e.g., R, Gr, Gb, B), Alternatively, it may correspond to an arbitrary number of channels. The average pixel values of each channel may respectively correspond to patches PT. For example, the processing unit 130 may detect average pixel values of patches in the R channel, average pixel values of patches in the G channel, and average pixel values of patches in the B channel from the image data information (IDI). Alternatively, the processing unit 130 may select the average pixel values of the patches in the R channel, the average pixel values of the patches in the Gr channel, the average pixel values of the patches in the Gb channel, and the average pixel values of the patches in the B channel from the image data information (IDI). Values can be detected.

In step S220, the saturation management unit 134 of the processing unit 130 may determine saturated patches using a threshold value (TH) for each channel. For example, the threshold value (TH) may be set as a certain ratio (eg, 90%, 95%, etc.) to the maximum pixel value, and may be set and adjusted by the application processor 200. Patches (PT) whose average pixel value is greater than or equal to the threshold (TH) may be determined to be saturated patches. Patches (PT) whose average pixel value is smaller than the threshold (TH) may be determined to be unsaturated patches. For N channels, the threshold value (TH) can be set to the same value or to different values.

In step S230, the saturation management unit 134 of the processing unit 130 may count the number of saturated patches for each channel. The number and positions of saturated patches of N channels may be the same or different from each other.

In step S240, the saturation management unit 134 of the processing unit 130 may calculate the saturation ratio based on the maximum number of saturated patches among the N channels. For example, the saturation management unit 134 may calculate saturation ratios for each of the N channels, and determine the maximum value among the saturation ratios of the N channels as the final saturation ratio.

By way of example, N channels may correspond to different frequencies. The colors of the object captured by the pixel unit 110 may vary, and only one channel among the N channels may be saturated. For example, a blue sky on a sunny day may cause saturation of pixels corresponding to a blue color filter, and a sunset sky may cause saturation of pixels corresponding to a red color filter. A forest on a sunny day can cause saturation of pixels corresponding to a green color filter. If the pixels of one channel are saturated, artifacts due to crosstalk may occur in the image data (ID). The image sensor 100 according to an embodiment of the present disclosure can efficiently prevent (or suppress) artifacts due to crosstalk by using the highest saturation ratio among the saturation ratios of N channels.

Figure 12 shows an example of a method for adjusting remosaic settings according to an embodiment of the present disclosure. Referring to FIGS. 1 and 12 , in step S310, the saturation management unit 134 of the processing unit 130 of the image sensor 100 may adjust the flatness threshold among the remosaic settings of the remosaic unit 131. The flatness threshold may be a standard by which the remosaic unit 131 determines whether the digital signal DS is flat.

In step S320, the saturation management unit 134 of the processing circuit 1300 of the image sensor 100 may adjust the local flatness threshold among the remosaic settings of the remosaic unit 131. The local flatness threshold is set by the remosaic unit 131. may be a standard for determining a locally flat area on the digital signal (DS).

In step S330, the saturation management unit 134 of the processing circuit 1300 of the image sensor 100 may adjust the smoothing power among the remosaic settings of the remosaic unit 131. The smoothing power is controlled by the remosaic unit ( 131) may correspond to the intensity of blurring (or low-pass filtering) on the digital signal DS to remove noise.

FIG. 13 shows an example of a trend in which the processing unit 130 of the image sensor 100 adjusts remosaic settings according to the saturation rate. In Figure 13, the horizontal axis corresponds to the saturation ratio, and the vertical axis may correspond to the flatness threshold, local flatness threshold, or smoothing power.

As shown in FIG. 12, as the saturation ratio increases, the saturation management unit 134 of the processing unit 130 may increase the flatness threshold, local flatness threshold, or smoothing power of the remosaic unit 131. Exemplarily, the saturation management unit 134 may increase the flatness threshold, local flatness threshold, or smoothing power one-dimensionally, two-dimensionally, n-dimensionally, or exponentially. Exemplarily, the saturation management unit 134 may increase the flatness threshold, local flatness threshold, or smoothing power at the same rate or at different rates.

Figure 14 shows an example of how the processing unit 130 of the image sensor 100 performs remosaic based on the remosaic setting. Referring to FIGS. 1 and 14 , in step S410, the remosaic unit 131 of the processing unit 130 may determine the flatness of the digital signal DS using the flatness threshold of the remosaic setting. For example, if the difference (eg, maximum difference) between pixel values of the N channels is less than the flatness threshold, the remosaic unit 131 may determine that the digital signal DS is flat. If the difference (eg, maximum difference) between pixel values of the N channels is greater than or equal to the flatness threshold, the remosaic unit 131 may determine that the digital signal DS is not flat.

If it is determined that the digital signal DS is flat in step S420, the remosaic unit 131 of the processing unit 130 may perform step S430. In step S430, the remosaic unit 131 may perform smoothening on the entire digital signal DS. For example, the remosaic unit 131 may perform low-pass filtering on the digital signal DS based on the smoothing power of the remosaic setting.

If it is determined in step S420 that the digital signal DS is not flat, the remosaic unit 131 of the processing unit 130 may perform step S440. In step S440, the remosaic unit 131 may determine a flat area and a pattern area using the local flatness threshold of the remosaic setting. For example, if the difference between pixel values of N channels in a specific area on the digital signal DS is smaller than the local flatness threshold, the remosaic unit 131 may determine the area to be a flat area. If the difference between pixel values of each of the N channels in a specific area on the digital signal DS is greater than or equal to the local flatness threshold, the remosaic unit 131 determines the area to be a pattern area (e.g., a non-flat area). can do.

In step S450, the remosaic unit 131 may perform flattening (eg, low-pass filtering) on a flat area on the digital signal DS based on the smoothing power of the remosaic setting.

As described above, the image sensor 100 according to an embodiment of the present disclosure determines the digital signal DS to be flatter as the saturation ratio increases, and may perform flattening more strongly. Accordingly, artifacts caused by crosstalk of pixels that are saturated can be removed (or suppressed) by smoothing.

Figure 15 shows another example of how the processing unit 130 of the image sensor 100 calculates the saturation ratio. Referring to FIGS. 1 and 15 , in step S510, the saturation management unit 134 of the processing unit 130 may detect the number of saturated pixels from the image data information (IDI). For example, the application processor 200 counts the number of saturated pixel values in each of the patches PT (see FIG. 8), and calculates the number of saturated pixel values of each patch PT into image data information ( IDI). The saturation management unit 134 may detect the number of saturated pixels in each patch PT from the image data information IDI.

In step S520, the saturation management unit 134 may calculate the saturation ratio based on the number of saturated pixels. For example, the saturation management unit 134 may calculate the ratio of the number of saturated pixels to the total number of pixels in the pixel unit 110 as the saturation ratio.

As another embodiment, the application processor 200 calculates the number of saturated pixel values to the number of pixel values of each patch (PT) as a saturation ratio of each patch (PT), and the saturation rate of each patch (PT) The ratio can be included in image data information (IDI). The saturation management unit 134 may detect the saturation ratio of each patch (PT) from the image data information (IDI), and calculate the average of the saturation ratios of the patches (PT) as the saturation ratio.

16 is a block diagram of an electronic device including a multi-camera module. FIG. 17 is a detailed block diagram of the camera module of FIG. 16.

Referring to FIG. 16 , the electronic device 1000 may include a camera module group 1100, an application processor 1200, a PMIC 1300, and an external memory 1400.

The camera module group 1100 may include a plurality of camera modules 1100a, 1100b, and 1100c. Although the drawing shows an embodiment in which three camera modules 1100a, 1100b, and 1100c are arranged, the embodiments are not limited thereto. In some embodiments, the camera module group 1100 may be modified to include only two camera modules. Additionally, in some embodiments, the camera module group 1100 may be modified to include i camera modules (i is a natural number of 4 or more). By way of example, each of the plurality of camera modules 1100a, 1100b, and 1100c of the camera module group 1100 may include the image sensor 100 of FIG. 1 . The application processor 1200 may include the application processor 200 of FIG. 1 .

Hereinafter, the detailed configuration of the camera module 1100b will be described in more detail with reference to FIG. 17, but the following description may be equally applied to other camera modules 1100a and 1100b depending on the embodiment.

Referring to FIG. 17, the camera module 1100b includes a prism 1105, an optical path folding element (OPFE) 1110, an actuator 1130, an image sensing device 1140, and a storage device. It may include unit 1150.

The prism 1105 includes a reflective surface 1107 of a light-reflecting material and can change the path of light L incident from the outside.

In some embodiments, the prism 1105 may change the path of light L incident in the first direction X to the second direction Y perpendicular to the first direction X. In addition, the prism 1105 rotates the reflecting surface 1107 of the light reflecting material in the A direction about the central axis 1106, or rotates the central axis 1106 in the B direction in the first direction (X). The path of the incident light (L) can be changed to the vertical second direction (Y). At this time, the OPFE 1110 may also move in the third direction (Z) perpendicular to the first direction (X) and the second direction (Y).

In some embodiments, as shown, the maximum rotation angle of the prism 1105 in direction A may be less than 15 degrees in the plus (+) A direction and greater than 15 degrees in the minus (-) A direction. However, the embodiments are not limited thereto.

In some embodiments, the prism 1105 may move about 20 degrees in the plus (+) or minus (-) B direction, or between 10 degrees and 20 degrees, or between 15 degrees and 20 degrees, where the moving angle is plus. It can move at the same angle in the (+) or minus (-) B direction, or it can move to an almost similar angle within a range of about 1 degree.

In some embodiments, the prism 1105 may move the reflective surface 1107 of the light reflecting material in a third direction (eg, Z direction) parallel to the extending direction of the central axis 1106.

OPFE 1110 may include, for example, an optical lens comprised of j groups (where j is a natural number). The j lenses may change the optical zoom ratio of the camera module 1100b by moving in the second direction (Y). For example, assuming that the basic optical zoom magnification of the camera module 1100b is Z, when moving the m optical lenses included in the OPFE 1110, the optical zoom magnification of the camera module 1100b is 3Z or 5Z or The optical zoom magnification can be changed to 5Z or higher.

The actuator 1130 may move the OPFE 1110 or an optical lens (hereinafter referred to as an optical lens) to a specific position. For example, the actuator 1130 may adjust the position of the optical lens so that the image sensor 1142 is located at the focal length of the optical lens for accurate sensing.

The image sensing device 1140 may include an image sensor 1142, control logic 1144, and memory 1146. The image sensor 1142 can sense an image of a sensing object using light (L) provided through an optical lens.

The control logic 1144 may control the overall operation of the camera module 1100b. For example, the control logic 1144 may control the operation of the camera module 1100b according to a control signal provided through the control signal line CSLb.

The memory 1146 may store information necessary for the operation of the camera module 1100b, such as calibration data 1147. The calibration data 1147 may include information necessary for the camera module 1100b to generate image data using light L provided from the outside. The calibration data 1147 may include, for example, information about the degree of rotation described above, information about the focal length, and information about the optical axis. When the camera module 1100b is implemented as a multi-state camera whose focal length changes depending on the position of the optical lens, the calibration data 1147 includes the focal length value for each position (or state) of the optical lens. May include information related to auto focusing.

The storage unit 1150 may store image data sensed through the image sensor 1142. The storage unit 1150 may be placed outside the image sensing device 1140 and may be implemented in a stacked form with a sensor chip constituting the image sensing device 1140. In some embodiments, the storage unit 1150 may be implemented as an Electrically Erasable Programmable Read-Only Memory (EEPROM), but the embodiments are not limited thereto.

Referring to FIGS. 16 and 17 together, in some embodiments, each of the plurality of camera modules 1100a, 1100b, and 1100c may include an actuator 1130. Accordingly, each of the plurality of camera modules 1100a, 1100b, and 1100c may include the same or different calibration data 1147 according to the operation of the actuator 1130 included therein.

In some embodiments, one camera module (e.g., 1100b) of the plurality of camera modules 1100a, 1100b, and 1100c is a folded lens including the prism 1105 and OPFE 1110 described above. type camera module, and the remaining camera modules (e.g., 1100a, 1100b) may be vertical type camera modules that do not include the prism 1105 and OPFE 1110, but embodiments are limited to this. It doesn't work.

In some embodiments, one camera module (e.g., 1100c) among the plurality of camera modules (1100a, 1100b, 1100c) is a vertical camera module that extracts depth information using, for example, IR (Infrared Ray). It may be a type of depth camera. In this case, the application processor 1200 merges the image data provided from the depth camera and the image data provided from another camera module (e.g., 1100a or 1100b) to create a 3D depth image. can be created.

In some embodiments, at least two camera modules (eg, 1100a, 1100b) among the plurality of camera modules 1100a, 1100b, and 1100c may have different fields of view (field of view). In this case, for example, the optical lenses of at least two camera modules (eg, 1100a, 1100b) among the plurality of camera modules 1100a, 1100b, and 1100c may be different from each other, but are not limited thereto.

Additionally, in some embodiments, the viewing angles of each of the plurality of camera modules 1100a, 1100b, and 1100c may be different. In this case, optical lenses included in each of the plurality of camera modules 1100a, 1100b, and 1100c may also be different from each other, but are not limited thereto.

In some embodiments, each of the plurality of camera modules 1100a, 1100b, and 1100c may be arranged to be physically separated from each other. That is, rather than dividing the sensing area of one image sensor 1142 into multiple camera modules 1100a, 1100b, and 1100c, an independent image is generated inside each of the multiple camera modules 1100a, 1100b, and 1100c. Sensor 1142 may be placed.

Referring again to FIG. 16 , the application processor 1200 may include an image processing device 1210, a memory controller 1220, and an internal memory 1230. The application processor 1200 may be implemented separately from the plurality of camera modules 1100a, 1100b, and 1100c. For example, the application processor 1200 and the plurality of camera modules 1100a, 1100b, and 1100c may be implemented separately as separate semiconductor chips.

The image processing device 1210 may include a plurality of sub-image processors 1212a, 1212b, and 1212c, an image generator 1214, and a camera module controller 1216.

The image processing device 1210 may include a plurality of sub-image processors 1212a, 1212b, and 1212c corresponding to the number of camera modules 1100a, 1100b, and 1100c.

Image data generated from each camera module 1100a, 1100b, and 1100c may be provided to the corresponding sub-image processors 1212a, 1212b, and 1212c through separate image signal lines (ISLa, ISLb, and ISLc). For example, image data generated from the camera module 1100a is provided to the sub-image processor 1212a through the image signal line (ISLa), and image data generated from the camera module 1100b is provided to the image signal line (ISLb). The image data generated from the camera module 1100c may be provided to the sub-image processor 1212c through the image signal line (ISLc). Such image data transmission may be performed using, for example, a Camera Serial Interface (CSI) based on Mobile Industry Processor Interface (MIPI), but embodiments are not limited thereto.

Meanwhile, in some embodiments, one sub-image processor may be arranged to correspond to a plurality of camera modules. For example, the sub-image processor 1212a and the sub-image processor 1212c are not implemented separately from each other as shown, but are implemented integrated into one sub-image processor, and the camera module 1100a and the camera module 1100c Image data provided from may be selected through a selection element (eg, multiplexer) and then provided to the integrated sub-image processor.

Image data provided to each sub-image processor 1212a, 1212b, and 1212c may be provided to the image generator 1214. The image generator 1214 may generate an output image using image data provided from each sub-image processor 1212a, 1212b, and 1212c according to image generation information or mode signal.

Specifically, the image generator 1214 merges at least some of the image data generated from the camera modules 1100a, 1100b, and 1100c with different viewing angles according to the image generation information or mode signal to produce an output image. can be created. Additionally, the image generator 1214 may generate an output image by selecting one of the image data generated from the camera modules 1100a, 1100b, and 1100c having different viewing angles according to the image generation information or mode signal. .

In some embodiments, the image generation information may include a zoom signal or zoom factor. Additionally, in some embodiments, the mode signal may be, for example, a signal based on a mode selected by a user.

When the image generation information is a zoom signal (zoom factor) and each camera module (1100a, 1100b, 1100c) has a different observation field (viewing angle), the image generator 1214 performs different operations depending on the type of zoom signal. can be performed. For example, when the zoom signal is the first signal, the image data output from the camera module 1100a and the image data output from the camera module 1100c are merged, and then the merged image signal and the camera module not used for merging are merged. An output image can be generated using the image data output from 1100b. If the zoom signal is a second signal different from the first signal, the image generator 1214 does not merge the image data and uses one of the image data output from each camera module 1100a, 1100b, and 1100c. You can select to create an output image. However, the embodiments are not limited to this, and the method of processing image data may be modified and implemented as necessary.

In some embodiments, the image generator 1214 receives a plurality of image data with different exposure times from at least one of the plurality of sub-image processors 1212a, 1212b, and 1212c, and generates high dynamic range (HDR) data for the plurality of image data. ) By performing processing, merged image data with increased dynamic range can be generated.

The camera module controller 1216 may provide control signals to each camera module 1100a, 1100b, and 1100c. The control signal generated from the camera module controller 1216 may be provided to the corresponding camera modules 1100a, 1100b, and 1100c through separate control signal lines (CSLa, CSLb, and CSLc).

One of the plurality of camera modules (1100a, 1100b, 1100c) is designated as a master camera (e.g., 1100b) according to image generation information or mode signals including a zoom signal, and the remaining camera modules (e.g., For example, 1100a, 1100c) can be designated as slave cameras. This information may be included in the control signal and provided to the corresponding camera modules 1100a, 1100b, and 1100c through separate control signal lines (CSLa, CSLb, and CSLc).

Camera modules operating as master and slave can be changed depending on the zoom factor or operation mode signal. For example, when the viewing angle of the camera module 1100a is wider than that of the camera module 1100b and the zoom factor indicates a low zoom ratio, the camera module 1100b operates as a master and the camera module 1100a operates as a slave. It can operate as . Conversely, when the zoom factor indicates a high zoom magnification, the camera module 1100a may operate as a master and the camera module 1100b may operate as a slave.

In some embodiments, the control signal provided from the camera module controller 1216 to each camera module 1100a, 1100b, and 1100c may include a sync enable signal. For example, if the camera module 1100b is a master camera and the camera modules 1100a and 1100c are slave cameras, the camera module controller 1216 may transmit a sync enable signal to the camera module 1100b. The camera module 1100b that receives this sync enable signal generates a sync signal based on the sync enable signal, and transmits the generated sync signal to the camera modules (1100b) through the sync signal line (SSL). 1100a, 1100c). The camera module 1100b and the camera modules 1100a and 1100c may be synchronized to this sync signal and transmit image data to the application processor 1200.

In some embodiments, a control signal provided from the camera module controller 1216 to the plurality of camera modules 1100a, 1100b, and 1100c may include mode information according to the mode signal. Based on this mode information, the plurality of camera modules 1100a, 1100b, and 1100c may operate in a first operation mode and a second operation mode in relation to the sensing speed.

In a first operation mode, the plurality of camera modules 1100a, 1100b, and 1100c generate an image signal at a first rate (e.g., generate an image signal at a first frame rate) and transmit it to a second rate higher than the first rate. The image signal may be encoded at a higher rate (for example, an image signal of a second frame rate higher than the first frame rate), and the encoded image signal may be transmitted to the application processor 1200. At this time, the second speed may be 30 times or less than the first speed.

The application processor 1200 stores the received image signal, that is, the encoded image signal, in the internal memory 1230 provided inside or the memory 1400 outside the application processor 1200, and then stores the received image signal, that is, the encoded image signal, in the internal memory 1230 or the memory 1400 external to the application processor 1200. Alternatively, the encoded image signal may be read from the external memory 1400, decoded, and image data generated based on the decoded image signal may be displayed. For example, a corresponding subprocessor among the plurality of subprocessors 1212a, 1212b, and 1212c of the image processing device 1210 may perform decoding and may also perform image processing on the decoded image signal.

In the second operation mode, the plurality of camera modules 1100a, 1100b, and 1100c generate image signals at a third rate lower than the first rate (for example, generate image signals at a third frame rate lower than the first frame rate). generation) and transmit the image signal to the application processor 1200. The image signal provided to the application processor 1200 may be an unencoded signal. The application processor 1200 may perform image processing on a received image signal or store the image signal in the internal memory 1230 or the external memory 1400.

The PMIC 1300 may supply power, for example, a power supply voltage, to each of the plurality of camera modules 1100a, 1100b, and 1100c. For example, the PMIC 1300, under the control of the application processor 1200, supplies first power to the camera module 1100a through the power signal line (PSLa) and the camera module (1100a) through the power signal line (PSLb). Second power may be supplied to 1100b), and third power may be supplied to the camera module 1100c through the power signal line (PSLc).

The PMIC 1300 can generate power corresponding to each of the plurality of camera modules 1100a, 1100b, and 1100c in response to the power control signal (PCON) from the application processor 1200, and also adjust the power level. there is. The power control signal (PCON) may include a power adjustment signal for each operation mode of the plurality of camera modules 1100a, 1100b, and 1100c. For example, the operation mode may include a low power mode, and in this case, the power control signal (PCON) may include information about the camera module operating in the low power mode and the set power level. The levels of power provided to each of the plurality of camera modules 1100a, 1100b, and 1100c may be the same or different from each other. Additionally, the level of power may change dynamically.

By way of example, the image sensor 100 described with reference to FIGS. 1 to 15 may correspond to the image sensor 1142 of FIG. 17 . The image sensor 1142 may adjust remosaic settings based on the saturation of pixel values. The image sensor 1142 may perform remosaic based on the remosaic setting and perform crosstalk correction. Illustratively, remosaic crosstalk correction may be performed sequentially or in reverse order. If the image sensor 1142 is implemented according to an embodiment of the present disclosure, the quality of image data acquired by the application processor 1200 may be improved.

In the above-described embodiments, components according to the technical idea of the present invention have been described using terms such as first, second, third, etc. However, terms such as first, second, third, etc. are used to distinguish components from each other and do not limit the present invention. For example, terms such as first, second, third, etc. do not imply order or any form of numerical meaning.

In the above-described embodiments, components according to embodiments of the present invention have been referenced using blocks. Blocks include various hardware devices such as IC (Integrated Circuit), ASIC (Application Specific IC), FPGA (Field Programmable Gate Array), and CPLD (Complex Programmable Logic Device), software such as firmware and applications running on the hardware devices, Alternatively, it may be implemented as a combination of a hardware device and software. Additionally, blocks may include circuits comprised of semiconductor elements within an IC or circuits registered as IP (Intellectual Property).

The above-described details are specific embodiments for carrying out the present invention. The present invention will include not only the above-described embodiments, but also embodiments that can be simply changed or easily changed in design. In addition, the present invention will also include technologies that can be easily modified and implemented using the embodiments. Therefore, the scope of the present invention should not be limited to the above-described embodiments, but should be determined by the claims and equivalents of the present invention as well as the claims described later.

10: Electronic device
100: image sensor
110: Pixel part
120: data conversion unit
130: processing unit
131: Remosaic section
132: Binning unit
133: Crosstalk correction unit
134: Saturation Management Department

Claims (20)

a pixel unit including a plurality of pixels and configured to generate an analog signal using the plurality of pixels;
a data converter that receives the analog signal and converts the analog signal into a digital signal; and
A processing unit configured to generate image data by performing crosstalk correction and remosaic on the digital signal, and output the image data to an external device,
The processing unit is configured to receive image data information from the external device, calculate a saturation ratio based on the image data information, and adjust settings of the remosaic based on the saturation ratio. According to paragraph 1,
The processing unit is further configured to detect average pixel values of groups of pixel values corresponding to the plurality of pixels on the image data from the image data information, and calculate the saturation ratio based on the average pixel values. sensor. According to paragraph 2,
The processing unit is further configured to calculate the saturation ratio by comparing the average pixel values with a threshold value. According to paragraph 3,
The processing unit is further configured to calculate a ratio of average pixel values higher than the threshold among average pixel values as the saturation ratio. According to paragraph 1,
The image data information includes information about the number of saturated pixel values among pixel values corresponding to the plurality of pixels, and
The processing unit is further configured to calculate the saturation ratio based on the number of saturated pixel values. According to paragraph 1,
The image data includes a plurality of channel data each corresponding to a plurality of channels,
The image data information includes a plurality of channel data information corresponding to each of the plurality of channels, and
The processing unit is further configured to calculate a plurality of saturation ratios corresponding to each of the plurality of channels. According to clause 6,
The processing unit is further configured to adjust the remosaic settings based on the highest saturation ratio among the plurality of saturation ratios. According to paragraph 1,
The processing unit includes a flatness threshold used to determine whether the image data is flat image data, a local flatness threshold used to determine a flat area of the image data, and a smoothing function used to smooth the flat area of the image data. An image sensor further configured to adjust at least one of smoothing power. According to clause 8,
The processing unit is further configured to increase at least one of the flatness threshold, the local flatness threshold, and the smoothing power as the saturation ratio increases. According to clause 9,
The processing unit is further configured to increase at least two of the flatness threshold, the local flatness threshold, and the smoothing power at the same rate as the saturation ratio increases. According to clause 9,
The processing unit is further configured to increase at least two of the flatness threshold, the local flatness threshold, and the smoothing power at different rates as the saturation ratio increases. According to paragraph 1,
The processing unit is further configured to correct crosstalk of saturated pixels among the plurality of pixels by adjusting the remosaic settings. According to clause 12,
The processing unit is further configured to perform the crosstalk correction of the unsaturated pixels by applying gain and offset to pixel values of the unsaturated pixels among the plurality of pixels. In a method of operating an image sensor image sensor including a plurality of pixels:
The image sensor receiving image data information from an external device;
calculating, by the image sensor, a saturation ratio of image data acquired by the image sensor from the image data information; and
Comprising the step of the image sensor adjusting remosaic settings based on the saturation ratio,
An operation method wherein the image sensor is configured to perform remosaicing on image data acquired by the image sensor based on the remosaic setting. According to clause 14,
The image data information includes information on image data of a previous frame output by the image sensor to the external device. According to clause 14,
An operating method wherein the image sensor performs the remosaic on image data of the current frame based on the remosaic setting. According to clause 14,
The steps for calculating the saturation ratio are:
detecting average pixel values of groups of pixel values corresponding to the plurality of pixels from the image data information;
Using a threshold value, determining saturated groups among the groups of pixel values;
counting the number of saturated groups; and
A method of operation comprising calculating the saturation ratio based on the number of saturated groups. According to clause 14,
The steps for calculating the saturation ratio are:
detecting the number of saturated pixels corresponding to a saturated pixel value among the plurality of pixels from the image data information; and
A method of operation comprising calculating the saturation ratio based on the number of saturated pixels. An image sensor that generates image data; and
A processor that generates image data information based on the image data and transmits the image data information to the image sensor,
The image sensor includes a plurality of pixels, calculates a saturation ratio of the plurality of pixels from the image data information, and performs crosstalk of the plurality of pixels based on the saturation ratio in one of a first mode and a second mode. It is configured to correct based on one,
In the first mode, the image sensor is configured to correct the crosstalk based on gain and offset, and
In the second mode, the image sensor is configured to correct the crosstalk based on the gain and the offset, and to further correct the crosstalk by adjusting a remosaic setting. According to clause 19,
The image sensor enters the first mode when the saturation ratio is lower than the reference value, and enters the second mode when the saturation ratio is higher than the reference value.

Images