KR20240037810A - Method of operating image sensor and image device performing the same - Google Patents

Abstract

In a method of operating an image sensor according to an embodiment of the present disclosure, first result image data is generated by performing first image capturing. The first resulting image data includes a plurality of first SDR image data and first HDR image data. Based on the resulting image data, first parameters are generated. The first parameters are related to capturing the first image. An adjustment factor is generated based on the first parameters. Second parameters are generated by adjusting the first parameters based on the adjustment factor. A second image capturing is performed based on the second parameters.

Description

Method of operating an image sensor and an image device that performs the same {METHOD OF OPERATING IMAGE SENSOR AND IMAGE DEVICE PERFORMING THE SAME}

The present invention relates to a semiconductor device, and more specifically, to a method of operating an image sensor and an image device that performs the method.

An image sensor converts light incident through a camera lens into digital image data. The image sensor generates a plurality of standard dynamic range (SDR) image data with different average brightnesses for high dynamic range (HDR), and based on the plurality of SDR image data HDR image data can be generated.

A plurality of SDR image data can be obtained by applying different exposure times and/or different sensor gains to one subject. However, in a backlit environment other than a general environment, the range between the exposure times and/or the sensor gains may exceed the hardware constraints of the image sensor for obtaining HDR image data. In this case, due to the hardware constraints, the SDR image data with the lowest average brightness among the plurality of SDR image data is captured brighter than the unintended level, which may lead to difficulties in securing sufficient dynamic range in the HDR image data. there is.

One purpose of the present disclosure is to provide a method of operating an image sensor that generates high dynamic range (HDR) image data.

One purpose of the present disclosure is to provide a method of operating an image sensor that generates HDR image data in a backlit environment.

One object of the present disclosure is to provide an imaging device that performs the above method.

In order to achieve the above object, in a method of operating an image sensor according to an embodiment of the present disclosure, first result image data is generated by performing first image capturing. The first resulting image data includes a plurality of first standard dynamic range (SDR) image data and first high dynamic range (HDR) image data. Based on the resulting image data, first parameters are generated. The first parameters are related to capturing the first image and include a plurality of first exposure times and a plurality of first sensor gains. An adjustment factor is generated based on the first parameters. Second parameters are generated by adjusting the first parameters based on the adjustment factor. The second parameters include a plurality of second exposure times and a plurality of second sensor gains. A second image capturing is performed based on the second parameters.

To achieve the above object, an image device according to an embodiment of the present disclosure includes an application processor and an image sensor. The application processor is configured to determine, based on first resultant image data comprising a plurality of first SDR image data and first HDR image data, a plurality of first exposure times and a plurality of first exposure times associated with first image capturing. 1 Generate first parameters including sensor gains. The application processor generates an adjustment factor based on the first parameters. The application processor adjusts the first parameters based on the adjustment factor to generate second parameters including a plurality of second exposure times and a plurality of second sensor gains.

In order to achieve the above object, in a method of operating an image sensor according to an embodiment of the present disclosure, first image capturing is performed to produce a first result including a plurality of first SDR image data and first HDR image data. Image data is created. Based on the resulting image data, first parameters related to capturing the first image and including a first long exposure parameter and a first short exposure parameter are generated. An adjustment factor is generated based on the first parameters. Second parameters including a second long exposure parameter and a second short exposure parameter are generated by adjusting the first parameters based on the adjustment factor. A second image capturing is performed based on the second parameters. The ratio of the second long exposure parameter and the second short exposure parameter is calculated to satisfy the hardware constraints of the image sensor for obtaining HDR image data.

In a method of operating an image sensor according to an embodiment of the present disclosure, second image capturing may be performed based on the results of performing first image capturing. First parameters may be generated based on the results of performing the first image capturing, and second parameters for performing the second image capturing may be generated based on the first parameters. The second parameters may be calculated to satisfy hardware constraints of an image device (or image sensor) for obtaining HDR image data. Therefore, the image device can obtain HDR image data with a sufficient dynamic range by stabilizing the brightness value of the subject and preventing saturation of the background, etc., even when the surrounding environment changes from a normal environment to a backlight environment. .

1 is a flowchart showing a method of operating an image sensor according to an embodiment of the present disclosure.
Figure 2 is a block diagram showing an imaging device according to an embodiment of the present disclosure.
FIG. 3A is a diagram for explaining a unit pixel included in the pixel array of FIG. 2, and FIG. 3B is a circuit diagram showing an example of a unit color pixel included in the unit pixel.
FIG. 4 is a block diagram showing an embodiment of the first parameter generator of FIG. 2.
FIGS. 5A and 5B are diagrams for explaining the first parameter of FIG. 4.
FIG. 6 is a flow chart illustrating one embodiment of an operation for generating the adjustment factor of FIG. 1.
FIG. 7 is a flowchart illustrating one embodiment of an operation for calculating the adjustment factor of FIG. 6.
FIG. 8 is a block diagram illustrating one embodiment of the adjustment factor generator of FIG. 2.
FIG. 9 is a diagram illustrating an embodiment of the illuminance-adjustment factor table of FIG. 8.
FIGS. 10A and 10B are diagrams illustrating an example of an operation for generating the second parameters of FIG. 1 .
FIG. 11 is a block diagram showing an embodiment of the second parameter generator of FIG. 2.
FIG. 12 is a diagram to explain hardware limitations of an image sensor.
13 is a flowchart showing a method of operating an image sensor according to an embodiment of the present disclosure.
FIG. 14 is a flowchart illustrating an example of an operation for generating the second result image data of FIG. 13 .
Figure 15 is a block diagram showing an imaging device according to an embodiment of the present disclosure.
FIG. 16 is a diagram for explaining operations of an image sensor according to embodiments of the present disclosure.
FIGS. 17A, 17B, 18A, 18B, 19A, and 19B are diagrams for explaining the first parameters, second parameters, first dynamic range compression ratio, and second dynamic range compression ratio of FIG. 1.
Figure 20 is a flowchart showing a method of operating an image sensor according to an embodiment of the present disclosure.
FIG. 21 is a diagram for explaining an example of the operation method of the image sensors of FIGS. 2 and 15.
Figure 22 is a block diagram showing a computing system including an image sensor according to embodiments of the present disclosure.
Figure 23 is a block diagram showing an electronic system including an image sensor according to embodiments of the present disclosure.

Below, embodiments of the present disclosure will be described clearly and in detail so that a person skilled in the art can easily practice the present disclosure.

1 is a flowchart showing a method of operating an image sensor according to an embodiment of the present disclosure. Figure 2 is a block diagram showing an imaging device according to an embodiment of the present disclosure.

1 and 2, in the method of operating the image sensor 100 according to an embodiment of the present disclosure, the image sensor 100 performs first image capturing 151 to capture a plurality of first standard dynamic images. First resulting image data (RES_IMG1s) comprising standard dynamic range (SDR) image data (SDR_IMG1s) and first high dynamic range (HDR) image data (HDR_IMG1) may be generated. There is (S100).

In one embodiment, the imaging device 10 may be a high-resolution imaging device that sequentially or simultaneously captures SDR image data and generates HDR image data based on the SDR image data. The image device 10 may include an image sensor 100 and an application processor 500. The image sensor 100 may include a pixel array 101, and the application processor 500 may include a first parameter generator 510, an adjustment factor generator 530, and a second parameter generator 550. .

In one embodiment, the image sensor 100 may perform first image capturing 151 and second image capturing 155, which are continuous image capturing operations. For example, the image sensor 100 may perform the second image capturing 155 after performing the first image capturing 151. For example, when the imaging device 10 operates in a video mode, the imaging device 10 may perform first and second image capturing operations 151 and 155 to capture successive frames, respectively. , when the imaging device 10 operates in a still image mode, the imaging device 10 may perform first image capturing 151 to capture a preview image.

In one embodiment, the pixel array 101 may include a plurality of unit pixels. The image sensor 100 uses the pixel array 101 to generate result image data including a plurality of SDR image data and one HDR image data in each of the first and second image captures 151 and 155. You can. For example, the image sensor 100 performs the first image capturing 151 to produce first result image data including a plurality of first SDR image data (SDR_IMG1s) and first HDR image data (HDR_IMG1). (RES_IMG1s) may be generated, and second image capturing 155 may be performed to generate second resultant image data including a plurality of second SDR image data and second HDR image data. The unit pixels will be described later with reference to FIGS. 3A and 3B.

In one embodiment, the image sensor 100 may generate HDR image data based on a plurality of SDR image data. For example, the image sensor 100 generates merged image data by merging a plurality of SDR image data into one, and sequentially performs segmentation and tone mapping on the merged image data. You can generate HDR image data by performing , but this is only an example.

The application processor 500 may generate first parameters PARAM1 based on the first result image data RES_IMG1s (S200).

In one embodiment, the first parameters PARAM1 may be related to first image capturing 151 and include a plurality of first exposure times ET1s and a plurality of first sensor gains SG1s. It may further include a first long exposure parameter, a first medium exposure parameter, and a first short exposure parameter.

In one embodiment, the plurality of first exposure times ET1s may include set times, and the plurality of first sensor gains SG1s may include set gains. Each of the first long exposure parameter, the first medium exposure parameter, and the first short exposure parameter may be obtained by multiplying one of the set times and one of the set gains. For example, the first long exposure parameter can be obtained by multiplying the longest time among the set times by the longest gain among the set gains, and the first short exposure parameter can be obtained by multiplying the longest time among the set times. It can be obtained by multiplying the shortest gain among the set gains, but this is only an example. In another embodiment, in order to adjust the target ratio between the first long exposure parameter and the first short exposure parameter, setting times for obtaining each of the first long exposure parameter and the first short exposure parameter are adjusted. , if the target ratio is not achieved despite adjusting the set times, set gains for obtaining the first long exposure parameter and the first short exposure parameter, respectively, may be secondarily adjusted. In another embodiment, the set gains may be set to ‘1’ by default.

The first parameters (PARAM1) will be described later with reference to FIGS. 4, 5A, and 5B.

The application processor 500 may generate an adjustment factor (ADJ_FCT) based on the first parameters (PARAM1) (S300).

In one embodiment, the application processor 500 generates a plurality of first SDR image data SDR_IMG1s based on some of the first exposure times ET1s and some of the first sensor gains SG1s. Alternatively, information related to the first HDR image data (HDR_IMG1) may be generated, and an adjustment factor (ADJ_FCT) may be generated based on the information. For example, the application processor 500 may configure some of the plurality of first exposure times ET1s and some of the plurality of first sensor gains SG1s (e.g., the first long exposure parameter and the first sensor gain SG1s). The first dynamic range compression ratio and the first illuminance may be calculated based on the short exposure parameter, and the adjustment factor ADJ_FCT may be calculated based on the first dynamic range compression ratio, the first illuminance, and the first adjustment factor relational expression.

In one embodiment, the application processor 500 may call the first adjustment factor relational expression from an illuminance-adjustment factor table based on the first illuminance. The adjustment factor (ADJ_FCT) will be described later with reference to FIGS. 6, 7, 8, and 9.

The application processor 500 may generate second parameters PARAM2 by adjusting the first parameters PARAM1 based on the adjustment factor ADJ_FCT (S400).

In one embodiment, the second parameters PARAM2 may be related to second image capturing 155 and include a plurality of second exposure times ET2s and a plurality of second sensor gains SG2s. It may further include a second long exposure parameter, a second medium exposure parameter, and a second short exposure parameter.

In one embodiment, the plurality of second exposure times ET2s may include set times, and the plurality of second sensor gains SG2s may include set gains. Each of the second long exposure parameter, the second medium exposure parameter, and the second short exposure parameter may be obtained by multiplying one of the set times and one of the set gains. The second parameters (PARAM2) will be described later with reference to FIGS. 10A, 10B, and 11.

The image sensor 100 may perform second image capturing 155 based on the second parameters PARAM2 (S500).

In one embodiment, the first long exposure parameter, the first medium exposure parameter, and the first short exposure parameter are each of a plurality of first SDR image data (SDR_IMG1s) included in the first result image data (RES_IMG1s). Can indicate parameters applied (or estimated to be applied) to generate . For example, the SDR image data with the brightest average brightness value among the first SDR image data (SDR_IMG1s) may be estimated to have been generated by applying the first long exposure parameter, and the SDR image data with the brightest average brightness value among the first SDR image data (SDR_IMG1s) ), the SDR image data having the lowest average brightness value may be estimated to have been generated by applying the first single exposure parameter, and the SDR image data having the middle average brightness value among the first SDR image data (SDR_IMG1s) It can be estimated that was created by applying the first medium exposure parameter. The second long exposure parameter, the second medium exposure parameter, and the second short exposure parameter may also have the same or similar relationship to the first long exposure parameter, the first medium exposure parameter, and the first short exposure parameter. , the image sensor 100 may perform second image capturing 155 by directly applying the second long exposure parameter, the second medium exposure parameter, and the second short exposure parameter.

While an imaging device is capturing a subject, rapid changes in the surrounding environment may occur. For example, the surrounding environment of the imaging device 10 may change from a normal environment to a backlit environment. In this case, the image device 10 may increase the long exposure parameter to the maximum value so that dark areas in the image data according to the backlight environment become brighter. For example, the image device 10 may increase each of the longest exposure time among the exposure times and/or the largest sensor gain among the sensor gains to a maximum value.

However, when the long exposure parameter is increased to the maximum as described above, the short exposure parameter may also increase above a certain level, which may be related to hardware limitations of the image sensor for obtaining HDR image data. For example, the hardware constraints include the ratio of the maximum and minimum shutter speeds of the electronic shutter for controlling each of the exposure times, and motion artifacts that may occur in the process of merging SDR image data. ) may include removal ability. In this case, SDR image data with the lowest average brightness among the SDR image data for obtaining HDR image data may be captured at an unintended brightness level, and accordingly, merging, division, and tone mapping to generate HDR image data may be performed. Because the process is not performed properly, it is difficult to secure sufficient dynamic range in HDR image data.

In the method of operating an image sensor according to an embodiment of the present disclosure with the above configuration, second image capturing can be performed based on the results of performing first image capturing. First parameters may be generated based on the results of performing the first image capturing, and second parameters for performing the second image capturing may be generated based on the first parameters. The second parameters may be calculated to satisfy hardware constraints of an image device (or image sensor) for obtaining HDR image data. Accordingly, the imaging device can obtain HDR image data with a sufficient dynamic range by stabilizing the brightness value of the subject and preventing saturation of the background, etc., even when the surrounding environment changes from a normal environment to a backlight environment.

FIG. 3A is a diagram for explaining a unit pixel included in the pixel array of FIG. 2, and FIG. 3B is a circuit diagram showing an example of a unit color pixel included in the unit pixel.

Referring to FIGS. 2 and 3A, the pixel array 101 may include a plurality of unit pixels (UPs), and each unit pixel (UP) corresponds to a specific color and includes subpixels. It may include through fourth unit color pixels CP1, CP2, CP3, and CP4. For example, the first unit color pixel CP1 may include subpixels Ga1, Ga2, Ga3, and Ga4 corresponding to green, and the second unit color pixel CP2 may include subpixels corresponding to red. may include (R1, R2, R3, R4), and the third unit color pixel (CP3) may include subpixels (B1, B2, B3, B4) corresponding to blue, and the fourth unit color pixel (CP3) may include subpixels (B1, B2, B3, B4) corresponding to blue. The color pixel CP4 may include subpixels Gb1, Gb2, Gb3, and Gb4 corresponding to green.

In one embodiment, subpixels corresponding to one color may output analog signals based on different exposure times. For example, among the subpixels corresponding to green, the subpixel Ga1 may output an analog signal during a long exposure time, and the subpixels Ga2 and Ga3 may output an analog signal during a medium exposure time shorter than the long exposure time. Analog signals can be output, and the subpixel Ga4 can output analog signals during a short exposure time shorter than the medium exposure time. Subpixels corresponding to red, blue, and other green colors may also operate similarly.

In one embodiment, the analog signals output based on different exposure times may be converted into a plurality of SDR image data based on different sensor gains including analog gain and digital gain. For example, the analog signals output from the subpixels (Ga1, Ga2, Ga3, Ga4) pass through a correlated double sampling (CDS) circuit and an analog-to-digital converter (ADC) circuit and are additionally processed through an image signal processing circuit. You can pass. Different sensor gains may be applied to the analog signals while passing through the CDS circuit, the ADC circuit, and the image signal processing circuit.

2, 3A, and 3B, the unit color pixel CP1 includes first to fourth photoelectric conversion elements PD1, PD2, PD3, and PD4, and first to fourth transfer transistors TX1 and TX2. , TX3, TX4), a reset transistor (RX), a selection transistor (SX), and a driving transistor (DX).

The first to fourth photoelectric conversion elements PD1 to PD4 may correspond to the subpixels Ga1, Ga2, Ga3, and Ga4, respectively, and may generate and accumulate photocharges based on the intensity of incident light. For example, each of the first to fourth photoelectric conversion elements PD1 to PD4 may be one of a photo diode, a photo transistor, a photo gate, and a pinned photo diode (PPD), but this is only an example. It's just that.

The first to fourth transfer transistors TX1 to TX4 may be connected to the first to fourth photoelectric conversion elements PD1 to PD4, respectively, and may be connected to the first to fourth photoelectric conversion elements PD1 to PD4 based on the control signals TG1, TG2, TG3, and TG4. Charges accumulated in the first to fourth photoelectric conversion elements PD1 to PD4 may be transferred to the floating diffusion region FD. The driving transistor (DX) may be connected to the floating diffusion region (FD), and the selection transistor (SX) may be connected to the driving transistor (DX), and the driving transistor (DX) and the selection transistor (SX) may be connected to the floating diffusion region (FD). Based on the voltage level of ) and the control signal (SEL), charges transferred to the floating diffusion region (FD) may be output to the column line of the pixel array. The reset transistor RX may be connected between the power supply voltage VPIX and the floating diffusion region FD, and may reset the floating diffusion region FD based on the control signal RG.

In one embodiment, the turn-off/turn-on timing of the first to fourth transfer transistors TX1 to TX4 are controlled based on the control signals TG1, TG2, TG3, and TG4 to turn the subpixels Ga1 and Ga2. , Ga3, Ga4) can be adjusted. For example, from the point when the first transfer transistor TX1 is turned off to accumulate photocharges in the first photoelectric conversion element PD1, the first transfer transistor TX1 is turned on to store the accumulated photocharges in the floating diffusion region. The exposure time for the subpixel (Ga1) may progress until the point of transmission to (FD).

FIG. 4 is a block diagram showing an embodiment of the first parameter generator of FIG. 2.

2 and 4, the first parameter generator 510 includes a first set time and set gain calculator 511, a second set time and set gain calculator 513, and a third set time and set gain calculator ( 515) may be included.

The first parameter generator 510 is related to first image capturing and generates first resultant image data RES_IMG1s including a plurality of first SDR image data SDR_IMG1s and first HDR image data HDR_IMG1. You can receive it. The first setup time and setup gain calculator 511 may calculate the setup time (ET11) and setup gain (SG11) based on the first HDR image data (HDR_IMG1), and the second setup time and setup gain calculator 513 Can calculate the setup time (ET13) and setup gain (SG13) based on all or part of the plurality of first SDR image data (SDR_IMG1s), and the third setup time and setup gain calculator 515 calculates the setup times. The setup time (ET12) and the setup gain (SG12) can be calculated based on (ET11, ET13) and the setup gains (SG11, SG13).

In one embodiment, the first parameters PARAM1 may include a plurality of first exposure times ET1s and a plurality of first sensor gains SG1s, and a plurality of first exposure times ET1s may include setting times (ET11, ET12, ET13), and the plurality of first sensor gains (SG1s) may include setting gains (SG11, SG12, SG13).

In one embodiment, the setting time ET11 is the longest among the plurality of first exposure times ET1s, the setting time ET13 is the shortest among the plurality of first exposure times ET1s, and the setting time ET12 is the longest among the plurality of first exposure times ET1s. It can have a value between the set time (ET11) and the set time (ET13). The set gains (SG11, SG12, SG13) may respectively correspond to the set times (ET11, ET12, ET13).

In one embodiment, the second setup time and setup gain calculator 513 calculates the setup time (ET13) and setup gain (SG13) based only on the SDR image data having the darkest average brightness among the plurality of SDR image data (SDR_IMGs). can also be calculated. In another embodiment, the second setup time and setup gain calculator 513 is provided from an image sensor (e.g., 100 in FIG. 2) instead of all or part of the plurality of first SDR image data (SDR_IMG1s). The setup time (ET13) and setup gain (SG13) may be calculated based on statistical data (STAT_DAT).

In one embodiment, the first parameters PARAM1 may further include a first long exposure parameter, a first medium exposure parameter, and a first short exposure parameter. For example, the first long exposure parameter can be obtained by multiplying the set time (ET11) by the set gain (SG11), and the first long exposure parameter can be obtained by multiplying the set time (ET12) by the set gain (SG12). The first single exposure parameter may be obtained by multiplying the set time (ET13) by the set gain (SG13), but the scope of the present disclosure is not limited thereto.

FIGS. 5A and 5B are diagrams for explaining the first parameter of FIG. 4.

Referring to FIGS. 2, 4, and 5A, the imaging device may generate a plurality of first SDR image data (SDR_IMG1s) by performing first image capturing on a subject standing indoors with its back to a glass window.

Referring to FIGS. 5A and 5B, the image device can generate SDR image data (SDR_IMG11) with the highest average brightness value based on the setup time (ET11) and the setup gain (SG11), and the setup time Based on (ET13) and set gain (SG13), SDR image data (SDR_IMG13) with the lowest average brightness value can be generated, and based on set time (ET12) and set gain (SG12), SDR image data (SDR_IMG13) can be generated SDR image data (SDR_IMG12) having an average brightness value in the middle of (SDR_IMG11, SDR_IMG13) can be generated.

In one embodiment, as SDR_IMG11 → SDR_IMG12 → SDR_IMG13 changes, the brightness of the subjects (Object_S1, Object_S2, Object_S3) and backgrounds included in the SDR image data (SDR_IMG11, SDR_IMG12, SDR _IMG13) may become darker. .

FIG. 6 is a flow chart illustrating one embodiment of an operation for generating the adjustment factor of FIG. 1.

Referring to FIGS. 1, 2, and 6, in the operation of generating the adjustment factor ADJ_FCT (S300), the application processor 500 may calculate the first dynamic range compression ratio (S310).

In one embodiment, the application processor 500 includes a first set time and a second set time among the plurality of first exposure times ET1s, and a first set gain and a second set time among the plurality of first sensor gains SG1s. The first dynamic range compression ratio may be calculated based on the second set gain. For example, the application processor 500 calculates the first dynamic range compression ratio by dividing the first set time multiplied by the first set gain by the second set time multiplied by the second set gain. You can. For example, the application processor 500 may calculate the first dynamic range compression ratio by dividing the first long exposure parameter by the first short exposure parameter.

In one embodiment, the first set time is the longest among the plurality of first exposure times (ET1s), the second set time is the shortest among the plurality of first exposure times (ET1s), and the first set gain may correspond to the first setting time, and the second setting gain may correspond to the second setting time.

The application processor 500 may calculate the first illuminance (S330). In one embodiment, the application processor 500 may calculate the first illuminance based on the first set time and the first set gain. For example, the application processor 500 may calculate the first illuminance based on the first long exposure parameter.

The application processor 500 may calculate an adjustment factor (ADJ_FCT) based on the first dynamic range compression ratio, the first illuminance, and the first adjustment factor relationship (S350).

FIG. 7 is a flowchart illustrating one embodiment of an operation for calculating the adjustment factor of FIG. 6.

Referring to FIGS. 1, 2, 6, and 7, in the operation of calculating the adjustment factor (ADJ_FCT) (S350), the application processor 500 determines the illuminance corresponding to the first illuminance based on the illuminance-adjustment factor table. The first adjustment factor relational expression may be called (S351). The illuminance-adjustment factor table will be described later with reference to FIG. 9.

The application processor 500 may calculate an adjustment factor (ADJ_FCT) by applying the first adjustment factor relational expression to the first dynamic range compression ratio (S353).

In one embodiment, the size of the adjustment factor (ADJ_FCT) may increase as the first dynamic range compression ratio increases.

FIG. 8 is a block diagram illustrating one embodiment of the adjustment factor generator of FIG. 2.

2, 6, 7, and 8, the adjustment factor generator 530 includes a dynamic range compression ratio calculator 531, a roughness calculator 533, an adjustment factor calculator 535, and a roughness-adjustment factor table 537. ) may include. The adjustment factor generator 530 may perform the operations (S300, S310, S330, S350, S351, and S353) of FIGS. 2, 6, and 7.

The dynamic range compression ratio calculator 531 is based on the set times (ET11, ET13) among the plurality of first exposure times (ET1s) and the set gains (SG11, SG13) among the plurality of first sensor gains (SG1s). The first dynamic range compression ratio (DRC_R1) can be calculated.

The illuminance calculator 533 calculates the first illuminance (ILLV1) based on the set time (ET11) among the plurality of first exposure times (ET1s) and the set gain (SG11) among the plurality of first sensor gains (SG1s). You can.

The adjustment factor calculator 535 calls the first adjustment factor relation (RLSx) corresponding to the first illuminance (ILLV1) from the illuminance-adjustment factor table 537, and calculates the first adjustment factor to the first dynamic range compression ratio (DRC_R1). The adjustment factor (ADJ_FCT) can be calculated by applying the relation (RLSx).

FIG. 9 is a diagram illustrating an embodiment of the illuminance-adjustment factor table of FIG. 8.

Referring to FIGS. 8 and 9, the illuminance-adjustment factor table (ILL_AF_Table) includes a plurality of illuminance sections (IM_LV1, IM_LV2, IM_LV3, IM_LV4, IM_LV5) and a plurality of relational equations (RLS1, RLS2, RLS3, RLS4, RLS5) may include.

In one embodiment, the plurality of illuminance sections IM_LV1 to IM_LV5 may divide the illuminance range that can be expressed as the illuminance of the image sensor. A plurality of relational equations (RLS1, RLS2, RLS3, RLS4, RLS5) may be defined for each of the plurality of illuminance sections (IM_LV1 to IM_LV5).

In one embodiment, when the first illuminance ILLV1 belongs to one of the plurality of illuminance intervals IM_LV1 to IM_LV5 (e.g., IM_LV1), the adjustment factor calculator 535 calculates the corresponding relational expression (e.g., RLS1) can be called the first adjustment factor relation (RLSx).

In one embodiment, the plurality of illuminance sections (IM_LV1 to IM_LV5) include a first illuminance section (e.g., IM_LV1) and a second illuminance section (e.g., IM_LV2) indicating a higher illuminance than the first illuminance section. may include. A plurality of relational expressions (RLS1 to RLS5) include a first relational expression (eg, RLS1) defined for the first illuminance section and a second relational expression (eg, RLS2) defined for the second illuminance section. It can be included. When applying the first relation and the second relation to an arbitrary dynamic range compression ratio, the first relation may be defined to represent a value greater than or equal to the second relation.

In one embodiment, each of the plurality of relations RLS1 to RLS5 may include a square root term of the first dynamic range compression ratio DRC_R1. For example, IM_LV5 may represent the highest illuminance section, IM_LV1 may represent the lowest illuminance section, and IM_LV4, IM_LV3, and IM_LV2 may represent illuminance sections that sequentially decrease between IM_LV5 and IM_LV1. In this case, RLS5 could be 'sqrt(DRC_R1)', each of RLS4 and RLS3 could be '((3+ sqrt(DRC_R1))/4)', and each of RLS2 and RLS1 could be '((7+ sqrt( It may be 'DRC_R1))/8)', but this is only an example.

FIGS. 10A and 10B are diagrams illustrating an example of an operation for generating the second parameters of FIG. 1 .

Referring to FIGS. 1, 2, and 10A, in the operation of generating the second parameters (PARAM2) (S400), the application processor 500 sets a plurality of first exposure times to be inversely proportional to the size of the adjustment factor (ADJ_FCT). A plurality of second exposure times ET2s and a plurality of second sensor gains SG2s may be generated by adjusting the ET1s and the plurality of second sensor gains SG1s (S410).

In one embodiment, the first parameters PARAM1 may further include a first long exposure parameter, a first medium exposure parameter, and a first short exposure parameter, and the second parameters PARAM2 may include a second long exposure parameter. , it may further include a second medium exposure parameter and a second short exposure parameter. For example, the first long exposure parameter, the first middle exposure parameter, and the first short exposure parameter each correspond to one of the plurality of first exposure times ET1s and one of the plurality of first sensor gains SG1s. It can be found by multiplying by one. For example, the second long exposure parameter, the second medium exposure parameter, and the second short exposure parameter each correspond to one of the plurality of second exposure times ET2s and one of the plurality of second sensor gains SG2s. It can be found by multiplying by one.

Referring to FIGS. 1, 2, and 10B, in the operation of generating second parameters (PARAM2) (S400), the application processor 500 divides the first long exposure parameter by the adjustment factor (ADJ_FCT) to calculate the second parameters (PARAM2). 2 Long exposure parameters can be calculated (S411).

FIG. 11 is a block diagram showing an embodiment of the second parameter generator of FIG. 2.

Referring to FIGS. 2 and 11 , the second parameter generator 550 may include first parameter calculators 551 , 552 , and 553 and dividers 555 , 556 , and 557 . The second parameter generator 550 may perform operations S410 and S411 of FIGS. 10A and 10B.

The first parameter calculators 551, 552, and 553 include a plurality of first exposure times ET1s including set times ET11, ET12, and ET13 and set gains SG11, SG12, and SG13. The first long exposure parameter (LE_PARAM1), the first medium exposure parameter (ME_PARAM1), and the first short exposure parameter (SE_PARAM1) may be calculated based on the plurality of first sensor gains (SG1s). For example, the first parameter calculator 551 may calculate the first long exposure parameter LE_PARAM1 by multiplying the set time ET11 by the set gain SG11, and the first parameter calculator 552 may calculate the first long exposure parameter LE_PARAM1 by multiplying the set time ET11 by the set gain SG11. ) can be multiplied by the set gain (SG12) to calculate the first medium exposure parameter (ME_PARAM1), and the first parameter calculator 553 multiplies the set time (ET13) by the set gain (SG13) to calculate the first short exposure parameter ( SE_PARAM1) can be calculated.

The divider 555 divides the first long exposure parameter (LE_PARAM1) by the adjustment factor (ADJ_FCT) to generate the second long exposure parameter (LE_PARAM2), and the divider 556 adjusts the first long exposure parameter (ME_PARAM1). Dividing by the factor (ADJ_FCT) generates the second medium exposure parameter (ME_PARAM2), and the divider 557 divides the first single exposure parameter (SE_PARAM1) by the adjustment factor (ADJ_FCT) to generate the second single exposure parameter (SE_PARAM2). can be created.

In FIG. 11, an embodiment in which parameters (LE_PARAM 1, ME_PARAM1, SE_PARAM1) that may be included in the first parameter (PARAM1) are generated in the second parameter generator 550 has been described. However, in another embodiment, the parameters (LE_PARAM 1, ME_PARAM1, SE_PARAM1) may be generated in a first parameter generator (eg, 510 in FIG. 2 or 4).

FIG. 12 is a diagram to explain hardware limitations of an image sensor.

In FIG. 12, the horizontal axis represents the surrounding environment of the image device, and the vertical axis represents the parameter ratio (PARAM_R), which is the ratio of the long exposure parameter (LE_PARAM) to the short exposure parameter (SE_PARAM).

Referring to FIGS. 11 and 12 , when the surrounding environment of the imaging device is a normal environment (Normal), the parameter ratio may be PARAM_A. However, when the surrounding environment of the imaging device changes from the general environment to a backlit environment, the parameter ratio may change to PARAM_B. However, due to hardware constraints of the image sensor for obtaining HDR image data, the parameter ratio is limited to not exceed PARAM_R_CONSTRAINT in the backlight environment, and the parameter ratio is adjusted to decrease from PARAM_B to PARAM_C. In this case, the long exposure parameter (LE_PARAM) may have a maximum value, and the short exposure parameter (SE_PARAM) may rise to an unintended level to satisfy the parameter ratio. For example, each of the exposure time and/or sensor gain included in the single exposure parameter (SE_PARAM) may rise beyond an unintended level.

13 is a flowchart showing a method of operating an image sensor according to an embodiment of the present disclosure.

Referring to FIGS. 1, 2, and 13, in the operating method of the image sensor of FIG. 13, compared to the operating method of the image sensor of FIG. 1, the image sensor 100 performs second image capturing to capture the second image. Additional resulting image data can be generated (S600). Descriptions that overlap with the operating method of the image sensor in FIG. 1 will be omitted.

In one embodiment, the second image capturing may be performed after the first image capturing.

In one embodiment, the second resulting image data may include a plurality of second SDR image data and second HDR image data.

FIG. 14 is a flowchart illustrating an example of an operation for generating the second result image data of FIG. 13 . Figure 15 is a block diagram showing an imaging device according to an embodiment of the present disclosure.

Referring to FIGS. 1, 2, 13, 14, and 15, the image device 10a may include an image sensor 100a and an application processor 500a.

The image sensor 100a includes a control logic 111, a row decoder 112, a pixel array 113, a ramp generator 114, a CDS-ADC 115, a readout circuit 116, and a dynamic range compression processing circuit ( 117) and a dynamic range compression ratio generator 118.

The application processor 500a may include a first parameter generator 510, an adjustment factor generator 530, and a second parameter generator 550.

In the operation of generating the second result image data (S600), the image sensor 100 may generate a plurality of second SDR image data (S610).

The dynamic range compression ratio generator 118 may calculate the second dynamic range compression ratio DRC_R2 based on the second parameters PARAM2 and the adjustment factor ADJ_FCT (S630).

In one embodiment, the dynamic range compression ratio generator 118 may calculate the second dynamic range compression ratio (DRC_R2) based on the first dynamic range compression ratio and the adjustment factor (ADJ_FCT). The second parameters PARAM2 may include second exposure times EP2s and second sensor gains SG2s, and may further include a second long exposure parameter and a second short exposure parameter. The second long exposure parameter can be obtained by dividing the first long exposure parameter by an adjustment factor (ADJ_FCT), and the second long exposure parameter can be obtained by dividing the first long exposure parameter by an adjustment factor (ADJ_FCT), The dynamic range compression ratio generator 118 may obtain the first dynamic range compression ratio by dividing the second long exposure parameter by the second short exposure parameter and dividing the first long exposure parameter by the first short exposure parameter.

In one embodiment, the second dynamic range compression ratio (DRC_R2) may be obtained by multiplying the first dynamic range compression ratio by an adjustment factor (ADJ_FCT).

The dynamic range compression processing circuit 117 may generate second HDR image data (S650).

In one embodiment, the dynamic range compression processing circuit 117 may generate the second HDR image data based on the plurality of second SDR image data and a second dynamic range compression ratio (DRC_R2). For example, the dynamic range compression processing circuit 177 may perform merging, dividing, and tone mapping processes to generate the second HDR image data based on the second dynamic range compression ratio (DRC_R2). By applying the second dynamic range compression ratio (DRC_R2), the average brightness value of the SDR image data corresponding to the Chapter 2 exposure parameters is substantially the same as the average brightness value of the SDR image data corresponding to the Chapter 1 exposure parameters. SDR image data corresponding to exposure parameters may be generated.

FIG. 16 is a diagram for explaining operations of an image sensor according to embodiments of the present disclosure.

Referring to FIGS. 15 and 16 , the image sensor 10a may operate in sequential time sections TP1, TP2, and TP3.

During TP1, the image sensor 10a may perform second image capturing (IMG_CAPT21, IMG_CAPT22, IMG_CAPT23) based on the second parameters (PARAM2).

In one embodiment, the second parameters PARAM2 may include second exposure times EP2s and second sensor gains SG2s, and the second exposure times EP2s may be It may include setting times (EP21, EP22, and EP23), and the second sensor gains (SG2s) may include setting gains (SG21, SG22, and SG23). The second parameters PARAM2 may further include a second long exposure parameter, a second medium exposure parameter, and a second short exposure parameter. The second long exposure parameter may have a value obtained by multiplying the set time (EP21) by the set gain (SG21), and the second long exposure parameter may have a value obtained by multiplying the set time (EP22) by the set gain (SG22). And, the second single exposure parameter may have a value obtained by multiplying the set time (EP23) by the set gain (SG23).

During TP2, the image sensor 10a may read out a plurality of second SDR image data (SDR_IMG2s).

In one embodiment, the plurality of second SDR image data SDR_IMG2s may include SDR image data SDR_IMG21, SDR_IMG22, and SDR_IMG23. The image sensor 10a may generate SDR image data (SDR_IMG21) based on the second long exposure parameter and generate SDR image data (SDR_IMG22) based on the second long exposure parameter, SDR image data (SDR_IMG23) may be generated based on the second short exposure parameter.

During TP3, the image sensor 10a may perform dynamic range compression processing to generate second HDR image data (HDR_IMG2).

In one embodiment, the image sensor 10a may perform the dynamic range compression processing based on the second dynamic range compression ratio (DRC_R2).

FIGS. 17A, 17B, 18A, 18B, 19A, and 19B are diagrams for explaining the first parameters, second parameters, first dynamic range compression ratio, and second dynamic range compression ratio of FIG. 1.

FIGS. 17A and 17B show the first parameters, the second parameters, the first dynamic range compression ratio and the second dynamic range compression ratio in a high-light environment, and FIGS. 18A and 18B show the first parameters in a medium-light environment. Parameters, second parameters, first dynamic range compression ratio and second dynamic range compression ratio, and FIGS. 19A and 19B show first parameters, second parameters, first dynamic range compression ratio and second dynamic range compression ratio in a low-light environment. 2 shows the dynamic range compression ratio.

In one embodiment, the relations RLS1 to RLS5 described above with reference to FIG. 9 may be used to calculate the first and second parameters and the first and second dynamic range compression ratios. For example, in FIGS. 17A and 17B, RLS5 may be used, in FIGS. 18A and 18B, one of RLS4 and RLS3 may be used, and in FIGS. 19A and 19B, one of RLS2 and RLS1 may be used. there is.

Figure 20 is a flowchart showing a method of operating an image sensor according to an embodiment of the present disclosure.

Referring to FIGS. 2, 15, and 20, the image sensors 100 and 100a perform first image capturing to generate first result image data including a plurality of first SDR image data and first HDR image data. can be created (S1000).

The application processor 500 (500a) may generate first parameters based on the first result image data (S2000).

In one embodiment, the first parameters may be related to capturing the first image and may include a first long exposure parameter and a first short exposure parameter.

The application processor 500 (500a) may generate an adjustment factor based on the first parameters (S3000).

The application processor 500 (500a) may generate second parameters by adjusting the first parameters based on the adjustment factor (S4000).

In one embodiment, the second parameters may include a second long exposure parameter and a second short exposure parameter.

The image sensors 100 and 100a may perform second image capturing based on the second parameters (S5000).

In one embodiment, the ratio of the second long exposure parameter and the second short exposure parameter may be calculated to satisfy hardware constraints of an image sensor for obtaining HDR image data.

FIG. 21 is a diagram for explaining an example of the operation method of the image sensors of FIGS. 2 and 15.

In FIG. 21, the horizontal axis may represent time, and the vertical axis may represent rows (or lines) of the pixel array. The pixel arrays of the image sensors 100 and 100a produce SDR image data (SDR_IMG11, SDR_IMG12, SDR_IMG13) corresponding to different exposure times in sequential time sections (STP1, STP2, STP3, STP4, STP5). can be output. ET may represent the time (e.g., exposure time) for the photoelectric conversion elements of the pixel arrays to accumulate photocharges, and TT may represent the time when the SDR image data (SDR_IMG11, SDR_IMG12, SDR_IMG13) is output through a readout circuit. It can indicate the transmission time.

Referring to FIG. 21, during STP1, photocharges for SDR image data (SDR_IMG11) may be accumulated and part of the SDR image data (SDR_IMG11) may be output. During STP2, photocharges for the SDR image data (SDR_IMG11, SDR_IMG12) are accumulated and a portion of the SDR image data (SDR_IMG11, SDR_IMG12) may be output. During STP3, photocharges for SDR image data (SDR_IMG11, SDR_IMG12, SDR_IMG13) are accumulated and some of the SDR image data (SDR_IMG11, SDR_IMG12, SDR_IMG13) may be output. During STP4, photo charges for the SDR image data (SDR_IMG12, SDR_IMG13) are accumulated and a portion of the SDR image data (SDR_IMG12, SDR_IMG13) may be output. During STP5, photo charges for SDR image data (SDR_IMG13) are accumulated and a portion of SDR image data (SDR_IMG13) may be output. In this way, a method in which all or part of SDR image data corresponding to different exposure times are output simultaneously during a specific time period can be referred to as a ‘staggered method’. In this case, referring to FIGS. 2 and 15 , HDR image data may be generated by the application processors 500 and 500a rather than the image sensors 100 and 100a, and the adjustment factor ADJ_FCT is generated by the image sensor 100. , 100a) and may be used as internal signals of the application processors 500 and 500a.

Figure 22 is a block diagram showing a computing system including an image sensor according to embodiments of the present disclosure.

Referring to FIG. 22, the computing system 700 may include a processor 710, a memory device 720, a storage device 730, an image sensor 740, an input/output device 750, and a power supply 760. there is. Although not shown in FIG. 22, the computing system 700 may further include ports capable of communicating with a video card, sound card, memory card, USB device, etc., or with other electronic devices.

Processor 710 may perform specific calculations or tasks. In one embodiment, the processor 710 may be a microprocessor or a central processing unit (CPU).

The processor 710 connects the memory device 720, the storage device 730, the image sensor 740, and the input/output device 750 through an address bus, a control bus, and a data bus. You can communicate with.

In one embodiment, processor 2010 may also be connected to an expansion bus, such as a Peripheral Component Interconnect (PCI) bus.

The memory device 720 may store data necessary for the operation of the computing system 2000. For example, the memory device 720 may be implemented with DRAM, mobile DRAM, SRAM, PRAM, FRAM, RRAM, and/or MRAM. there is.

The storage device 730 may include a solid state drive, hard disk drive, CD-ROM, etc. The input/output device 750 may include input means such as a keyboard, keypad, mouse, etc., and output means such as a printer, display, etc. The power supply 760 may supply operating voltage required for operation of the computing system 700.

The image sensor 740 may be connected to the processor 2010 through the buses or other communication links to perform communication. The image sensor 740 may correspond to the image sensors 100 and 100a described above with reference to FIGS. 2 and 15 .

Figure 23 is a block diagram showing an electronic system including an image sensor according to embodiments of the present disclosure.

Referring to FIG. 23, the electronic system 1000 may be implemented as a data processing device capable of using or supporting the MIPI interface, and may include an application processor 1110, an image sensor 1140, and a display 1150. there is. The electronic system 1000 may further include an RF chip 1160, GPS 1120, storage 1170, microphone 1180, DRAM 1185, and speaker 1190, and UWB 1210 and WLAN ( Communication can be performed using WIMAX (1220), WIMAX (1230), etc.

The application processor 1110 may represent a controller or processor that controls the operations of the image sensor 1140 and the display 1150.

The application processor 1110 includes a DSI host 1111 that communicates with the DSI device 1151 of the display 1150, a CSI host 1112 that communicates with the CSI device 1141 of the image sensor 1140, and an RF chip 1160. It may include a PHY (1161), a PHY (1113) that transmits and receives data according to DigRF, and a DigRF MASTER (1114) that controls the DigRF SLAVE (1162) of the RF chip (1160).

In one embodiment, DSI host 1111 may include an optical serializer (SER) and DSI device 1151 may include an optical deserializer (DES). In one embodiment, CSI host 1112 may include an optical deserializer (DES) and CSI device 1141 may include an optical serializer (SER).

The image sensor 1040 may be an image sensor according to embodiments of the present invention, and may operate based on an operation method according to embodiments of the present invention.

In one embodiment, the electronic system 1000 includes a personal computer (PC), a workstation, a laptop, a cellular phone, a smart phone, an MP3 player, a personal digital assistant (PDA), PMP (Portable Multimedia Player), digital TV, digital camera, portable game console, navigation device, wearable device, IoT (Internet of Things) device, IoE (Internet of Everything) device, It may be an electronic system such as an e-book, VR (Virtual Reality) device, AR (Augmented Reality) device, drone, etc.

As described above, in the method of operating an image sensor according to an embodiment of the present disclosure, second image capturing may be performed based on the results of performing first image capturing. First parameters may be generated based on the results of performing the first image capturing, and second parameters for performing the second image capturing may be generated based on the first parameters. The second parameters may be calculated to satisfy hardware constraints of an image device (or image sensor) for obtaining HDR image data. Accordingly, the image device can obtain HDR image data with a sufficient dynamic range by stabilizing the brightness value of the subject and preventing saturation of the background, etc., even when the surrounding environment changes from a normal environment to a backlight environment.

The above-described contents are specific embodiments for carrying out the present disclosure. The present disclosure will include not only the above-described embodiments, but also embodiments that are simply designed or can be easily changed. Additionally, the present disclosure will also include techniques that can be easily modified and implemented using the embodiments. Therefore, the scope of the present disclosure should not be limited to the above-described embodiments, but should be determined by the claims and equivalents of this invention as well as the claims described later.

10: Image device
100: image sensor
101: pixel array
151: Capturing the first image
155: Capturing second image
500: Application processor
510: first parameter generator
530: Adjustment factor generator
550: second parameter generator

Claims (10)

performing first image capturing to generate first resulting image data including a plurality of first standard dynamic range (SDR) image data and first high dynamic range (HDR) image data;
based on the first resulting image data, generating first parameters related to capturing the first image and including a plurality of first exposure times and a plurality of first sensor gains;
generating an adjustment factor based on the first parameters;
adjusting the first parameters based on the adjustment factor to generate second parameters including a plurality of second exposure times and a plurality of second sensor gains; and
A method of operating an image sensor comprising performing second image capturing based on the second parameters. The method of claim 1, wherein generating the second parameters comprises:
Generating the plurality of second exposure times and the plurality of second sensor gains by adjusting the plurality of first exposure times and the plurality of first sensor gains to be inversely proportional to the magnitude of the adjustment factor. How an image sensor works. The method of claim 1, wherein generating the adjustment factor comprises:
Calculating a first dynamic range compression ratio based on a first set time and a second set time among the plurality of first exposure times, and a first set gain and a second set gain among the plurality of first sensor gains. ;
calculating a first illuminance based on the first set time and the first set gain; and
A method of operating an image sensor comprising calculating the adjustment factor based on the first dynamic range compression ratio, the first illuminance, and a first adjustment factor relationship. The method of claim 3, wherein the first set time is the longest among the plurality of first exposure times and the second set time is the shortest among the plurality of first exposure times,
The first set gain corresponds to the first set time, and the second set gain corresponds to the second set time. The method of claim 4, wherein the first dynamic range compression ratio is:
A method of operating an image sensor in which the value obtained by multiplying the first set time by the first set gain is divided by the value obtained by multiplying the second set time by the second set gain. The method of claim 5, wherein the size of the adjustment factor increases as the first dynamic range compression ratio increases. The method of claim 3, wherein calculating the adjustment factor comprises:
Based on the illuminance-adjustment factor table, calling the first adjustment factor relational expression corresponding to the first illuminance; and
A method of operating an image sensor comprising calculating the adjustment factor by applying the first adjustment factor relational expression to the first dynamic range compression ratio. The method of claim 7, wherein the illuminance-adjustment factor table is:
A plurality of illuminance sections dividing the illuminance range that can be expressed as the illuminance of the image sensor; and
A method of operating an image sensor including a plurality of relational expressions respectively defined for the plurality of illuminance sections. The method of claim 8, wherein the plurality of illuminance sections include a first illuminance section and a second illuminance section showing a higher illuminance intensity than the first illuminance section,
The plurality of relational expressions include a first relational expression defined for the first illuminance section and a second relational expression defined for the second illuminance section,
When applying the first relation and the second relation to the first dynamic range compression ratio, the first relation indicates a larger value than the second relation. Based on the first resulting image data comprising a plurality of first standard dynamic range (SDR) image data and first high dynamic range (HDR) image data, a plurality of first exposures associated with the first image capturing generate first parameters including times and a plurality of first sensor gains;
generate an adjustment factor based on the first parameters,
an application processor configured to adjust the first parameters based on the adjustment factor to generate second parameters including a plurality of second exposure times and a plurality of second sensor gains; and
An image device comprising an image sensor configured to perform the first image capturing and perform a second image capturing based on the second parameters.

Images