WO2016006981A1 - Method and apparatus for minimum shooting automatic exposure by using noise-aware linear model - Google Patents

Abstract

A method and an apparatus for minimum shooting automatic exposure by using a noise-aware linear model are disclosed. An automatic exposure control method provided in the present invention comprises the steps of: obtaining a first image captured in a first frame by using a first exposure time and a first amplification ratio; updating an exposure time from the first exposure time to a second exposure time; calculating a histogram with respect to the first image if the exposure time has been updated to the second exposure time; modifying the histogram with respect to the first image on the basis of saturated pixels; and extracting a target exposure time on the basis of the modified histogram with respect to the first image.

Description

Automatic Exposure Method and Device for Minimal Shot Using Noise-Aware Linear Model

The present invention relates to an automatic exposure method and apparatus for reaching a target brightness for an unspecified scene, and to an automatic exposure method and apparatus for reaching a target brightness for an unspecified scene even in a nonlinear section of an overexposure state.

Recently, with the rapid development of digital camera technology, digital cameras that support higher resolution and various functions are being commercialized. Such digital cameras are often used in other portable digital devices such as mobile phones, notebook computers, and personal digital assistants (PDAs). Such digital cameras are basically provided with an image sensor such as a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like, which captures and converts light of an object into an electrical signal. However, with the current technology, it is difficult to fundamentally block noise generated when photographing a subject by such an image sensor. Therefore, most digital cameras have a function of removing or correcting the generated noise, and the importance of such a function is increasing in consideration of the current trend of pursuing a high quality image.

SUMMARY OF THE INVENTION The present invention provides a minimum scene automatic exposure method for capturing an image having a target brightness target even in a nonlinear section of an overexposure state by appropriately adjusting both exposure time, amplification ratio, or exposure time and amplification ratio. To provide a device. In addition, by applying a noise-aware offset in consideration of noise in various illuminance environments such as outdoor backlighting, an image of a target brightness is obtained with minimal shooting.

In an aspect, the automatic exposure control method proposed by the present invention may include acquiring a first image photographed in a first frame using a first exposure time and a first amplification ratio, and adjusting the exposure time from the first exposure time. Updating to a second exposure time, estimating a histogram for the first image if the exposure time is updated to the second exposure time, and generating a histogram for the first image based on saturated pixels. And deforming a target exposure time based on the deformed histogram of the first image.

The extracting of the target exposure time may include determining whether the brightness of the first image reaches the target brightness when the second exposure time is applied based on the modified histogram.

When the exposure time is updated to the second exposure time, estimating the histogram for the first image may include estimating the histogram of the first image when the image brightness of the overexposed state is smaller than the target brightness and the image brightness of the overexposed state is the target brightness. It can be estimated by dividing by a larger case.

When the image brightness of the overexposed state is smaller than the target brightness, the exposure time may be calculated from the intensity histogram of the initial image.

When the image brightness of the overexposed state is less than the target brightness, using the following equation to estimate L1 using the histogram of the initial image,

Here, h (x) represents the histogram of the initial image.

When the image brightness in the overexposed state is greater than the target brightness, it is possible to infer a saturated pixel that decreases when the exposure time decreases from the initial exposure time.

When the image brightness in the overexposed state is greater than the target brightness, the following equation is used to estimate the saturated pixel that decreases when the exposure time is decreased from the initial exposure time,

 z = 0.1389-0.004079x-0.3513y

Here, the x axis represents the brightness of the initial image, the y axis represents the ratio of saturated pixels, and the z axis represents the exact ratio of exposure time.

In another aspect, the automatic exposure control apparatus proposed by the present invention is a photographing unit for obtaining a first image photographed in a first frame using the first exposure time and the first amplification ratio, the exposure time of the first exposure Updating from time to a second exposure time, a histogram estimator for estimating a histogram for the first image when the exposure time is updated to the second exposure time, based on the saturated pixels The apparatus may include a histogram transform unit configured to modify a histogram for an image, and a target exposure time extractor configured to extract a target exposure time based on the modified histogram for the first image.

The target exposure time extractor may determine whether the brightness of the first image reaches a target brightness when the second exposure time is applied based on the modified histogram.

The histogram estimating unit may estimate by dividing the case in which the image brightness in the overexposed state is smaller than the target brightness and the case in which the image brightness in the overexposed state is larger than the target brightness.

In one aspect, the minimal scene automatic exposure method proposed in the present invention comprises the steps of photographing a first image taken by using a first frame having a predetermined exposure time and amplification ratio, calculating the initial brightness of the first frame Comparing the difference between the initial brightness and the target brightness with a predetermined error margin; when the difference between the initial brightness and the target brightness is greater than the predetermined error margin, selecting an exposure time and an amplification ratio mode; If the difference between the initial brightness and the target brightness is less than the predetermined error margin, repeating the step of photographing the image, compensating for the initial brightness by using an offset value considering noise according to the exposure time and the amplification ratio mode. And applying the compensated exposure time and amplification ratio to a second frame and using the second frame. Shooting by a second image, and may include the step of repeating from the step of recording the image.

When the difference between the initial brightness and the target brightness is greater than the predetermined error margin, selecting the exposure time and the amplification ratio mode may include: shutter priority mode for adjusting exposure time, amplification ratio priority mode for adjusting amplification ratio, and exposure time And a mixing mode for adjusting the amplification ratio.

Compensating the initial brightness by using an offset value considering noise according to the selected exposure time and amplification ratio mode, using the following equation to compensate for the initial brightness,

L _{i + 1} -N _{i + 1} = S _{i + 1} / S _i -G _{i + 1} / G _i (L _i -N _i )

Here, L brightness, G may represent an amplification ratio, S may represent an exposure time, and N may represent an offset value considering noise. In addition, by applying an offset value considering the noise in an illumination environment including an indoor environment, an outdoor environment, a backlight, and a lighting condition, brightness may be compensated with only minimal shooting.

In the step of taking the image from the step of repeating, the image having the target brightness may be obtained through the repeated shooting of up to three times.

According to another aspect, the minimum scene automatic exposure apparatus proposed by the present invention is a photographing unit for taking a first image photographed using a first frame having a predetermined exposure time and amplification ratio, the initial of the first frame A calculator for calculating a brightness, a comparison unit for comparing the difference between the initial brightness and the target brightness with a predetermined error margin, an exposure time and an amplification ratio mode when the difference between the initial brightness and the target brightness is greater than the predetermined error margin. A mode selection unit for selecting a brightness controller for compensating initial brightness by using an offset value considering noise based on the exposure time and amplification ratio mode, and applying an exposure time and an amplification ratio of the compensated brightness to a second frame The controller may include a controller configured to control the photographing unit to photograph the second image by using the second frame.

The mode selector may cause the photographing unit to photograph an image when the difference between the initial brightness and the target brightness is smaller than the predetermined error margin.

The mode selector may select one of the exposure time and amplification ratio modes including a shutter priority mode for adjusting exposure time, an amplification ratio priority mode for adjusting amplification ratio, and a mixing mode for adjusting exposure time and amplification ratio.

The brightness adjusting unit compensates for the initial brightness using the following equation,

L _{i + 1} -N _{i + 1} = S _{i + 1} / S _i -G _{i + 1} / G _i (L _i -N _i )

Here, L brightness, G may represent an amplification ratio, S may represent an exposure time, and N may represent an offset value considering noise.

The brightness adjusting unit may compensate for brightness with only minimal shooting by applying an offset value considering the noise in an illumination environment including an indoor environment, an outdoor environment, a backlight, and a constant lighting condition.

The controller may control to capture an image by using a target brightness obtained through at least three repetitive shots.

According to embodiments of the present invention, the exposure time, the amplification ratio, or both the exposure time and the amplification ratio may be appropriately adjusted to capture an image having a target target brightness even in a non-linear section in an overexposure state. In addition, by applying a noise-aware offset in consideration of noise in various illuminance environments such as outdoor backlighting, an image having a target brightness may be obtained with minimal shooting.

1 is a flowchart illustrating a method for automatically minimizing photographing using a noise recognition linear model according to an exemplary embodiment of the present invention.

2 is a graph showing a relationship between exposure time and brightness according to an embodiment of the present invention.

3 is a graph showing a relationship between amplification ratio and brightness at zero exposure time according to an embodiment of the present invention.

4 is a graph illustrating a relationship between an amplification ratio and brightness according to a change in exposure time at a fixed brightness of a scene according to an embodiment of the present invention.

5A is a flowchart illustrating a method for automatically minimizing scene exposure according to an embodiment of the present invention.

5B is a diagram illustrating a configuration of a minimum scene automatic exposure apparatus according to an embodiment of the present invention.

5C is a diagram illustrating a relationship between brightness and exposure time in an overexposed state according to an embodiment of the present invention.

6 is a diagram illustrating a histogram of brightness of an image at each exposure time according to an embodiment of the present invention.

7 is a diagram illustrating a brightness histogram of an image for each exposure time according to an embodiment of the present invention.

8 is a diagram illustrating an intensity histogram and an extended histogram of an initial image, according to an exemplary embodiment.

9 is a graph showing exposure time and brightness according to an embodiment of the present invention.

FIG. 10 is a diagram illustrating a minimum photographing auto exposure apparatus using a noise recognition linear model according to one embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating a method for automatically minimizing photographing using a noise recognition linear model according to an exemplary embodiment of the present invention.

In the method of minimizing photographing automatic exposure using a noise-aware linear model, acquiring a first image photographed in a first frame using a first exposure time and a first amplification ratio (110), and extracting the exposure time from the first exposure time. Updating 120 to the second exposure time; estimating a histogram for the first image 130 when the exposure time is updated to the second exposure time; and based on the saturated pixels Deforming the histogram for the first image (140), extracting a target exposure time based on the modified histogram for the first image (150).

In operation 110, a first image captured in the first frame may be acquired using the first exposure time and the first amplification ratio. The first image photographed using the first frame having the predetermined exposure time and the amplification ratio may be photographed. For example, an image can be captured continuously to apply an automatic exposure algorithm to a video camera, the brightness of which must be calculated every frame. In this case, the initial brightness of the first frame may be calculated.

In step 120, the exposure time may be updated from the first exposure time to the second exposure time.

In operation 130, when the exposure time is updated to the second exposure time, a histogram for the first image may be estimated.

When the exposure time is updated to the second exposure time, estimating the histogram for the first image includes: when the image brightness of the overexposed state is less than the target brightness and the image brightness of the overexposed state is the target It can be estimated by dividing by the case of greater than brightness.

If the image brightness in the overexposed state is smaller than the target brightness, the exposure time can be calculated from the intensity histogram of the initial image. When the image brightness in the overexposed state is smaller than the target brightness, Equation 12 is used to estimate L1 using the histogram of the initial image, where h (x) may represent the histogram of the initial image. When the image brightness in the overexposed state is greater than the target brightness, it is possible to infer a saturated pixel that decreases when the exposure time decreases from the initial exposure time.

In step 140, the histogram for the first image may be modified based on the saturated pixels.

If the image brightness in an overexposure state is greater than the target brightness, Equation 17 is used to estimate the saturated pixel that decreases as the exposure time decreases from the initial exposure time, where the x axis is the brightness of the initial image, y The axis can represent the ratio of saturated pixels and the z axis can represent the exact ratio of exposure time.

In operation 150, a target exposure time may be extracted based on the modified histogram of the first image. The extracting of the target exposure time may include determining whether the brightness of the first image reaches the target brightness when the second exposure time is applied based on the modified histogram. It will be described in more detail with reference to Figures 2 to 10.

2 is a graph showing a relationship between exposure time and brightness according to an embodiment of the present invention.

Since the exposure time determines the amount of light incident on the lens of the camera, it can be predicted that the brightness of the image is proportional to the exposure time. This relationship can be expressed as in Equation (1).

      Equation (1)

L _{i + 1} = S _{i + 1} / S _i * L _i

Where S is the exposure time, L is the brightness of the image, and i is the frame index.

For example, in order to show the validity of Equation (1), a color checker is photographed using a PointGrey Flea3 camera as shown in FIG. In fixed light conditions, the exposure time is increased from 0ms to 100ms, and the amplification ratio is increased from 1 to 5 times. In order to measure the brightness of the captured image, an RGB-YUV conversion is performed on the RGB values of all the pixels in the image, and then the average of the Y values of all the pixels is obtained. As shown in FIG. 2 (b), it can be seen that the linearity between the brightness of the image and the exposure time for a fixed value of various amplification ratios is clearly maintained.

Ideally, if the exposure time is 0ms, there is no light coming through the lens, so the brightness value of the image should also be zero. However, as shown in FIG. 2 (b), the brightness value of the image obtained through the exposure time of 0 ms is not zero. In other words, there is an element within the camera that maintains a fixed brightness of the image. In particular, electrons and holes irregularly generated in a depletion region of a device such as an optical device may cause dark current when there is no light incident on the optical device. This is called dark current noise, which is relatively small in size, but is one of the major sources of noise in image sensors. The brightness appearing in the image of the 0 ms exposure time may be mainly due to noise generated irrespective of the brightness of the pixel, such as dark current noise, read-out noise, and quantization noise. Since the dark current noise may be represented by the number of photons, the brightness value resulting from the dark current noise may be affected by the amplification ratio of the camera.

3 is a graph showing a relationship between amplification ratio and brightness at zero exposure time according to an embodiment of the present invention.

In order to confirm the influence of the amplification ratio in the image having an exposure time of 0 ms, the image is taken while changing the amplification ratio while maintaining the exposure time of 0 ms as shown in FIG. For example, the brightness of images obtained from various image sensors such as TOSHIBA CIS sensor 330, three types of PointGrey Flea3 cameras 310, 320, 360, and two types of PointGrey Grasshopper cameras 340, 350, and You can see the linear relationship between the amplification ratios of the cameras. The offset brightness at the exposure time of 0 ms can be expressed as in Equation (2).

       Equation (2)

N = c + a * G

Where G is the amplification ratio, N is the offset brightness at 0 ms exposure time, and c and a are constant values, which differ from camera to camera.

In conclusion, when the illuminance and the amplification ratio of the specific environment are constant, the relationship between the exposure time and the brightness of the image may be expressed as in Equation (3).

      Equation (3)

L _{i + 1} -N = S _{i + 1} / S _i- (L _i -N)

After taking an image through an initial exposure time in a general situation, an image having a target brightness value may be obtained by adjusting the exposure time again using Equation 3.

The amplification ratio of the camera is a parameter that determines how many times the amount of incident photons are amplified to determine the intensity of the image. Therefore, the amplification ratio is always linearly related to the brightness of the image. The relationship between the brightness and the amplification ratio of such an image can be expressed as in Equation (4).

       Equation (4)

L _{i + 1} = G _{i + 1} / G _i * L _i

Gi is the amplification ratio of the i-th image taken. As described above, to check Equation (4), a color checker is photographed using the same camera in a fixed illuminance environment as shown in FIG. At this time, the amplification ratio was increased from 1 to 8 times, the exposure time was increased from 40ms to 80ms. Thereafter, brightness values of the photographed images were measured as shown in FIG. 3.

4 is a graph illustrating a relationship between an amplification ratio and brightness according to a change in exposure time at a fixed brightness of a scene according to an embodiment of the present invention.

The camera used in this experiment (Flea3) could not take images at 0x amplification ratio, so it is possible to extend the line connecting the brightness values from 1x to 8x to adjust the brightness value of the image at 0x amplification ratio. Obtain As shown in Figure 4, it can be seen that the linearity between the amplification ratio and the brightness value of the image is maintained. As described above, it can be seen that the brightness value of the image obtained at the 0x amplification ratio is also not zero. At this time, unlike using a fixed amplification ratio in Equation (2), it is necessary to reflect the change in the amplification ratio in the equation. Therefore, Equation (2) may be modified as in Equation (5).

       Equation (5)

N _i = c + a * G _i

By considering such a cancellation value Ni, when the illuminance and the exposure time in a specific environment are fixed, the relationship between the brightness of the image and the amplification ratio can be represented. If the brightness value of the pure image is obtained by subtracting the offset value, the brightness value of the image may be expressed as in Equation (6).

      Equation (6)

L _i = L ^net _i + N _i

Then, since Linet has a linear relationship with the amplification ratio, Equation 4 may be represented as Equation 7.

      Equation (7)

L ^net _{i + 1} = G _{i + 1} / G _i * L ^net _i

In conclusion, the relationship between the amplification ratio and the brightness of the image may be expressed as Equation (8) in consideration of the offset value Ni.

      Equation (8)

L _{i + 1} -N _{i + 1} = G _{i + 1} / G _i * (L _i -N _i )

After taking an image through an initial amplification ratio in a general situation, an image having a target brightness value may be obtained by adjusting the amplification ratio again using Equation 8.

Equations (3) and (8), which adjust exposure time and amplification ratio, respectively, can be integrated into a single equation. Then, for an image given the initial exposure time Si and the amplification ratio Gi, an appropriate exposure time Si + 1 for the target brightness value Li + 1 can be determined. Then, in order to determine an appropriate amplification ratio, the output brightness Li + 1 in Equation (9) may be substituted for the initial brightness Li in Equation (8), and may be expressed as Equation (10).

      Equation (9)

L _{i + 1} -N _i = S _{i + 1} / S _i * (L _i -N _i )

In addition, another integrated equation can be obtained by considering the exposure time that follows by determining the amplification ratio. By applying equation (8) to equation (3), the same relationship as in equation (10) can be obtained.

  Equation (10)

L _{i + 1} = G _{i + 1} / G _i * S _{i + 1} / S _i * (L _i -N _i )

Then, the equation representing the linear relationship of the brightness compensated by the offset value considering the noise with the exposure time and the amplification ratio can be rewritten. This may be expressed as in Equation (11).

 Equation (11)

L _{i + 1} -N _{i + 1} = G _{i + 1} / G _i * S _{i + 1} / S _i * (L _i -N _i )

Here, L may represent a brightness value, G may represent an amplification ratio, S may represent an exposure time, and N may represent an offset value considering noise. Equation (11) is a special case, and may explain other relations in Equations (3) and (8). When the exposure time is equal to Si without changing to Si + 1, Equation (11) may be represented by Equation (8). In this case, the amplification ratio is fixed similarly to equation (3). Based on the last relationship of equation (11), various combinations of exposure time and amplification ratio for target brightness can be expected after one frame image taking any initial exposure time and amplification ratio. A flowchart of this proposed method is shown in FIG. 5.

5A is a flowchart illustrating a method for automatically minimizing scene exposure according to an embodiment of the present invention.

The proposed minimum scene automatic exposure method includes the steps of capturing a first image photographed using a first frame having a predetermined exposure time and amplification ratio (410), and calculating an initial brightness of the first frame (420). Comparing the difference between the initial brightness and the target brightness with a predetermined error margin (430); when the difference between the initial brightness (Li) and the target brightness (Ltarget) is greater than the predetermined error margin (T), exposure Selecting a time and amplification ratio mode (441), compensating for initial brightness using an offset value considering noise according to the exposure time and amplification ratio mode (450), exposure time and amplification ratio of the compensated brightness May be applied to a second frame, photographing a second image by using the second frame, and repeating from capturing the image (460).

In operation 410, a first image photographed using the first frame having a predetermined exposure time and amplification ratio may be photographed. For example, an image can be captured continuously to apply an automatic exposure algorithm to a video camera, the brightness of which must be calculated every frame.

In operation 420, an initial brightness of the first frame may be calculated.

In operation 430, the difference between the initial brightness Li and the target brightness Ltarget may be compared with a predetermined error margin T. The margin error T may be selected as shown in Table 1.

margin	condition			Flea 3	AE w / o N	AE with N
One%	indoor	normal	6	2.8	One
		backlit	6	5.6	3.2
		frontlit	6.57	5.4	2.8
	outdoor	normal	9.25	4.5	3
		backlit	10	6	4.6
		frontlit	9.8	5.5	3.6
5%	indoor	normal	3.6	2	One
		backlit	4.37	3	2.3
		frontlit	5.71	3	1.6
	outdoor	normal	7.5	3.7	2.3
		backlit	7.4	4	3
		frontlit	8.8	4.3	2.7

When the difference between the initial brightness Li and the target brightness Ltarget is greater than the predetermined error margin T, in step 441, an exposure time and an amplification ratio mode may be selected. Then, an appropriate amplification ratio and exposure time for achieving the target brightness may be estimated based on Equation (11). Exposure time and amplification ratio modes include shutter-priority to adjust exposure time, gain-priority to adjust amplification ratio, and mixed mode to adjust exposure time and amplification ratio. shutter), and one of the exposure time and amplification ratio mode can be selected. The appropriate exposure time or amplification ratio for the scene can be determined based on the selected mode and can take the next frame with the determined value.

On the other hand, if the difference between the initial brightness (Li) and the target brightness (Ltarget) is less than the predetermined error margin (T), it can be repeated from the step of photographing the image in step 442. In this case, the initial exposure time Si and the initial amplification ratio Gi value may be maintained as they are, and the order i of the captured images may be replaced by i + 1. In this way, it is possible to determine whether more variation in exposure time or amplification ratio is needed by comparing the current brightness and the target brightness.

In operation 450, the initial brightness may be compensated by using an offset value considering noise according to the exposure time and the amplification ratio mode. In this case, the initial brightness is compensated by using Equation (11), and the offset value considering the noise is applied at various illumination environments including indoor environment, outdoor environment, backlit, and frontlit state. Brightness can be compensated by only shooting with.

In operation 460, the exposure time and the amplification ratio of the compensated brightness may be applied to the second frame, the second image may be photographed using the second frame, and the image may be repeated from the photographing of the image. In this case, the initial exposure time Si, the initial amplification ratio Gi, and the order i value of the photographed image may be replaced with Si + 1, Gi + 1, and i + 1, respectively.

Through this process, the proposed method can obtain an image of a target brightness with only one previous frame in a general illumination environment that does not provide excessive saturation or dark pixels. In addition, the brightness can be compensated with only minimal shooting by applying an offset value considering the noise in various illumination environments including an indoor environment, an outdoor environment, a backlit, and a frontlit state.

5B is a diagram illustrating a configuration of a minimum scene automatic exposure apparatus according to an embodiment of the present invention.

The proposed minimum scene auto exposure apparatus 500 may include a photographing unit 510, a calculating unit 520, a comparator 530, a mode selecting unit 540, a brightness adjusting unit 550, and a controller 560. have.

The photographing unit 510 may photograph the first image photographed using the first frame having a predetermined exposure time and amplification ratio. For example, an image can be captured continuously to apply an automatic exposure algorithm to a video camera, the brightness of which must be calculated every frame.

The calculator 520 may calculate the initial brightness of the first frame.

The comparator 530 may compare the difference between the initial brightness and the target brightness with a predetermined error margin. The margin error T may be selected as shown in Table 1.

The mode selector 540 may select an exposure time and an amplification ratio mode when the difference between the initial brightness Li and the target brightness Ltarget is greater than the predetermined error margin T. In this case, one of shutter-priority adjusting the exposure time, gain-priority adjusting the amplification ratio, and mixed gain-shutter adjusting the exposure time and amplification ratio can be selected. You can choose. On the other hand, when the difference between the initial brightness Li and the target brightness Ltarget is smaller than the predetermined error margin T, the mode selector 540 may cause the photographing unit 510 to take an image again. have. In this case, the initial exposure time Si and the initial amplification ratio Gi value may be maintained as they are, and the order i of the captured images may be replaced by i + 1. In this way, it is possible to determine whether more variation in exposure time or amplification ratio is needed by comparing the current brightness and the target brightness.

The brightness controller 550 may compensate for the initial brightness by using an offset value considering noise based on the exposure time and the amplification ratio mode. In this case, the initial brightness is compensated by using Equation (11), and the offset value considering the noise is applied at various illumination environments including indoor environment, outdoor environment, backlit, and frontlit state. Brightness can be compensated by only shooting with.

The controller 560 may apply the compensated exposure time and the amplification ratio to the second frame, and control the photographing unit to photograph the second image using the second frame. In this case, the initial exposure time Si, the initial amplification ratio Gi, and the order i value of the photographed image may be replaced with Si + 1, Gi + 1, and i + 1, respectively.

The proposed minimum scene auto exposure apparatus 500 may obtain a target brightness using only one previous frame in a general illumination environment that does not provide server saturation or dark pixels by performing the aforementioned method. In addition, the brightness can be compensated with only minimal shooting by applying an offset value considering the noise in various illumination environments including an indoor environment, an outdoor environment, a backlit, and a frontlit state.

Hereinafter, the effects of the proposed minimum scene automatic exposure method and apparatus are verified through experiments. To confirm that the target brightness has been reached by estimating the appropriate exposure time and amplification ratio, shutter-priority and amplification ratio are adjusted to compare the exposure time by comparing Ltarget and Li + 1 as shown in Table 2. Three modes of adjusting the gain-priority mode to adjust, the exposure time and the mixed mode to adjust the amplification ratio were tested. At this time, initial value Si = 30ms, Gi = 1, Li = 32.6.

	S i + 1	g i + 1	L target	L i + 1
Gain-prioritymode	60.69	One	60	60.04
	105.5	One	100	100.13
	172.71	One	160	159.3
Shutter-prioritymode	30	2.02	60	60.36
	30	3.51	100	100.3
	30	5.75	160	159.77
Mixed mode	52.74	2	100	100.14
	60	1.76	100	100.54
	90	1.17	100	100.8
			error (%)	0.352

From the initial state, it is possible to take an autoexposure image that adjusts the estimated exposure time (in shutter-priority mode) or the estimated gain-priority mode or both the estimated exposure time and in mixed mode (in mixed mode). have. The brightness obtained at Li + 1 can be calculated using only one previous frame. For all experiments involving three modes, the brightness of the auto-exposed image using the proposed minimum scene auto exposure method can be close to the target brightness as shown in Table 2.

The minimum number of frames for autoexposure is particularly important for dynamic lighting changes. The prior art does not consider the offset value considering this noise. However, the proper exposure time or amplification ratio was estimated based on the proportional equation. Table 1 shows a comparison of the number of images required for autoexposure between Flea3, and the autoexposure method with offset value N considering noise and the offset value N considering noise without automatic exposure method. In other words, Table 1 shows the number of repetitive shots required for the Flea3 camera to perform autoexposure to achieve the target brightness, and the repetition required to reach the target brightness for the case where the proposed method does not consider the noise-based cancellation value The number of shots was compared.

In an indoor environment of general illumination, two or more frames are required when no offset value considering noise is applied, whereas the proposed method can reach the target brightness using only one frame. In the case of receiving backlight and constant lighting, it may generally require more frames to deal with saturated pixels caused by direct light. In the prior art, more frames may be required, but by applying an offset value considering noise, the target brightness can be obtained using only two or three frames. In outdoor conditions, the average of the frames may be larger than indoor conditions because the saturated portion of the scene is more dominant. Nevertheless, it requires a minimum frame for the target brightness by applying an offset value considering noise.

 The prior art technique is based on a direct equation and would have required more frames for a target brightness similar to the result of auto exposure (AE) without the application of noise-compensated offsets. If the margin for target brightness is increased by 5%, the proposed method can take accurate exposed images through repeated shooting within 3 frames.

In summary, the linear model considering the proposed noise makes it possible to reach the target brightness by using a single frame even in a general illumination environment, a backlight and a constant light state, even if a few more frames are required. This simple and effective algorithm can be applied to H / W for memory reduction compared to a lookup table based on the prior art, and is not complicated compared to the optimization based on the prior art. In addition, three operation modes are proposed in which the exposure time and the amplification ratio can be separated or the exposure time and the amplification ratio can be used simultaneously in performing the automatic exposure. For each mode, it was confirmed that an image accurately exposed at the target brightness was obtained with a minimum of scenes. Based on the proposed algorithm, it is possible to develop a camera system that can capture more realistic images by applying the selection mode according to the image environment.

5C is a diagram illustrating a relationship between brightness and exposure time in an overexposed state according to an embodiment of the present invention.

The linearity between the brightness of the image and the exposure time or gain can be guaranteed to have nearly saturated pixels as shown in FIGS. 2 and 4. When there are many saturated pixels in the image as in Fig. 5 (a), the relationship between the brightness and the exposure time is not linear as in 5 (b).

6 is a diagram illustrating a histogram of brightness of an image at each exposure time according to an embodiment of the present invention.

FIG. 6 shows intensity histograms with various exposures (40 ms, 60 ms, 120 ms) as shown in FIG. 5 (b). While the exposure time increases from 40 ms in FIG. 6 (a) to 60 ms in FIG. 6 (b), the variation in intensity distribution can be increased. An intensity histogram for an exposure time of 120 ms can be expected as shown in Fig. 6 (c), but the actual histogram of 120 ms is saturated as shown in Fig. 6 (d) due to the limitation of an 8-bit image. This can lead to nonlinearities in brightness and exposure time.

7 is a diagram illustrating a brightness histogram of an image for each exposure time according to an embodiment of the present invention.

To address the nonlinearity of brightness and exposure time, two cases of overexposed conditions can be separated, as shown in FIG. 7. Case 1 means that the image brightness is smaller than the target brightness when the image is in an overexposed condition. Case2 is in reverse saturation, where the image brightness is greater than the target brightness. For these two cases, before applying the noise-recognition linear relationship to Equation 9, additional techniques for case1 and case2 can be proposed to reduce the mismatch between targets L and L1 caused by nonlinearity. Case1 may be generated when the brightness of the image is smaller than the target brightness. In this case, the appropriate exposure time can be calculated directly from the intensity histogram of the initial image. Based on Equation 12, S1 of FIG. 7 can be estimated.

     Equation 12

L _{i + 1} -N _{i + 1} = G _{i + 1} / G _i * S _{i + 1} / S _i * (L _i -N _i )

8 is a diagram illustrating an intensity histogram and an extended histogram of an initial image, according to an exemplary embodiment.

When S1 is applied to the next frame, the brightness L1 of the image may be smaller than the target brightness. Therefore, before applying S1 to the next frame, L1 can be estimated in advance. The histogram of the image taken at the initial exposure time may be defined as h (x) in FIG. 8 (a).

When increasing the exposure time to S1, the changed histogram bin x0 can be approximated from x.

       Equation 13

x '= (x-N) * S + N

x '= 255 (if x'> 255, x '= 255)

This may be represented as h0 (x0) in FIG. 8 (b). N is the noise model proposed in Equation 2, and S means S1 = Sinit. These pixels inferred to X0 can be larger than 255 and can be assumed to be truncated at 255 because of saturation. In Fig. 8 (b), L1 can be estimated from h (0).

     Equation 14

By replacing x0 in Equation 10, Equation 14 can be obtained.

     Equation 15

By using Equation 15, L1 can be estimated using the histogram of the initial image. k is the maximum value of x not exceeding 255 when x increases by S1 = Sinit time, and k can be inferred.

9 is a graph showing exposure time and brightness according to an embodiment of the present invention.

        Equation 16

k = floor ((255-N) / S + N)

Experimentally expected L1 shows about 0.5% error compared to the brightness of images taken with S1. We need more steps to find the right exposure time, SOPT, to accurately shoot the exposed image. As shown in FIG. 9, we can find S2 for reaching the target brightness in a straight line that includes both (Sinit, Linit) and (S1, L1).

L2 can also be estimated through equation (15). By repeating this sequence n times, if the expected Ln equals Ltarget, Sn can be defined as Sopt, and we can apply Sopt to the next frame. n generally does not exceed 3; In conclusion, if the brightness of the initial image is less than the target brightness in the overexposed state, then the proposed method can be used to take an accurate exposed image immediately after the initial frame.

In case2, when we shoot an image with overexposure, the image brightness may be greater than the target brightness. Unlike case 1, one can infer how many saturated pixels decrease as the exposure time decreases from the initial exposure time. As the exposure time decreases, it is difficult to predict pixel values where each saturated pixel decreases or maintains its value.

The proposed method for Case2 empirically estimates the ratio of exposure time to the ratio of saturated pixels and the brightness. A database can be obtained from various scenes by estimating the appropriate ratio of exposure time to the ratio of the brightness of a given initial image and the ratio of saturated pixels.

In particular, the embodiment of the present invention collected about 200 data through experiments in various atmospheres, such as three-dimensional plot data and overexposure conditions, the results can be expressed as follows.

        Equation 17

z = 1.389-0.004079x-0.3513y

The ratio of exposure time can be determined from the equation. In the next frame, the adjusted exposure time, which can be calculated by multiplying the initial exposure time by the reduction ratio determined from Equation 14, can be applied.

The R-square of equation 17 for the plotted points is 0: 8217. Equation 17 may not include all the conditions including excessive exposure. Case 2 shows that the experiment requires less iterations to capture an image to reach the target brightness than using the conventional autoexposure algorithm.

FIG. 10 is a diagram illustrating a minimum photographing auto exposure apparatus using a noise recognition linear model according to one embodiment of the present invention.

The automatic exposure control apparatus may include a photographing unit 1110, an update unit 1120, a histogram estimating unit 1130, a histogram deforming unit 1140, and a target exposure time extracting unit 1150.

The photographing unit 1110 may acquire the first image photographed in the first frame by using the first exposure time and the first amplification ratio. The first image photographed in the first frame may be acquired using the first exposure time and the first amplification ratio. The first image photographed using the first frame having the predetermined exposure time and the amplification ratio may be photographed. For example, an image can be captured continuously to apply an automatic exposure algorithm to a video camera, the brightness of which must be calculated every frame. In this case, the initial brightness of the first frame may be calculated.

The updater 1120 may update the exposure time from the first exposure time to the second exposure time.

The histogram estimator 1130 may estimate a histogram of the first image when the exposure time is updated to the second exposure time. When the exposure time is updated to the second exposure time, when the histogram for the first image is estimated when the image brightness of the overexposed state is less than the target brightness, and the image brightness of the overexposed state is less than the target brightness. It can be estimated by dividing into large cases.

If the image brightness in the overexposed state is smaller than the target brightness, the exposure time can be calculated from the intensity histogram of the initial image. When the image brightness in the overexposed state is smaller than the target brightness, Equation 12 is used to estimate L1 using the histogram of the initial image, where h (x) may represent the histogram of the initial image. When the image brightness in the overexposed state is greater than the target brightness, it is possible to infer a saturated pixel that decreases when the exposure time decreases from the initial exposure time.

The histogram transform unit 1140 may transform the histogram for the first image based on the saturated pixels.

If the image brightness in an overexposure state is greater than the target brightness, Equation 17 is used to estimate the saturated pixel that decreases as the exposure time decreases from the initial exposure time, where the x axis is the brightness of the initial image, y The axis can represent the ratio of saturated pixels and the z axis can represent the exact ratio of exposure time.

The target exposure time extractor 1150 may extract the target exposure time based on the modified histogram of the first image. The target exposure time extractor 1150 may determine whether the brightness of the first image reaches the target brightness when the second exposure time is applied based on the modified histogram.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable arrays (FPAs), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of explanation, one processing device may be described as being used, but one of ordinary skill in the art will appreciate that the processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims (15)

1. In the automatic exposure control method,

   Obtaining a first image captured in the first frame using the first exposure time and the first amplification ratio;

   Updating the exposure time from the first exposure time to the second exposure time;

   Estimating a histogram for the first image when the exposure time is updated to the second exposure time;

   Modifying a histogram for the first image based on saturated pixels; And

   Extracting a target exposure time based on the modified histogram for the first image

   Automatic exposure control method comprising a.
2. The method of claim 1,

   Extracting the target exposure time

   Determining whether the brightness of the first image reaches a target brightness when the second exposure time is applied based on the modified histogram

   Automatic exposure control method comprising a.
3. The method of claim 1,

   If the exposure time is updated to the second exposure time, estimating the histogram for the first image,

   And estimating when the image brightness in the overexposed state is smaller than the target brightness and the case in which the image brightness in the overexposed state is larger than the target brightness.
4. The method of claim 3,

   And when the brightness of the image in the overexposed state is less than a target brightness, calculating the exposure time from the intensity histogram of the initial image.
5. The method of claim 3,

   When the image brightness of the overexposed state is less than the target brightness, using the following equation to estimate L1 using the histogram of the initial image,

   Wherein h (x) represents a histogram of the initial image.
6. The method of claim 3,

   And when the image brightness in the overexposure state is greater than a target brightness, inferring a saturated pixel that decreases when the exposure time decreases from the initial exposure time.
7. The method of claim 3,

   When the image brightness in the overexposed state is greater than the target brightness, the following equation is used to estimate the saturated pixel that decreases when the exposure time is decreased from the initial exposure time,

   Here, the x-axis represents the brightness of the initial image, the y-axis represents the ratio of saturated pixels, the z-axis represents the exact ratio of the exposure time.
8. In the automatic exposure control device,

   A photographing unit obtaining a first image photographed in a first frame using a first exposure time and a first amplification ratio;

   An updater for updating the exposure time from the first exposure time to the second exposure time;

   A histogram estimating unit estimating a histogram for the first image when the exposure time is updated to the second exposure time;

   A histogram modifying unit configured to modify a histogram for the first image based on saturated pixels; And

   A target exposure time extracting unit extracting a target exposure time based on the modified histogram of the first image

   Automatic exposure control device comprising a.
9. The method of claim 8,

   The target exposure time extraction unit,

   And determining whether the brightness of the first image reaches a target brightness when the second exposure time is applied based on the modified histogram.
10. The method of claim 8,

    The histogram collecting unit,

    And estimating when the image brightness in the overexposed state is smaller than the target brightness and the case in which the image brightness in the overexposed state is larger than the target brightness.
11. The method of claim 10,

    And when the brightness of the image in the overexposed state is smaller than a target brightness, calculating the exposure time from the intensity histogram of the initial image.
12. The method of claim 10,

    And when the image brightness in the overexposed state is greater than a target brightness, inferring saturated pixels that decrease when the exposure time decreases from the initial exposure time.
13. In the minimum scene auto exposure device,

    A photographing unit which photographs a first image photographed using a first frame having a predetermined exposure time and an amplification ratio;

    A calculator configured to calculate an initial brightness of the first frame;

    A comparison unit comparing the difference between the initial brightness and the target brightness with a predetermined error margin;

    A mode selection unit for selecting an exposure time and an amplification ratio mode when the difference between the initial brightness and the target brightness is greater than the predetermined error margin;

    A brightness controller for compensating initial brightness using an offset value considering noise based on the exposure time and amplification ratio mode; And

    A control unit which applies the compensated exposure time and amplification ratio to the second frame and controls the photographing unit to photograph the second image by using the second frame

    Minimal scene auto exposure device comprising a.
14. The method of claim 13,

    The mode selector,

    If the difference between the initial brightness and the target brightness is less than the predetermined error margin, the photographing unit to take the image again

    Minimal scene autoexposure.
15. The method of claim 13,

    The mode selector,

    Selecting one of the exposure time and amplification ratio modes including a shutter priority mode for adjusting the exposure time, an amplification ratio priority mode for adjusting the amplification ratio, and a blending mode for adjusting the exposure time and amplification ratio

    Minimal scene autoexposure.

Images