JP5621335B2 - Image processing apparatus, image processing method, and image pickup apparatus - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for adjusting a dynamic range in an image sensor, and an image imaging apparatus including the image processing apparatus.

  The dynamic range in the image sensor represents the ratio of the minimum object illuminance and the maximum illuminance that can be expressed in one image. In general, by moving the dynamic range in parallel with the luminance axis by adjusting the lens aperture and shutter speed, it is possible to shoot from a bright subject to a dark subject with appropriate gradation. An electronic shutter that adjusts the exposure time by controlling the charge accumulation time of the image sensor instead of actually changing the shutter speed is also widely used.

  However, when photographing a subject whose luminance difference between the bright part and the dark part is larger than the dynamic range of the image sensor, the bright part is overexposed and the dark part is overshadowed.

  In order to avoid this, conventionally, a brightness histogram of the imaging frame F (i) is generated to determine the brightness of the imaging frame F (i), and an optimal exposure time is set according to whether the imaging frame F (i) is too bright or too dark. It is known to adjust the luminance of the next imaging frame (i + 1). For example, in Patent Document 1, an optimum gain and an optimum gamma correction value are set in addition to the exposure time to obtain an image with a wide dynamic range.

  In the method of setting the optimum exposure time and gamma correction value from the luminance histogram of the conventional imaging frame F (i) as described above, the exposure time and gamma correction may affect each other, which is ideal. There was a question of whether an image could be obtained.

  As a solution to this, first, an optimal exposure time is set based on the luminance histogram of the imaging frame F (i), and then, based on the luminance histogram of the imaging frame (i + 1) photographed at that exposure time. It is possible to set an optimal gamma correction for the image and reflect the gamma correction value in the next imaging frame (i + 2). However, a delay of 2 frames occurs until an ideal image is obtained. . When using at a reduced frame rate, a delay of several tens of frames occurs.

  The present invention includes an image processing apparatus and an image processing method capable of obtaining a wide dynamic range image with reduced frame delay and more reliably approaching ideal imaging, and the image processing apparatus. An object is to provide an image pickup apparatus.

  An image processing apparatus according to a first aspect of the present invention includes a gamma correction removing unit that removes the effect of gamma correction from the image data of the image frame F (i) output from the image sensor, and an image frame from which the effect of the gamma correction is removed. A brightness histogram creating means for creating a brightness histogram from the image data of F (i), an optimum exposure time calculating means for calculating an optimum exposure time from the characteristics of the brightness histogram, and when the image sensor has taken an image at the optimum exposure time Brightness histogram estimation means for estimating the brightness (hereinafter referred to as virtual brightness) of image data of the image frame F (i + 1) of the image and creating a virtual brightness histogram, and optimal gamma for calculating an optimal gamma correction value from the characteristics of the virtual brightness histogram Correction value calculation means, and the following is obtained by setting the optimum exposure time and the optimum gamma correction value Characterized by sending the image data of the image frame F (i + 1).

  According to a second aspect of the present invention, in the image processing apparatus according to the first aspect of the present invention, the image processing apparatus according to the first aspect includes lightness classification means for classifying the lightness of the image data of the image frame F (i) output from the image sensor, When the brightness of the image data i) satisfies a predetermined condition, the image data of the image frame F (i) is sent as it is.

  According to a third aspect of the present invention, in the image processing apparatus according to the second aspect, the lightness classification means classifies the image data of the image frame F (i) into high lightness, intermediate lightness, and low lightness, and the intermediate lightness. When the number of pixels is larger than the predetermined number and the number of high and low lightness pixels is smaller than the predetermined number, or when the number of pixels of the high lightness, intermediate lightness, and low lightness is smaller than the predetermined number The image data of the image frame F (i) is transmitted as it is.

According to a fourth aspect of the present invention, in the image processing apparatus according to any one of the first to third aspects, the luminance histogram estimation means is an imaging frame F () in which the effect of gamma correction is removed by the gamma component removal means. From the brightness Y (x, y) gammaoff of the image data i), the optimum exposure time Tb calculated by the optimum exposure time calculation means, and the current exposure time Tn, a virtual brightness Y ′ (x, y) is obtained.
Y ′ (x, y) = Y (x, y) gammaoff * Tb / Tn
Is calculated and estimated.

  According to a fifth aspect of the present invention, in the image processing device according to any one of the first to fourth aspects, the luminance histogram estimation means determines whether or not the virtual luminance is appropriate. Feedback is made to the time calculation means to adjust the optimum exposure time, the virtual brightness is estimated again based on the adjusted optimum exposure time, and the virtual brightness histogram is recreated.

  According to a sixth aspect of the present invention, in the image processing apparatus according to the fifth aspect, the luminance histogram estimation means is configured to determine whether the number of pixels of the virtual luminance near the maximum brightness is larger than a predetermined number, or the virtual luminance near the minimum brightness. When the number of pixels is larger than a predetermined number, it is determined that the virtual luminance is not appropriate.

  According to a seventh aspect of the present invention, in the image processing apparatus according to the fifth or sixth aspect, the feedback to the optimum exposure time calculation means is as long as the optimum exposure time and optimum gamma correction value are set within one frame delay. It is characterized by being performed even once.

  An image processing method according to an eighth aspect of the present invention is a gamma correction removing step for removing gamma correction effect from image data of an image frame F (i) output from an image sensor, and an image frame from which the gamma correction effect is removed. A brightness histogram creating step of creating a brightness histogram from the image data of F (i), an optimal exposure time calculating step of calculating an optimal exposure time from the characteristics of the brightness histogram, and when the image sensor has taken an image at the optimal exposure time A luminance histogram estimation step for estimating the luminance of the image data of the image frame F (i + 1) (hereinafter referred to as virtual luminance) and creating a virtual luminance histogram, and an optimal gamma for calculating an optimal gamma correction value from the characteristics of the virtual luminance histogram A correction value calculating step, and calculating the optimum exposure time and the optimum gamma correction value. Characterized by sending the image data of the next image frame F obtained by the constant (i + 1).

  A ninth aspect of the present invention is the image processing method according to the eighth aspect, further comprising a lightness classification step of classifying the lightness of the image data of the image frame F (i) output from the image sensor, wherein the image frame F ( When the brightness of the image data i) satisfies a predetermined condition, the image data of the image frame F (i) is sent as it is.

  According to a tenth aspect of the present invention, in the image processing method according to the eighth or ninth aspect, the luminance histogram estimating step determines whether or not the virtual luminance is appropriate, and if not, feeds back to the optimum exposure time calculating means. To adjust the optimum exposure time, estimate the virtual brightness again based on the adjusted optimum exposure time, and recreate the virtual brightness histogram.

  According to an eleventh aspect of the present invention, in the image processing method according to the tenth aspect, the feedback to the optimum exposure time calculation step can be performed any number of times as long as the optimum exposure time and the optimum gamma correction value are set within one frame delay. It is performed.

  An image pickup apparatus according to a twelfth aspect of the invention includes an optical system that collects an optical image of a subject, an image pickup device that converts the optical image collected by the optical system into an electrical signal, and outputs image data; The image processing apparatus according to claim 1, and a display device that displays image data sent from the image processing apparatus.

  According to the image processing apparatus and the image processing method of the present invention, the optimum exposure time is calculated from the luminance histogram of the imaging frame F (i) from which the effect of the gamma correction is removed, and the imaging when the exposure time is set is performed. The brightness of the frame F (i + 1) is estimated, an optimal gamma correction value is calculated based on the brightness histogram (virtual brightness histogram), and the optimized exposure time and gamma correction value are calculated for the next actual imaging frame F ( By reflecting in i + 1), an image with a wide dynamic range can be obtained with a delay of only one frame and approaching ideal imaging. Further, if the brightness of the imaging frame F (i) is classified and a predetermined condition is satisfied, unnecessary imaging delay can be suppressed by outputting the imaging frame F (i) as it is. . Therefore, it is possible to provide an image capturing apparatus capable of adjusting a wide dynamic range that can more reliably approach ideal imaging with a small frame delay.

1 is an overall configuration diagram of an image pickup apparatus including an image processing apparatus of the present invention. It is a detailed functional block diagram of one Embodiment of the image processing apparatus in FIG. 3 is an overall process flowchart of the image processing apparatus of FIG. 2. It is a figure explaining the type of the brightness | luminance histogram of an image frame. FIG. 3 is a detailed functional block diagram of a luminance histogram estimation unit in FIG. 2. It is a figure which shows an example of a gamma correction curve. It is a detailed functional block diagram of another embodiment of the image processing apparatus in FIG. FIG. 8 is an overall process flowchart of the image processing apparatus of FIG. 7. FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is an overall configuration diagram of an image pickup apparatus including an image processing apparatus according to the present invention. This image capturing apparatus is used as, for example, an in-vehicle camera apparatus, but the application is not limited thereto.

  In FIG. 1, the optical system 100 efficiently collects subject light on the image sensor 200. The image sensor 200 has an optical sensor such as a CCD or CMOS, and converts it into a strong or weak electric signal (analog signal) corresponding to the light intensity. The image sensor 200 further includes an electronic shutter that adjusts the exposure time of the optical sensor, an AGC circuit that adjusts an analog image signal output from the optical sensor to a predetermined level, and converts the image signal into digital image data. An ADC circuit for performing gamma correction on the image data, and the like.

  The image processing apparatus 300 includes a frame memory 310 and an image processing unit 320. Image data output from the image sensor 200 is stored in the frame memory 310 and input to the image processing unit 320. In the embodiment, the image processing unit 320 classifies the brightness of the image data output from the image sensor 200, and when the image processing unit 320 satisfies a predetermined condition, the current imaging frame F (i stored in the frame memory 310 is stored. ) Image data is output as it is. On the other hand, when the predetermined condition is not satisfied or in another embodiment, after removing the effect of gamma correction from the image data of the current imaging frame F (i) output from the imaging device 200, the luminance histogram of the image data is used. The optimum exposure time is calculated, and the brightness (virtual brightness) of the image data obtained by shooting with the exposure time is estimated (predicted), and the optimum gamma correction value is calculated from the virtual brightness histogram. The time and the optimum gamma correction value are set in the image sensor 200. Then, the image processing unit 320 transmits the image data of the next imaging frame F (i + 1) that is output from the imaging device 200 and newly stored in the frame memory 310. Details of the image processing unit 320 will be described later.

  A DAC (digital / analog converter) 400 receives image data output from the frame memory 310 and converts the image data into, for example, an NTSC video signal. The display device 500 displays the video signal.

  FIG. 2 is a detailed functional block diagram of an embodiment of the image processing unit 320 which is the main configuration of the image processing apparatus 300. In the present embodiment, the image processing unit 320 includes a lightness classification unit 321, a gamma component removal unit 322, a luminance histogram creation unit 323, an optimal exposure time calculation unit 324, a luminance histogram estimation unit 325, an optimal gamma correction value calculation unit 326, and The control unit 317 controls the operation of each unit, the read / write operation of the frame memory 310, the operation of the image sensor 200, and the like.

  Note that the image processing unit 320 includes, for example, a CPU, a ROM, a RAM, and the like, and the functions of each unit are realized by the CPU executing a program stored in the ROM. The RAM is used to store image data being processed, necessary threshold values, parameters, and the like. Of course, each unit of the image processing unit 320 may be configured by hard logic.

  FIG. 3 shows an overall process flowchart of the present embodiment. Hereinafter, the operation of the image processing apparatus 300 (particularly the image processing unit 320) of this embodiment will be described in detail with reference to FIG.

  First, the control unit 327 initializes the frame number i (step 1001). Then, the image data of the imaging frame F (i) output from the imaging device 200 is stored in the frame memory 310 and input to the lightness classification unit 321 (step 1002).

  The brightness classification unit 321 classifies the brightness of the input image data of the imaging frame F (i) (step 1003). Specifically, a luminance histogram of input image data is created, divided into high brightness, intermediate brightness, and low brightness ranges, the number of pixels is counted, and the count value is set as a predetermined threshold value. Compared and classified into one of the following types. Note that a threshold for dividing the luminance histogram into each range of high brightness, intermediate brightness, and low brightness, and a threshold for comparing the count values of each range are determined in advance.

Type a: The number of pixels of low brightness is large, and the number of pixels of intermediate brightness and high brightness is small.
Type b: The number of pixels of high brightness is large, and the number of pixels of low brightness and intermediate brightness is small.
Type c: The number of pixels of intermediate brightness is large, and the number of pixels of low brightness and high brightness is small.
Type d: The number of pixels of low brightness and high brightness is large, and the number of pixels of intermediate brightness is small.
Type e: The number of pixels of low brightness, intermediate brightness, and high brightness are all small.
FIG. 4 is a diagram showing the types a to e as luminance histograms.

  The control unit 327 receives the classification result from the lightness classification unit 321, and determines which type the imaging frame F (i) belongs to (step 1004). In the case of types c and e, the control unit 327 controls the frame memory 310 so as to output the image data of the imaging frame F (i) stored in the frame memory 310 to the DAC 400 as it is (step S31). 1005), i ← i + 1 (step 1006), and the process proceeds to step 1018. That is, in the case of types c and e, the captured image is not too bright and not too dark, and is in a good state, so that neither the exposure time nor the gamma correction needs to be changed. In this embodiment, in such a case, unnecessary frame delay can be suppressed by outputting the image data of the current imaging frame F (i) as it is.

  On the other hand, in the case of types a, b, and d, the control unit 327 sends the image data of the current imaging frame F (i) stored in the frame memory 310 to the gamma component removal unit 322 with respect to the freem memory 310. Control to input. Hereinafter, each process from the gamma component removal unit 322 to the optimum gamma correction value calculation unit 326 is performed during the blanking period of the image sensor 200.

  The gamma component removal unit 322 removes the effect of gamma correction from the input image data of the current imaging frame F (i), and outputs image data in a state where the optical sensor of the imaging device 200 is photoelectrically converted (step 1007). Specifically, the luminance information Y (x, y) of each pixel of the current imaging frame F (i) is multiplied by an inverse gamma correction function, thereby obtaining the raw luminance information Y (output from the optical sensor of the imaging device 200). x, y) _gammaoff is obtained.

Now, it is assumed that the luminance Y (x, y) obtained by performing gamma correction on the luminance Y (x, y) _gammaoff output from the optical sensor of the image sensor 200 is expressed by the following equation.
Y (x, y) = a * ln (Y (x, y) _gammaoff) + b (a and b are coefficients)
In this case, in order to exclude the effect of gamma correction, the following calculation may be performed.
Y (x, y) _gammaoff = e ^ ((Y (x, y) -b) / a) (^ is a power)

  The luminance histogram creation unit 823 receives the image data of the current imaging frame F (i) from which the gamma correction effect has been removed by the gamma component removal unit 322, and creates the luminance histogram (step 1008).

  The optimum exposure time calculation unit 324 calculates the optimum exposure time from the characteristics of the brightness histogram created by the brightness histogram creation unit 323 (step 1009). Specifically, the optimum exposure time is determined as follows according to the luminance histogram types a to e (see FIG. 4).

Type a: The exposure time is set longer than the current set value.
Type b: The exposure time is set shorter than the current set value.
Type c: The exposure time is kept at the current set value.
Type d: If the number of high lightness pixels> the number of low lightness pixels, the exposure time is shorter than the current set value.
If the number of high lightness pixels <the number of low lightness pixels, the exposure time is set longer than the current set value.
Type e: Leave the exposure time at the current set value.
In types a, b, and d, how long or short the exposure time is with respect to the current set value may be determined according to the number of low-lightness pixels or the number of high-lightness pixels. In the case of type d, when the number of high lightness pixels is equal to the number of low lightness pixels, whether the exposure time should be shorter or longer than the current set value may be determined in advance by user setting or the like.

  The optimum exposure time calculation unit 324 sets the calculated optimum exposure time in the image sensor 200 (step 1010). As will be described later, the optimum exposure time calculation unit 324 can adjust the calculated exposure time by a feedback instruction from the luminance histogram estimation unit 325. In this case, the adjusted exposure time is set in the image sensor 200 as the optimum exposure time.

  The luminance histogram estimation unit 325 receives the image data of the current imaging frame F (i) from which the gamma correction effect has been removed from the gamma component removal unit 322, and further receives the optimum exposure time from the optimum exposure time calculation unit 324. And predicting and calculating the luminance (referred to as virtual luminance) of each pixel of the next imaging frame F (i + 1) output from the optical sensor of the imaging device 200 when the optimum exposure time is set, and creating the virtual luminance histogram (Step 1011). In addition, the luminance histogram estimation unit 325 checks whether or not the virtual luminance is appropriate. If the virtual luminance is not appropriate, the optimal exposure time is adjusted by applying feedback to the optimal exposure time calculation unit 324. Then, the virtual brightness is predicted and calculated again using the adjusted optimum exposure time, and the virtual brightness histogram is recreated.

FIG. 5 shows a detailed functional block diagram of the luminance histogram estimation unit 325. The virtual luminance calculation unit 3251 uses the luminance Y (x, y) _gammaoff of the image data of the current imaging frame F (i) from which the gamma correction effect from the gamma component removal unit 322 is removed, and the optimum exposure time calculation unit 324. From the calculated optimum exposure time Tb and the exposure time Tn of the current imaging frame F (i), the virtual luminance Y ′ (x, y) is calculated as follows.
Y ′ (x, y) = Y (x, y) _gammaoff * Tb / Tn
This is repeated for all pixels of the current imaging frame F (i). Here, the exposure time Tn is the optimal exposure time calculated last time, and the virtual brightness calculation unit 3251 holds the one sent from the optimal exposure time calculation unit 324.

  The virtual luminance histogram creation unit 3253 creates a virtual luminance histogram (virtual luminance histogram) calculated by the virtual luminance calculation unit 3251.

  The appropriate luminance determination unit 3252 checks whether or not the virtual luminance calculated by the virtual luminance calculation unit 3251 is appropriate. If the virtual luminance is not appropriate, the optimal luminance time calculation unit 324 is fed back to adjust the optimal exposure time. For example, when the image data is 8 bits, the number of pixels whose virtual luminance Y ′ is around 255 (near maximum brightness) is counted, and the count value is larger than a preset threshold value (too bright), the optimum The optimum exposure time calculation unit 324 is fed back so as to shorten the exposure time Tb. Further, when the number of pixels where the virtual luminance Y ′ is near 0 (near the minimum brightness) is counted and the count value is larger than a preset threshold value (too dark), the optimum exposure time Tb is optimally lengthened. This is fed back to the exposure time calculation unit 324.

  The optimum exposure time calculation unit 324 adjusts the optimum exposure time Tb in response to feedback from the brightness histogram estimation unit 325 (appropriate brightness determination unit 3252), and sends the optimum exposure time Tb to the brightness histogram estimation unit 325 again.

  The virtual luminance calculation unit 3251 again calculates the virtual luminance Y ′ using the adjusted optimal exposure time Tb from the optimal exposure time calculation unit 324, and the virtual luminance histogram creation unit 3253 calculates based on the virtual luminance. Recreate the virtual brightness histogram. In addition, the appropriate luminance determination unit 3252 further checks whether the virtual luminance is appropriate. If the virtual luminance is not appropriate, the appropriate luminance determination unit 3252 feeds back the optimum exposure time calculation unit 324 again. This feedback may be performed any number of times as long as the optimum exposure time and the optimum gamma correction value described later fall within one frame delay.

2 and 3, the optimum gamma correction value calculation unit 326 calculates the optimum gamma correction value from the characteristics of the virtual luminance histogram created by the luminance histogram estimation unit 325 (step 1012). Again, the optimum gamma correction value is determined as follows according to the types a to e (see FIG. 4) of the luminance histogram.
Type a: Ensures low brightness contrast.
Type b: A high brightness contrast is ensured.
Type c: Ensures contrast of intermediate brightness.
Type d: If the number of high lightness pixels> the number of low lightness pixels, a high lightness contrast is secured.
If the number of high lightness pixels <the number of low lightness pixels, a low lightness contrast is secured.
Type e: A general gamma curve is used. For example, it is the same as the case of type c.

  In the types a, b, c, and d, how much contrast is to be secured (how much gamma correction is to be performed) can be determined according to the number of low lightness pixels, high lightness pixels, or intermediate lightness pixels. Good. In type d, when the number of high lightness pixels≈the number of low lightness pixels, it may be determined in advance by user setting or the like whether to secure high lightness or low lightness contrast.

  The optimal gamma correction value calculation unit 326 sets the calculated optimal gamma correction value in the image sensor 200 (step 1013). FIG. 6 shows an example of a gamma correction curve corresponding to the types a to e.

  The control unit 327 sets i ← i + 1 (step 1014). Then, the image data of the imaging frame F (i + 1) output next from the imaging device 200 is stored in the frame memory 310 and is output to the DAC 400 side as it is (steps 1015 and 1016). The image data of the imaging frame F (F + 1) has been subjected to optimal exposure time and optimal gamma correction, and is close to ideal imaging.

  Thereafter, the control unit 327 sets i ← i + 2 (step 1017), proceeds to step 1018, and determines whether or not the final frame has been reached (step 1018). If the final frame has not been reached, the process returns to step 1002, and if the final frame has been reached, the process ends.

  According to the present embodiment, the optimal exposure time is calculated and set based on the luminance histogram of the imaging frame F (i) from which the effect of gamma correction has been removed, and when this optimal exposure time is set, imaging is performed. Estimate (predict) the virtual brightness of the imaging frame F (i + 1) output by the element, calculate and set the optimal gamma correction value based on the virtual brightness histogram, and then perform the exposure time and gamma correction optimized in this way. By reflecting this in the actual imaging frame F (i + 1), it is possible to obtain an image with a wide dynamic range by approaching ideal imaging with a delay of only one frame. Also, if it is not necessary to classify the brightness of the imaging frame F (i) and change the exposure time and gamma correction, the imaging frame F (i) is output immediately, thereby reducing unnecessary frame delay. Can be deterred.

  FIG. 7 is a detailed functional block diagram of another embodiment of the image processing unit 310 that is the main configuration of the image processing apparatus 300. The difference in configuration from FIG. 2 is that there is no lightness classification unit 321, and the rest is the same as FIG. 2.

  FIG. 8 shows an overall processing flowchart of the image processing apparatus 300 (particularly, the image processing unit 320) of the present embodiment. First, the control unit 327 initially sets the frame number i (step 2001). Then, the image data of the imaging frame F (i) output from the imaging element 200 is input to the gamma component removal unit 322 via the frame memory 310 (step 2002). Subsequent steps 2003 to 2014 are the same as steps 1007 to 1018 in FIG. In step 2002, the image data of the imaging frame F (i) output from the imaging device 200 may be directly input to the gamma component removal unit 322 without passing through the frame memory 310.

  Also in this embodiment, an image having a wide dynamic range can be obtained by approaching an ideal image with a delay of only one frame, as in the previous embodiment. In addition, the configuration can be simplified and the processing can be simplified. In this embodiment, even if it is not necessary to change the exposure time and the gamma correction, a delay of one frame is required, but the appearance on the display surface is not much different from the previous embodiment.

  As mentioned above, although two embodiment was shown, this invention is not limited to this. For example, the image processing unit 320 may include a gamma correction unit. In this case, a gamma correction step is added after step 1002 or step 2002 in FIGS. In steps 1013 and 2009, the optimum gamma correction value is set to the gamma correction unit in the image processing unit 320.

  Further, in the description of the embodiment, the optimum exposure time calculation unit and the optimum gamma correction value calculation unit individually set the calculated optimum exposure time or optimum gamma correction value, respectively. The optimum gamma correction value may be received and set collectively.

  For example, in-vehicle cameras are often used in environments where the illuminance tends to change, such as entering and exiting a tunnel. To cope with this, first, the screen brightness is roughly adjusted by exposure adjustment, and then fine adjustment is performed by gamma correction. In this case, if gamma correction is performed in a state where the exposure adjustment is not correct, bold gamma correction may be performed, causing image deterioration. However, the image pickup apparatus including the image processing apparatus of the present invention According to this, such a situation can be avoided.

DESCRIPTION OF SYMBOLS 100 Optical system 200 Image pick-up element 300 Image processing apparatus 310 Frame memory 320 Image processing part 321 Brightness classification part 322 Gamma component removal part 323 Luminance histogram creation part 324 Optimal exposure time calculation part 325 Luminance histogram estimation part 326 Optimal gamma correction value calculation part 327 Control unit 400 DAC
500 display device 3251 virtual luminance calculation unit 3252 appropriate luminance determination unit 3253 virtual luminance histogram creation unit

JP 2008-005365 A

Claims (10)

Gamma correction removing means for removing the effect of gamma correction from the image data of the image frame F (i) output from the image sensor;
A luminance histogram creating means for creating a luminance histogram from the image data of the image frame F (i) from which the effect of the gamma correction is removed;
Optimum exposure time calculating means for calculating an optimum exposure time from the characteristics of the luminance histogram;
Luminance histogram estimation means for estimating the luminance (hereinafter referred to as virtual luminance) of the image data of the image frame F (i + 1) when the image pickup device is photographed with the optimum exposure time, and creating a virtual luminance histogram;
Optimal gamma correction value calculating means for calculating an optimal gamma correction value from the characteristics of the virtual luminance histogram ;
Brightness classification means for classifying the brightness of the image data of the image frame F (i) output from the image sensor into high brightness, intermediate brightness, and low brightness ;
Sending out image data of the next image frame F (i + 1) obtained by setting the optimum exposure time and the optimum gamma correction value, the number of pixels of intermediate brightness is larger than a predetermined number, and high brightness and low brightness When the number of pixels is smaller than a predetermined number , the image processing apparatus transmits the image data of the image frame F (i) as it is. The luminance histogram estimation means is calculated by the luminance Y (x, y) gammaoff of the image data of the imaging frame F (i) from which the effect of gamma correction has been removed by the gamma correction removal means and the optimum exposure time calculation means. From the optimum exposure time Tb and the current exposure time Tn, the virtual luminance Y ′ (x, y) is
Y ′ (x, y) = Y (x, y) gammaoff * Tb / Tn
The image processing apparatus according to claim 1 , wherein the image processing apparatus calculates and estimates as follows. The brightness histogram estimation means determines whether or not the virtual brightness is appropriate. If the virtual brightness is not appropriate, the optimum exposure time calculation means is fed back to adjust the optimum exposure time, and based on the adjusted optimum exposure time. The image processing apparatus according to claim 1 , wherein the virtual luminance is estimated again and a virtual luminance histogram is recreated. The luminance histogram estimation means is not suitable for virtual luminance when the number of pixels of virtual luminance near the maximum brightness is larger than a predetermined number, or when the number of pixels of virtual luminance near the minimum luminance is larger than a predetermined number. The image processing apparatus according to claim 3 , wherein: 5. The image processing apparatus according to claim 3, wherein the feedback to the optimum exposure time calculation means is performed any number of times as long as the optimum exposure time and the optimum gamma correction value are set within one frame delay. . A gamma correction removal step for removing the effect of gamma correction from the image data of the image frame F (i) output from the image sensor;
A luminance histogram creating step of creating a luminance histogram from the image data of the image frame F (i) from which the effect of the gamma correction is removed;
An optimum exposure time calculating step for calculating an optimum exposure time from the characteristics of the luminance histogram;
A luminance histogram estimating step of estimating a luminance (hereinafter referred to as virtual luminance) of image data of an image frame F (i + 1) when the image pickup device is photographed with the optimum exposure time, and creating a virtual luminance histogram;
An optimal gamma correction value calculating step for calculating an optimal gamma correction value from the characteristics of the virtual luminance histogram ;
A lightness classification step of classifying the lightness of the image data of the image frame F (i) output from the image sensor into high lightness, intermediate lightness, and low lightness ;
Sending out image data of the next image frame F (i + 1) obtained by setting the optimum exposure time and the optimum gamma correction value, the number of pixels of intermediate brightness is larger than a predetermined number, and high brightness and low brightness When the number of pixels is smaller than a predetermined number , the image processing method is characterized in that the image data of the image frame F (i) is transmitted as it is. A lightness classification step of classifying the lightness of the image data of the image frame F (i) output from the image sensor;
7. The image processing method according to claim 6 , wherein when the brightness of the image data of the image frame F (i) satisfies a predetermined condition, the image data of the image frame F (i) is sent as it is. The luminance histogram estimation step determines whether or not the virtual luminance is appropriate. If the virtual luminance is not appropriate, feedback is made to the optimum exposure time calculation step to adjust the optimum exposure time, and based on the adjusted optimum exposure time. 8. The image processing method according to claim 6 , wherein the virtual luminance is estimated again and a virtual luminance histogram is recreated. 9. The image processing method according to claim 8, wherein the feedback to the optimum exposure time calculating step is performed any number of times as long as the optimum exposure time and the optimum gamma correction value are set within one frame delay. An optical system for condensing the optical image of the subject;
An image sensor that converts an optical image collected by the optical system into an electrical signal and outputs image data;
The image processing apparatus according to any one of claims 1 to 5 ,
A display device for displaying image data sent from the image processing device;
An image pickup apparatus comprising:

Images