JP7081635B2 - Image processing equipment - Google Patents

Description

The present invention relates to an image processing apparatus that obtains one composite image by synthesizing a plurality of images acquired by a plurality of imaging.

Conventionally, in order to generate an image having a wide dynamic range (hereinafter referred to as "wide dynamic range image"), a plurality of images of the same subject taken by changing the exposure amount are combined, for example, by addition averaging to obtain one image. An image pickup device for obtaining a composite image is known (Patent Document 1). The image synthesizing means of this image pickup apparatus uses the image data obtained with an exposure amount higher than the standard as the data on the low brightness side, and the image data obtained with the standard exposure amount as the data on the high brightness side, and exposes less than the standard. The image data obtained by the amount is used as the data on the ultra-high brightness side, and the image data is synthesized by all or any combination.

Japanese Unexamined Patent Publication No. 7-135599

By the way, image sensors such as CCDs and CMOS image sensors tend to generate fixed pattern noise at high output (low ISO sensitivity), and random noise such as optical shot noise at low output (high ISO sensitivity). Cheap. Fixed pattern noise is noise whose value can be estimated if the conditions are the same to some extent, so it can be reduced by image processing, etc., whereas random noise is generated at unspecified positions, so it is used in image processing, etc. Difficult to mitigate.

An object of the present invention is to provide an image processing apparatus devised to reduce random noise generated in a wide dynamic range image in order to solve the above problems.

In one aspect of the present invention, the image processing apparatus has a first image data generated by the image pickup operation of the image pickup unit under the first exposure condition and an exposure time shorter than the exposure time under the first exposure condition. When the sensitivity of the input unit to which the second image data generated by the image pickup operation of the image pickup unit under the second exposure condition is input and the sensitivity of the image pickup unit is lower than a predetermined value, the first image data And a composite image data obtained by synthesizing the plurality of the second image data, and when the sensitivity of the imaging unit is higher than the predetermined value, the plurality of the first image data and the plurality of the second image data are generated. It is provided with a generation unit for generating a composite image data obtained by synthesizing the image data of the above.

The generation unit may increase the number of the second image data to be synthesized when the sensitivity of the image pickup unit is lower than a predetermined value, as the sensitivity of the image pickup unit is lower.

In the present invention, the image data captured under the exposure condition with the lowest exposure amount among the image data for the present processing is combined with the preprocessing image data obtained by imaging the image data a plurality of times, so that random noise is surely generated. Can be mitigated.

It is a block diagram which shows the electrical outline of the electronic camera which used this invention. It is a block diagram which shows the detailed structure of an image processing circuit. It is a block part which shows the structure of the alignment process. It is a flowchart which shows the operation procedure of the composite shooting mode. It is a flowchart which shows the operation procedure at the time of high sensitivity. It is a flowchart which shows the continuation of the operation procedure described with FIG. It is explanatory drawing which shows the example which changed the acquisition number of short-time exposure image data in a low-sensitivity range. It is explanatory drawing which shows the example which changed the acquisition number of each image data for short time exposure, standard exposure, and long time exposure in the range of high sensitivity.

An electronic camera will be described as an embodiment of the image processing apparatus of the present invention. The electronic camera has a composite shooting mode in which a single wide dynamic range image data is generated by synthesizing a plurality of image data obtained by imaging with different exposure amounts by an image sensor in response to one shutter release. Have.

As shown in FIG. 1, the electronic camera 10 includes a lens optical system 11, an aperture 12, a mechanical shutter 13, an image sensor 14, an AFETG (analog front-end timing generator) 15, a lens drive unit 16, an aperture drive unit 17, and a shutter drive unit. It is composed of 18, an AE / AF circuit 19, a control unit 20, an operation unit 21, an LCD 22, an image processing circuit 23, a RAM 24, a ROM 25, a VRAM 26, a non-volatile memory 27, a memory slot 28, an external memory 29, and the like.

The lens driving unit 16 performs scaling and focusing operations of the lens optical system 11. The aperture drive unit 17 drives the aperture 12. The shutter drive unit 18 drives the mechanical shutter 13. The image sensor 14 is an image sensor such as a CCD or CMOS that photoelectrically converts a subject image imaged via the lens optical system 11 and the diaphragm 12.

The AFETG15 is composed of a TG (timing generator) 30, a V-driver (vertical drive unit) 31, and an AFE (analog front end) 32.

The TG 30 gives the image pickup device 14 the read timing of the image signal via the V-drive 31. The AFE 32 amplifies the analog image signal read from the image sensor 14 by removing noise and increasing the analog gain, and then performs A / D conversion to convert it into digital image data, and the converted image data is converted into an AE / AF circuit. It is output to 19 and the image processing circuit 23.

The AE / AF circuit 19 extracts and controls AF (autofocus) information that controls focus based on contrast information of image data, and optimal AE (autoexposure) information and white balance information based on image data. Send to department 20. The control unit 20 controls the lens drive unit 16, the aperture drive unit 17, the image processing circuit 23, and the like based on the AF information, the AE information, and the white balance information.

The image processing circuit 23 includes white balance processing, compression coding, decoding processing, image composition processing, and the like. The VRAM 26 is an image memory for displaying on the LCD 22. The LCD 22 displays the captured image, menus for various settings of the camera, and a shooting mode.

An external memory 29 is detachably connected to the memory slot 28. The compression-encoded image data generated by the image processing circuit 23 is stored in the external memory 29.

The RAM 24 is used as a work area for the control unit 20 and the image processing circuit 23. The ROM 25 stores a program executed by the control unit 20. The non-volatile memory 27 stores various adjustment data of the electronic camera 10, a shooting mode set by the user, a sensitivity (ISO sensitivity), and the like.

The control unit 20 is configured around a CPU (central processing unit) that operates according to a program stored in the ROM 25, and controls each unit of the electronic camera 10. For example, the control unit 20 gives an instruction to read the image signal of the image sensor 14 to the TG 30 of the AFET G15, inputs the image signal read from the image sensor 14 to the image processing circuit 23 via the AFE 32, and causes the image processing circuit 23 to read the image signal. After executing the processing corresponding to the shooting mode at that time, the processed image data is controlled to be recorded in the external memory 29 via the bus 33.

The operation unit 21 includes a sensitivity setting unit, a mode setting unit, a shutter button, a power button, and the like, and these send an operation signal and a setting signal to the control unit 20. The sensitivity setting unit is an operation unit for changing the sensitivity (ISO sensitivity) of the image pickup device 14 in a plurality of stages. The sensitivity is predetermined to be arbitrary.

When the control unit 20 obtains the subject brightness information from the AE / AF circuit 19, the control unit 20 obtains the standard exposure according to the program diagram corresponding to the sensitivity set at that time so as to approach the appropriate exposure determined by the sensitivity and the subject brightness. Determine the amount (exposure amount obtained by the aperture value and shutter speed value).

The mode setting unit is an operation unit that selectively selects any one of the composite shooting mode, the normal shooting mode, and the reproduction mode. The composite shooting mode is a mode in which a plurality of images for this processing generated by imaging the same subject with different exposure amounts by the image sensor 14 are combined to obtain one wide dynamic range image data.

In the composite shooting mode, the exposure amount is controlled by the combination of the electronic shutter and the aperture 12 that change the accumulation time of the image sensor 14. The shutter speed may be determined by, for example, opening and closing the mechanical shutter 13 instead of the electronic shutter, or the first timing of exposure may be determined by the electronic shutter and the end of exposure may be determined by the mechanical shutter 13.

The normal shooting mode is a mode in which a single image data is generated by taking an image once with the image sensor 14 with a standard exposure amount. The reproduction mode is a mode in which the image data recorded in the external memory 29 is reproduced on the LCD 22.

As shown in FIG. 2, the control unit 20 includes a sensitivity determination unit 35 and a counter 36. The sensitivity determination unit 35 recognizes the set sensitivity set in the sensitivity setting unit at the time of shooting, and compares the recognized sensitivity with a predetermined threshold value to determine whether the sensitivity is high or low. The counter 36 counts the number of times the image data is captured. The control unit 20 changes the degree of amplification of the analog image data by increasing the analog gain of the AFE 32 according to the sensitivity determined by the sensitivity determination unit 35.

The image processing circuit 23 includes a distribution unit 37, a pre-processing synthesis unit 38, and a main processing composition unit 39. The distribution unit 37 distributes the image data to the pre-processing synthesis unit 38 and the main processing synthesis unit 39, and this distribution processing is the image data for pre-processing or the image for main processing at the time of high sensitivity or low sensitivity. It is controlled by the control unit 20 based on the data.

The processing synthesis unit 39 performs processing to generate one wide dynamic range image data by synthesizing a plurality of images for this processing captured at different exposure amounts in the composite shooting mode. The image data for this processing is composed of, for example, short-time exposure image data, standard exposure image data, and long-time exposure image data. The short-time exposure image data is image data generated by taking an image with an image sensor 14 with an exposure amount smaller than the standard. The standard exposure image data is image data generated by taking an image with a standard exposure amount by the image sensor 14. The long-time exposure image data is image data generated by taking an image with an image pickup element 14 with an exposure amount larger than the standard.

When the pre-processing compositing unit 38 determines that the sensitivity is low in the compositing shooting mode, a plurality of pre-processing images generated by taking an image with an image pickup element 14 with an exposure amount smaller than the standard in order to reduce random noise. The data is combined to generate one noise reduction image data, and the noise reduction image data is output to the processing synthesis unit 39 as short-time exposure image data among the processing image data.

Further, when the preprocessing and synthesizing unit 38 determines that the sensitivity is high in the composite shooting mode, the preprocessing and synthesizing unit 38 combines a plurality of image data generated by taking an image with an exposure amount smaller than the standard by the image pickup element 14 into one image data. The short-time exposure image data is used as the short-time exposure image data, and the standard exposure image data is obtained by synthesizing a plurality of image data generated by imaging with a standard exposure amount into one image data, which is more than the standard. A combination of a plurality of image data generated by imaging with an exposure amount into one image data is output to the processing and synthesizing unit 39 as long-time exposure image data.

The pre-processing and main processing synthesis units 38 and 39 include buffer memories 40 and 44, alignment processing 41 and 45, and image composition 43 and 47, respectively. The buffer memories 40 and 44 temporarily store a plurality of image data. The alignment processes 41 and 45 compare the image data and correct the coordinates so that the positions of the image data match. In the image composition 43 and 47, after adding each image data based on the corrected coordinate information, an averaging process of dividing by the number of image data is performed to generate one image data.

As shown in FIG. 3, the alignment processes 41 and 45 are composed of an evaluation unit 48, a determination unit 49, and an alignment unit 50. The evaluation unit 48 evaluates each image data by obtaining at least one of predetermined conditions, for example, the level (brightness) of the image data, the contrast, and the high frequency component of the spatial frequency. In this case, the brightest image data among the image data whose black-and-white level of the image data has not reached the saturation level, the image data having the highest contrast value, or the image data having the highest high-frequency component has a higher evaluation value. ..

Instead, the main subject is detected from each image data, and the standard image data is used for evaluation such as the condition that the evaluation value is higher as the amount of movement of the detected main subject is smaller, the focus state, and the degree of deformation of the main subject. You may decide.

The determination unit 49 determines the image data most suitable for composition as the reference image data based on the evaluation value of each image data obtained by the evaluation unit 48. The alignment unit 50 performs alignment processing so that the coordinate positions match the remaining image data with reference to the reference image data. As a result, the image quality of the image data synthesized by the image synthesis 43 and 47 is improved. After evaluating each image data, the evaluation unit 48 may perform a process of selecting not to use the image data for compositing if the evaluation value is lower than a predetermined threshold value. ..

Next, the operation of the above configuration will be described with reference to FIG. The sensitivity of the image sensor 14 is preset to an initial value, for example, the ISO sensitivity "400". The photographer selects a composite shooting mode when shooting a wide dynamic range image.

The control unit 20 determines the standard exposure amount at that time in response to one shutter release (S-1), confirms the shooting mode at that time (S-2), and further performs composite shooting. In the case of the mode, it is determined whether or not the sensitivity set at that time is less than the threshold value (S-3). For example, when the set sensitivity is ISO sensitivity "50" and the threshold value is ISO sensitivity "600", it is determined that the set sensitivity is low.

When the control unit 20 determines that the sensitivity is low, the control unit 20 sets an exposure amount smaller than the standard, for example, an exposure amount corrected for exposure of "-1EV", and continuously shoots images with the image sensor 14, for example. The five pre-processing image data are acquired (S-4), and the distribution unit 37 is controlled to take the five pre-processing image data into the buffer memory 40 of the pre-processing synthesis unit 38.

By receiving a signal in response to the count-up from the control unit 20, the preprocessing synthesis unit 38 reads out five preprocessing image data in order from the buffer memory 40, and the evaluation unit 48 preliminarily reads each preprocessing image data. Evaluation is performed based on the determined conditions, the reference image data is determined by the determination unit 49 based on the evaluation result (S-5), and the alignment unit 50 determines the remaining image data based on the reference image data. After the alignment process (S-6), the image data that has been aligned with the reference image data is combined in order by the addition averaging process in the image composition 43, and the short-time exposure for this process is performed. Image data is generated (S-7), and the short-time exposure image data is output to the buffer memory 44 of the processing synthesis unit 39.

When acquiring the five preprocessing image data, the control unit 20 initializes the counter 36 (S-8), and the counter 36 counts the number of times of imaging in a short exposure (S-9). It is determined whether or not the number of times of imaging has reached a predetermined number of times (S-10).

Subsequently, the control unit 20 changes the exposure amount to a standard exposure amount and captures one image to generate standard exposure image data for this processing (S-11), and controls the distribution unit 37 to synthesize the standard exposure image data in this processing. The data is taken into the buffer memory 44 of the unit 39, and subsequently, the exposure amount is set to a larger exposure amount than the standard, for example, the exposure amount is adjusted to "+1 EV", and one image is taken to generate long-exposure image data for this processing ( S-12), the long exposure image data is taken into the buffer memory 44.

The processing synthesis unit 39 responds to the three image data for the present processing, that is, the short-time exposure image data, the standard exposure image data, and the long-time exposure image data, being taken into the buffer memory 44, and the alignment processing 45 The same process as the alignment process 41 described above is performed to determine the reference image data (S-13), and the remaining image data is aligned with respect to the reference image data (S-14). , Image data that has been aligned with respect to the reference image data is combined in order by addition averaging processing to generate wide dynamic range image data (S-15), and the wide dynamic range image data is stored in the external memory 29. Record. The wide dynamic range image obtained in the composite shooting mode in the low-sensitivity determination is an image in which the random noise that tends to occur in the low-sensitivity shooting is reduced in addition to suppressing the occurrence of overexposure and underexposure.

Next, in the determination of FIG. 4 (S-3), the sensitivity determination unit 35 determines that the set sensitivity is high because the set sensitivity exceeds the threshold value, for example, when the set sensitivity is ISO [800]. In this case, as shown in FIG. 5, a plurality of control units 20 are set to an exposure amount smaller than the standard, for example, an exposure amount corrected for exposure of "-1EV", and continuously imaged by the image sensor 14. The image data for preprocessing, for example, 6 images is acquired (S-16), and the distribution unit 37 is controlled to take in the image data for preprocessing of 6 images into the buffer memory 40 of the preprocessing synthesis unit 38.

The pre-processing synthesis unit 38 reads out six pre-processing image data in order from the buffer memory 40 in response to the count-up of the number of shots taken in the short-time exposure from the control unit 20, and positions the same processing described above. Performed by the matching process 41 and the image composition 43 (S-17), the short-time exposure image data for the main process is generated (S-18), and the short-time exposure image data is stored in the buffer memory 44 of the main process synthesis unit 39. Output to.

Subsequently, the control unit 20 sets the standard exposure amount and continuously shoots images with the image sensor 14, so that a plurality of images, for example, four images for preprocessing, which are smaller than the number of images captured by the short exposure, are captured. Acquired (S-19), the distribution unit 37 is controlled, and the four pre-processing image data are taken into the buffer memory 40 of the pre-processing synthesis unit 38.

The preprocessing synthesis unit 38 reads out four preprocessing image data in order from the buffer memory 40 in response to the count-up of the number of shots taken in the standard exposure from the control unit 20, and aligns the same processing described above. Performed in the process 41 and the image composition 43 (S-20), the standard exposure image data for this process is generated (S-21), and the standard exposure image data is output to the buffer memory 44 of the process synthesis unit 39. ..

Subsequently, as shown in FIG. 6, the control unit 20 sets an exposure amount higher than the standard, for example, an exposure amount with an exposure compensation of "+1 EV", and continuously takes an image with the image sensor 14, thereby performing standard exposure. Acquires a plurality of images for preprocessing, for example, two images for preprocessing (S-22), which is smaller than the number of images captured by It is taken into the buffer memory 40.

The preprocessing synthesis unit 38 reads out two preprocessing image data in order from the buffer memory 40 in response to the count-up of the number of shots taken in the standard exposure from the control unit 20, and aligns the same processing described above. The process 41 and the image composition 43 are performed (S-23) to generate the long-exposure image data for the main process (S-24), and the long-exposure image data is transferred to the buffer memory 44 of the process synthesis unit 39. Output.

Then, as described with reference to FIGS. 4 (S-13 to S-15), the processing synthesis unit 39 responds to the three image data for the processing being taken into the buffer memory 44, and the alignment processing 45 Determines the reference image data in (S-25), aligns the remaining image data with respect to the reference image data (S-26), and aligns the reference image data. The image data of the above is combined in order by the addition averaging process to generate a wide dynamic range image data (S-27), and the wide dynamic range image data is recorded in the external memory 29. The wide dynamic range image generated by the high-sensitivity determination is an image in which the random noise that tends to occur during high-sensitivity shooting is reduced in addition to suppressing the occurrence of overexposure and underexposure.

It is desirable that the control unit 20 controls to clear the buffer memories 40 and 44 after recording the wide dynamic range image data.

Further, in the above embodiment, it goes without saying that the number of shots and the exposure compensation value determined according to the sensitivity are not limited to the above-mentioned numerical values. For example, as shown in FIG. 7, in the range where the set sensitivity is less than the threshold value, it is preferable to increase the number of acquired short-time exposure image data as the set sensitivity is lower. Further, for example, as shown in FIG. 8, it is preferable to increase the number of acquired image data for each of the short exposure, the standard exposure, and the long exposure as the set sensitivity becomes higher in the range where the set sensitivity exceeds the threshold value. Is. Assuming that the number of acquired image data for short-time exposure, standard exposure, and long-time exposure is A, B, and C in order, the number of acquired image data is set so as to maintain the relationship of A> B> C. It is preferable to change it.

When setting manually, a change operation unit such as a cursor button or an OK button is provided in the operation unit, and using these operation units, for example, the number of shots taken in a short exposure when it is determined to be low sensitivity, and high sensitivity. The number of shots taken with a short exposure, the number of shots with a standard exposure, and the number of shots with a long exposure may be set respectively.

In the above embodiment, three image data of short-time exposure, standard exposure, and long-time exposure are combined to generate a wide dynamic range image, but the present invention is not limited to this, and two different exposures are generated. A wide dynamic range image may be generated by synthesizing image data or synthesizing four or more image data to which image data of ultra-low time exposure or ultra-long exposure is added.

In the above embodiment, when it is determined that the sensitivity is low, the preprocessing image is first imaged a plurality of times with a short exposure, then with a standard exposure, and finally with a long exposure. In the present invention, the order is not limited to this, and for example, the image may be taken in the reverse order, first with standard exposure, then multiple times with short exposure, and finally with long exposure.

Similarly, when it is determined that the sensitivity is high, the imaging period for acquiring the preprocessing image data for the short-time exposure image data and the imaging for acquiring the preprocessing image data for the standard exposure image data are taken. The order of the period and the imaging period for acquiring the preprocessing image data for the long exposure data may be any order.

In the above embodiment, the image data is aligned and synthesized every time the image data is acquired. However, all the images are first captured and the image data is recorded in the RAM, and the image data is to be synthesized from the RAM after the shooting is completed. The image data may be read out and combined. In addition, all images are first captured and image data is recorded in a flash memory or the like, image processing software is installed in an electronic camera or the like in advance, and the software is executed after shooting to capture a plurality of images. The data may be combined.

Further, although the present invention is described using the electronic camera 10 in the above embodiment, the present invention is not limited to this and can be used for a video camera, a television camera, or the like.

14 Image sensor 12 Aperture 13 Mechanical shutter 15 AFETG
23 Image processing circuit 38 Pre-processing synthesis unit 39 Main processing composition unit

Claims (2)

The first image data generated by the imaging operation of the imaging unit under the first exposure condition and the imaging operation of the imaging unit under the second exposure condition, which is shorter than the exposure time under the first exposure condition. The second image data generated by, the input unit to which is input, and
When the sensitivity of the image pickup unit is lower than the predetermined value, a composite image data obtained by combining the first image data and a plurality of the second image data is generated, and the sensitivity of the image pickup unit is higher than the predetermined value. If it is also high, a generator that generates a composite image data obtained by synthesizing the plurality of the first image data and the plurality of the second image data, and a generation unit.
An image processing device comprising. The image processing apparatus according to claim 1, wherein the generation unit increases the number of the second image data to be synthesized when the sensitivity of the image pickup unit is lower than a predetermined value, as the sensitivity of the image pickup unit becomes lower.

Images