JP2014143457A - HDR switching control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a switching control method to HDR photographing in which photographing and combining are performed in a plurality of exposures to expand a dynamic range.SOLUTION: An imaging device includes: imaging means for converting an optical image into an electric signal; exposure control means for controlling exposure of the imaging means; photographing and combining means for photographing a plurality of images in different exposures and combining the images; histogram acquisition means for acquiring a histogram of the image output from the imaging means; and combining determination means for determining switching between a normal photographing and photographing for combining from the histogram. The combining determination means has a threshold of a proper luminance range, and provides hysteresis to a threshold of a switching range.

Description

  The present invention relates to a switching control method to HDR imaging in which shooting / combination is performed with a plurality of exposures to expand a dynamic range.

  In recent years, imaging devices such as home video cameras and digital still cameras have become popular. Regarding the image pickup device, in addition to the CCD image pickup device, a CMOS image pickup device is being widely used. These image pickup devices are increasing in number of pixels in response to market demand, but the dynamic range of the image pickup device itself tends to be narrowed accordingly.

  As one of the solutions to this problem, dynamic range extended photography (HDR) has been proposed. Patent Document 1 proposes a method of extending the dynamic range by combining an under image and an over image. As a result, it is possible to take an image that exceeds the dynamic range of the image sensor, which is a convenient function for the user. However, when switching from normal shooting to HDR shooting, if switching frequently occurs, the image looks flickering, and the user may feel unsightly.

JP-A-6-273354

  However, in the method disclosed in Patent Document 1 and the like, dynamic switching between normal shooting and HDR shooting is not taken into consideration, and flickering as described above occurs if the method is used as it is.

In the imaging apparatus according to the present invention, first,
Imaging means for converting an optical image into an electrical signal;
A composite photographing means for photographing and combining a plurality of images with different exposures from the exposure control means for performing exposure control of the imaging means;
A histogram acquisition means for acquiring a histogram of an image output from the imaging means;
Having a composite determination means for determining switching between normal shooting and composite shooting from the histogram;
The combination determination means has a threshold value of an appropriate luminance range,
The switching range threshold value is provided with hysteresis.

Second,
In the imaging device described in the first aspect,
Immediately after switching from normal shooting to composite shooting, the threshold value of the appropriate luminance range is narrowed.

Third,
In the imaging device described in the first aspect,
Immediately after switching from composite photography to normal photography, the threshold value of the appropriate luminance range is widened.

Fourth,
In the imaging device described in the second and third,
After changing the threshold value of the appropriate luminance range, it returns to the original threshold value with a time constant.

Fifth,
Imaging means for converting an optical image into an electrical signal;
A composite photographing means for photographing and synthesizing a plurality of images with different exposures from the exposure control means for performing exposure control of the imaging means;
A histogram acquisition means for acquiring a histogram of an image output from the imaging means;
Having a composite determination means for determining switching between normal shooting and composite shooting from the histogram;
The composition determination means has a threshold value of the appropriate luminance ratio,
The switching ratio threshold value is provided with hysteresis.

Sixth,
In the imaging device described in the fifth aspect,
Immediately after switching from normal shooting to composite shooting, the threshold value of the appropriate luminance ratio is lowered.

Seventh,
In the imaging device described in the fifth aspect,
Immediately after switching from composite photography to normal photography, the threshold value of the appropriate luminance ratio is raised.

Eighth,
In the imaging device described in the sixth and seventh,
After changing the threshold value of the appropriate luminance ratio, it returns to the original threshold value with a time constant.

Ninth,
In the imaging device described in the first and second,
Having scene discrimination means,
The threshold value is changed according to the scene determined by the scene determination means.

With the configuration described in the means for solving the problem,
In switching between normal video and HDR video, images of normal shooting and composite shooting are not frequently switched, and high-quality images and moving images without flickering can be shot.

2 is a block diagram of the first and second embodiments. FIG. It is the figure which showed the moving image shooting flow in Example 1 * 2 of this invention. It is the figure which showed the synthetic | combination ratio of HDR in Example 1 * 2 of this invention. It is the figure which showed the appropriate luminance threshold value of HDR synthetic | combination determination in Example 1 of this invention. It is the figure which showed the brightness | luminance range hysteresis of the HDR synthetic | combination determination in Example 1 of this invention. It is the figure which showed the hysteresis time constant at the time of switching from a normal moving image in Example 1 of this invention to a HDR moving image. It is the figure which showed the hysteresis time constant when switching from the HDR moving image in Example 1 of this invention to a normal moving image. It is the figure which showed the luminance ratio of the HDR synthetic | combination determination in Example 2 of this invention. It is the figure which showed the brightness | luminance ratio hysteresis of the HDR synthetic | combination determination in Example 2 of this invention.

  The present invention will be described using a digital camera as an example.

<First Embodiment>
Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram showing the configuration of the first and second embodiments of the present invention.

  In FIG. 1, reference numeral 100 denotes an image processing apparatus.

  10 is a photographing lens, 12 is a mechanical shutter having an aperture function, 14 is an image sensor that converts an optical image into an electrical signal, and 16 is an A / D converter that converts an analog signal output of the image sensor 14 into a digital signal. .

  A timing generation circuit 18 supplies a clock signal and a control signal to the image sensor 14 and the A / D converter 16, and is controlled by the memory control circuit 22 and the system control circuit 50. In addition to the twelve mechanical shutters, the accumulation time can be controlled as an electronic shutter by controlling the reset timing of the eighteen image sensors, and can be used for moving image shooting and the like.

  An image processing circuit 20 performs predetermined pixel interpolation processing and color conversion processing on the data from the A / D converter 16 or the data from the memory control circuit 22. Further, an electronic zoom function is realized by performing image cutting and scaling processing by 20 image processing circuits.

  Further, the image processing circuit 20 performs predetermined calculation processing using the captured image data, and the system control circuit 50 controls the exposure control means 40 and the distance measurement control means 42 based on the obtained calculation result. TTL AF processing, AE processing, and EF processing are performed.

  Further, the image processing circuit 20 performs predetermined arithmetic processing using captured image data, and also performs TTL AWB (auto white balance) processing based on the obtained arithmetic result.

  A memory control circuit 22 controls the A / D converter 16, the timing generation circuit 18, the image processing circuit 20, the memory 30, and the compression / decompression circuit 32.

  The data of the A / D converter 16 is written into the memory 30 via the image processing circuit 20 and the memory control circuit 22, or the data of the A / D converter 16 is directly written via the memory control circuit 22.

  Reference numeral 28 denotes an image display unit composed of a TFT LCD or the like, and display image data written in the memory 20 is displayed by the image display unit 28 via the memory control circuit 22.

  If the image data captured using the image display unit 28 is sequentially displayed, the electronic viewfinder function can be realized.

  The image display unit 28 can arbitrarily turn on / off the display according to an instruction from the system control circuit 50. When the display is turned off, the power consumption of the image processing apparatus 100 can be greatly reduced. I can do it.

  Reference numeral 30 denotes a memory for storing captured still images and moving images, and has a sufficient storage capacity to store a predetermined number of still images and a predetermined time of moving images.

  This makes it possible to write a large amount of images to the memory 30 at high speed even in continuous shooting or panoramic shooting in which a plurality of still images are continuously shot.

  The memory 30 can also be used as a work area for the system control circuit 50.

  Reference numeral 31 denotes a non-volatile memory composed of a flash ROM or the like. The program code executed by the system control circuit 50 is written in the nonvolatile memory 31, and the program code is executed while being read sequentially. In addition, an area for storing system information and an area for storing user setting information are provided in the nonvolatile memory, and various information and settings are read and restored at the next startup.

  Reference numeral 32 denotes a compression / decompression circuit that compresses and decompresses image data by adaptive discrete cosine transform (ADCT) or the like, reads an image stored in the memory 30, performs compression processing or decompression processing, and stores the processed data in the memory 30. Write to.

  Reference numeral 40 denotes an exposure control means for controlling the shutter 12 having a diaphragm function, and has a flash light control function in conjunction with the flash 48.

  Reference numeral 42 denotes distance measurement control means for controlling the focusing of the photographing lens 10, and reference numeral 44 denotes zoom control means for controlling the zooming of the photographing lens 10.

  A flash 48 has an AF auxiliary light projecting function and a flash light control function.

  The exposure control means 40 and the distance measurement control means 42 are controlled using the TTL method. Based on the calculation result obtained by calculating the captured image data by the image processing circuit 20, the system control circuit 50 performs the exposure control means 40 and the distance measurement. Control is performed on the control means 42.

  A system control circuit 50 controls the entire image processing apparatus 100.

  Reference numerals 60, 62, 64, 66, 70, and 72 denote operation means for inputting various operation instructions of the system control circuit 50. A single unit such as a switch, a dial, a touch panel, pointing by line-of-sight detection, a voice recognition device, or the like Consists of multiple combinations.

  Here, a specific description of these operating means will be given.

  Reference numeral 60 denotes a mode dial switch, which can switch and set various function modes such as power-off, automatic shooting mode, shooting mode, panoramic shooting mode, and moving image shooting mode.

  Reference numeral 62 denotes a shutter switch SW1, which is turned on during the operation of the shutter button, and instructs to start operations such as AF (autofocus) processing, AE (automatic exposure) processing, and AWB (auto white balance) processing.

  A shutter switch SW2 64 is turned on when the operation of the shutter button is completed. In the case of flash photography, after performing EF (flash pre-emission) processing, the image sensor 14 is exposed for the exposure time determined in the AE processing. In flash photography, light is emitted during this exposure period, and the exposure period. Simultaneously with the end of the exposure, the exposure control means 40 shields the light, thereby completing the exposure to the image sensor 14. A read process for writing the image data into the memory 30 via the A / D converter 16 and the memory control circuit 22 for the signal read from the image sensor 14; The image data is read from the memory 30 and compressed by the compression / decompression circuit 32. Next, an instruction to start a series of processes of recording processing for writing the image data to the recording medium 200 is given.

  Reference numeral 66 denotes a display changeover switch, which can change the display of the image display unit 28. With this function, when photographing is performed using the optical viewfinder 104, it is possible to save power by cutting off the current supply to the image display unit including a TFT LCD or the like.

  Reference numeral 70 denotes an operation unit including various buttons, a touch panel, a rotary dial, and the like, which include a menu button, a set button, a macro button, a multi-screen playback page break button, a flash setting button, a single shooting / continuous shooting / self-timer switching button, and the like. . Menu move + (plus) button, menu move-(minus) button, playback image move + (plus) button, playback image move-(minus) button, shooting image quality selection button, exposure compensation button, date / time setting button, etc. There is also.

  Reference numeral 72 denotes a zoom switch unit as zoom operation means for a user to give an instruction to change the magnification of a captured image. Hereinafter, the zoom switch 72 is also referred to. The zoom switch 72 includes a tele switch that changes the imaging field angle to the telephoto side and a wide switch that changes the imaging angle of view to the wide angle side. By using the zoom switch 72, the zoom control unit 44 is instructed to change the imaging field angle of the photographing lens 10 and becomes a trigger for performing an optical zoom operation. In addition, it also serves as a trigger for electronic zooming change of the imaging angle of view by image cropping by the image processing circuit 20 or pixel interpolation processing.

  Reference numeral 86 denotes a power supply means including a primary battery of an alkaline battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li ion battery, an AC adapter, or the like.

  Reference numeral 90 denotes an interface with a recording medium such as a memory card or hard disk, and reference numeral 92 denotes a connector for connecting to a recording medium such as a memory card or hard disk.

  Reference numeral 102 denotes protection means that is a barrier that prevents the imaging unit from being soiled or damaged by covering the imaging unit including the lens 10 of the image processing apparatus 100.

  Reference numeral 104 denotes an optical viewfinder, which can take an image using only the optical viewfinder without using the electronic viewfinder function of the image display unit 28.

  A communication unit 110 has various communication functions such as USB, IEEE1394, LAN, and wireless communication.

  Reference numeral 112 denotes a connector for connecting the image processing apparatus 100 to another device by the communication unit 110 or an antenna in the case of wireless communication.

  Reference numeral 200 denotes a recording medium such as a memory card or a hard disk.

  The recording medium 200 includes a recording unit 202 composed of a semiconductor memory, a magnetic disk, or the like, an interface 204 with the image processing apparatus 100, and a connector 206 for connecting to the image processing apparatus 100.

  Reference numeral 210 denotes a recording medium such as a memory card or a hard disk.

  FIG. 2 is a diagram showing a moving image shooting flow in the first and second embodiments of the present invention.

  When there is an instruction to start moving image recording, photometric processing is performed in S101. In the photometric process, an appropriate exposure is calculated, and an image is taken in S102. In the photometric process of S101, the luminance distribution information of the subject is calculated, and it is determined whether to perform HDR shooting of S103 using the luminance distribution information. Details of the HDR imaging determination will be described later with reference to FIG. If it is determined in S103 that HDR movie shooting is to be performed, an under image is shot in S104. The under-image shooting will be described later in detail with reference to FIG. For the under image to be captured here, an exposure difference is set according to the shutter speed or sensitivity with respect to the appropriate exposure. Which to select will be described later with reference to FIG.

  If two types of images with different exposures are acquired, a composition process is performed in S105. Details of the combining process will be described later with reference to FIG.

  Since the combined image remains in the RGB-Bayer data format output from the image sensor, the development processing is performed in the YUV format in S106, the encoding process is performed, and the moving image is recorded on the recording medium in S107. I do.

  If there is a recording end instruction in S108, the moving image recording routine is exited. Until there is an end instruction, the process returns to S101 to continue moving image recording.

  FIG. 3 is a diagram showing the HDR synthesis ratio in Examples 1 and 2 of the present invention.

  As shown in the composition ratio table of FIG. 3, the composition ratio is determined by the pixel level of the appropriate image. When the pixel level of the appropriate image is low, the pixel has a very low luminance level, so the composition ratio from the appropriate image is increased and the composition ratio from the under image is decreased. On the other hand, when the pixel level of the appropriate image is high, the pixel has a very high luminance level. Therefore, the composition ratio from the under image in which no whiteout occurs is increased and the composition ratio from the appropriate image is decreased. By executing this synthesis process, an image having a wider dynamic range than an image obtained by one exposure can be obtained.

  FIG. 4 is a diagram showing an appropriate luminance threshold value for HDR combination determination in the first and second embodiments of the present invention.

  In the photometric process of S101, the luminance distribution of the subject is acquired. In the luminance distribution of FIG. 4A, since most of the subject luminance is within the appropriate luminance range, it is determined that the recording is not the HDR moving image but the normal moving image recording. Since there are many pixels in the luminance distribution of FIG. 4-B that are not within the appropriate luminance range, it is determined that HDR imaging is necessary because the dynamic range is insufficient.

  Specifically, a histogram is generated, and it is determined that the pixels that fall within the luminance from the appropriate luminance range lower limit Lv_Lo to the upper limit Lv_Hi are within the appropriate luminance range. Whether or not HDR movie shooting is performed is determined based on whether or not a ratio TekiseRatio obtained by dividing the number of pixels PixNumTekeii within the appropriate luminance range by the total number of pixels PixNumTotal is equal to or greater than a predetermined value as shown in Expression (1).

If TekiseRatio is equal to or greater than a predetermined value, normal shooting is performed, and if it is equal to or less than a predetermined value, HDR shooting is performed. This determination needs to be performed constantly during recording of a moving image, and follows a scene change. TekiseRatio has hysteresis so that normal shooting and HDR video are not frequently switched. This will be described in detail later with reference to another drawing.

  FIG. 5 is a diagram showing luminance range hysteresis in HDR synthesis determination in Example 1 of the present invention.

  Switching between normal shooting and HDR shooting is determined by the ratio of pixels within the appropriate luminance range.

  FIG. 5A shows a threshold when switching from a normal moving image to an HDR moving image. A solid line indicates a threshold, and a dotted line indicates a threshold when returning from an HDR video to a normal video.

  FIG. 5B shows a threshold value when switching from the HDR moving image to the normal moving image. The solid line indicates the threshold value, and the dotted line indicates the threshold value when returning from the normal moving image to the HDR moving image.

  As described above, by changing the threshold value when entering the HDR video and when exiting the HDR video, there is no determination to return immediately after switching, so frequent switching does not occur.

  By changing this threshold value depending on the scene, it is possible to realize optimal switching determination. During backlighting, the threshold value is changed by landscape shooting or night scene shooting.

  FIG. 6 is a diagram showing a hysteresis time constant when switching from the normal moving image to the HDR moving image in Example 1 of the present invention.

  FIG. 6A shows threshold values when switching from a normal moving image to an HDR moving image. The solid lines indicate the switching threshold values Lv_Lo_1 and Lv_Hi_1, and the dotted lines indicate the threshold values Lv_Lo_2 and Lv_Hi_2 immediately after switching. FIG. 6B shows a change in threshold when switching from a normal moving image to an HDR moving image. The thresholds immediately after the switching are Lv_Lo_2 and Lv_Hi_2, but gradually return toward Lv_Lo_1 and Lv_Hi_1, and finally become equal to Lv_Lo_1 and Lv_Hi_1. Thus, immediately after switching from the normal moving image to the HDR moving image, the threshold value is narrowed and the threshold value is gradually returned to prevent flickering of switching.

  FIG. 7 is a diagram illustrating a hysteresis time constant when switching from the HDR moving image to the normal moving image according to the first embodiment of the present invention.

  FIG. 7A shows a threshold when switching from the HDR moving image to the normal moving image. The solid lines indicate the switching threshold values Lv_Lo_1 and Lv_Hi_1, and the dotted lines indicate the threshold values Lv_Lo_2 and Lv_Hi_2 immediately after switching. FIG. 7B shows a change in threshold when switching from an HDR video to a normal video. The threshold immediately after switching is Lv_Lo_3 and Lv_Hi_3, but gradually returns toward Lv_Lo_1 and Lv_Hi_1, and finally becomes equal to Lv_Lo_1 and Lv_Hi_1. In this way, immediately after switching from the HDR video to the normal video, the threshold value is expanded, and by gradually returning the threshold value, flickering of the switching can be prevented.

<Second Embodiment>
The second embodiment of the present invention will be described below with reference to the drawings. Also in the present embodiment, the same configuration and processing as in the first embodiment are performed from FIG. 1 to FIG.

  FIG. 8 is a diagram showing the luminance ratio of HDR combination determination in Example 2 of the present invention.

  FIG. 8 shows the luminance range for switching. When the PixelRatio is set to the ratio within the brightness, if the appropriate ratio TexiRatio falls below the appropriate brightness ratio threshold Ratio_1, the video is switched from the normal movie to the HDR movie. On the contrary, when the appropriate ratio TekiseRatio exceeds the appropriate luminance ratio threshold Ratio_2, the HDR moving image is switched to the normal moving image. At this time, in order to provide hysteresis, the relationship between Ratio_1 and Ratio_2 is as follows.

FIG. 9 is a diagram showing luminance ratio hysteresis in HDR synthesis determination in Example 2 of the present invention.

  FIG. 9A shows a change in the threshold value of the luminance ratio when switching from the normal moving image to the HDR moving image. When it falls below the appropriate luminance ratio threshold Ratio_1, it is determined that the normal moving image is switched to the HDR moving image, and at the same time, the appropriate luminance ratio threshold is changed to Ratio_2. The appropriate luminance ratio threshold gradually returns from Ratio_2 toward Ratio_1, and finally becomes equal to Ratio_1.

  FIG. 9B shows a change in the threshold value of the luminance ratio when switching from the HDR moving image to the normal moving image. If it exceeds the appropriate luminance ratio threshold Ratio_1, it is determined that the HDR moving image is switched to the normal moving image, and at the same time, the appropriate luminance ratio threshold is changed to Ratio_3. The appropriate luminance ratio threshold gradually returns from Ratio_3 toward Ratio_1, and finally becomes equal to Ratio_1.

  Ratio relationship between Ratio_1, Ratio_2, and Ratio_3 is expressed by the following expression (3).

By changing the appropriate luminance ratio threshold value depending on the scene, an optimal switching determination can be realized. During backlighting, the threshold value is changed by landscape shooting or night scene shooting.

  As described above, when switching from a normal video to an HDR video, by providing hysteresis to the luminance range threshold and the luminance range ratio threshold, frequent switching between the normal video and the HDR video can be prevented and recorded. You can get high-quality video without flickering.

  In the two embodiments, the threshold value is changed based on the scene discrimination result. However, the threshold value is changed according to the rotation direction of the camera, acceleration information, focal length information, subject distance information, and exposure correction amount. Also good. In addition, although it has been described that the time constant after changing the threshold is constant, the speed of the time constant may be changed according to the scene.

DESCRIPTION OF SYMBOLS 10: Shooting lens 12: Shutter 14: Image pick-up element 16: A / D converter 18: Timing generation circuit 20: Image processing circuit 22: Memory control circuit 28: Image display part 30: Memory 32: Image compression / expansion circuit 40: Exposure control means 42: Distance measurement control means 44: Zoom control means 48: Flash 50: System control circuit 60: Mode dial switch 62: Shutter switch SW1 64: Shutter switch SW2
66: Display changeover switch 70: Operation unit 72: Zoom switch 90: Interface 92: Connector 100: Image processing apparatus 102: Protection means 104: Optical viewfinder 110: Communication means 112: Connector (or antenna)
200: Recording medium 202: Recording unit 204: Interface 206: Connector

Claims (9)

Imaging means for converting an optical image into an electrical signal;
Exposure control means for performing exposure control of the imaging means;
A composite photographing means for photographing and combining a plurality of images with different exposures;
A histogram acquisition means for acquiring a histogram of an image output from the imaging means;
Having a composite determination means for determining switching between normal shooting and composite shooting from the histogram;
The combination determination means has a threshold value of an appropriate luminance range,
An imaging device characterized in that hysteresis is given to a switching range threshold value. The imaging apparatus according to claim 1, wherein the threshold value of the appropriate luminance range is narrowed immediately after switching from normal shooting to composite shooting. The imaging apparatus according to claim 1, wherein the threshold value of the appropriate luminance range is widened immediately after switching from composite photography to normal photography. The imaging apparatus according to claim 2 or 3, wherein after the threshold value of the appropriate luminance range is changed, the original threshold value is restored with a time constant. Imaging means for converting an optical image into an electrical signal;
Exposure control means for performing exposure control of the imaging means;
A composite photographing means for photographing and combining a plurality of images with different exposures;
A histogram acquisition means for acquiring a histogram of an image output from the imaging means;
Having a composite determination means for determining switching between normal shooting and composite shooting from the histogram;
The composition determination means has a threshold value of the appropriate luminance ratio,
An image pickup apparatus characterized in that hysteresis is provided to a switching ratio threshold value. The imaging apparatus according to claim 5, wherein the threshold value of the appropriate luminance ratio is lowered immediately after switching from normal shooting to composite shooting. The imaging apparatus according to claim 5, wherein the threshold value of the appropriate luminance ratio is increased immediately after switching from composite photography to normal photography. The imaging apparatus according to claim 6 or 7, wherein after the threshold value of the appropriate luminance ratio is changed, the original threshold value is restored with a time constant. The imaging apparatus according to claim 1, further comprising a scene determination unit, wherein the threshold value is changed according to the scene determined by the scene determination unit.