JP7442989B2 - Imaging device, control method for the imaging device, and program - Google Patents

Description

The present invention relates to a technique for generating an image corresponding to a scene with a wide dynamic range in an imaging device.

2. Description of the Related Art Conventionally, a method disclosed in Patent Document 1 has been proposed as one method for photographing a moving subject using an imaging device such as a video camera. In this method, the moving speed of a moving subject is estimated from multiple frames of video footage, and shooting conditions such as shutter speed and aperture are adjusted according to the estimated moving speed. By using this method, images can be photographed under optimal photographing conditions adjusted for the moving subject.

Furthermore, a method has been proposed in which the exposure amount is controlled for each pixel to perform imaging in a dynamic range that corresponds to the brightness range of the scene (see Patent Document 2).

Japanese Patent Application Publication No. 2004-120298 Japanese Patent Application Publication No. 2010-136205

Exposure is determined by shutter speed, aperture, and sensitivity, but when shooting video, the aperture is often fixed, so the shutter speed is adjusted to adjust exposure. However, if the shutter speed is determined according to the moving speed of a moving subject according to the method described in Patent Document 1, it becomes difficult to handle scenes with a wide dynamic range. In other words, in the method described in Patent Document 1, when the shutter speed is slow, bright areas around the moving subject tend to be blown out, and when the shutter speed is fast, the bright areas around the moving subject tend to be blown out. Dark areas are more likely to be crushed.

On the other hand, when a scene including a moving subject is photographed using the method described in Patent Document 2, if there are areas with different brightness within one moving subject, a different shutter speed is set for each pixel in that area. may be done. In this case, even for the same moving subject, images with different amounts of blur due to movement may be generated depending on the pixel position. A moving subject portion of such an image will give a sense of discomfort.

The present invention has been made in view of these problems, and provides an imaging device and a control for the imaging device that can generate an image corresponding to a scene with a wide dynamic range without creating a sense of discomfort in a moving subject portion. The purpose is to provide methods and programs.

In one embodiment of the present invention, an imaging device capable of capturing an image by controlling imaging conditions for each pixel or region is configured to capture the image based on motion information of a subject. The same shutter speed is applied to the identifying means for identifying the moving object area, the acquiring means for acquiring the brightness information of the object, and the pixels corresponding to the moving object area , and the same shutter speed is applied according to the brightness information. The shooting conditions are set for each pixel or region so that the ISO sensitivity is applied , and the same ISO sensitivity is applied to pixels corresponding to non-moving object areas that are not the moving object areas, and the brightness information is The present invention is characterized by comprising a setting means for setting the photographing conditions for each pixel or each region so that a corresponding shutter speed is applied.

According to the present invention, it is possible to generate an image corresponding to a scene with a wide dynamic range without causing a sense of discomfort in a moving subject portion.

FIG. 1 is a configuration diagram of an imaging device in an embodiment. FIG. 3 is a schematic diagram showing a brightness map and a moving object map in one embodiment. 3 is a flowchart showing the overall flow of imaging processing in one embodiment. 3 is a flowchart showing the flow of imaging condition setting processing in one embodiment. 5 is a flowchart showing the flow of ISO sensitivity setting change processing in one embodiment. 3 is a flowchart showing the flow of shutter speed setting change processing in one embodiment. FIG. 3 is a diagram illustrating an example of an imaging condition change table in one embodiment. 3 is a flowchart showing the overall flow of imaging processing in one embodiment. It is a figure showing an example of a moving object map value update table in one embodiment. FIG. 3 is a diagram illustrating an example of an imaging condition change table in one embodiment.

[Example 1]

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not necessarily limited to the illustrated configurations.

(First embodiment)
[overview]
An overview of the embodiment will be explained. In this embodiment, the exposure conditions for each pixel on the sensor are determined based on the moving object map provided by the moving object map acquisition section and the brightness map provided by the brightness map acquisition section. As the sensor, for example, a sensor as shown in Patent Document 2, which can set photographing conditions for each pixel, is used. Therefore, the photographing conditions include shutter speed and sensitivity setting values for each pixel.

Further, in this embodiment, the moving object map and the brightness map are configured with a smaller number of pixels than the number of sensor pixels. With reference to the values of the moving object map, for pixels that are likely to be moving objects, the shutter speed setting value is set to the same value for the subject. Then, the exposure settings for each pixel are changed based on the brightness map. The details of these processes will be explained below.

[Device configuration]
FIG. 1 is a configuration diagram of an imaging device in one embodiment. The imaging device in this embodiment can set photographing conditions such as shutter speed and ISO sensitivity for each pixel, and can photograph an image by controlling the exposure amount for each pixel.

The imaging unit 101 is a device for detecting light from a subject. The imaging unit 101 includes, for example, a zoom lens, a focus lens, a blur correction lens, an aperture, a shutter, an optical low-pass filter, an iR cut filter, a color filter, and a sensor such as CMOS or CCD.

The A/D converter 102 is a device for converting the detected amount of light from a subject into a digital value.

The signal processing unit 103 is a device that performs signal processing on the digital values to generate a digital image, and performs, for example, demosaicing processing, white balance processing, gamma processing, and the like.

The D/A converter 104 performs processing to convert the digital image obtained by the signal processor 103 into analog for display.

The encoder unit 105 is a device that performs data compression processing on the digital image, and performs processing such as compression using the JPEG method, for example.

The media interface (media I/F) unit 106 is an interface for connecting the imaging device to a PC or other media (eg, hard disk, memory card, CF card, SD card, USB memory, etc.).

The CPU 107 is involved in all of the processing of each component mentioned above. The ROM 108 and the RAM 109 provide the CPU 107 with programs, data, work areas, etc. necessary for processing. Further, if a control program necessary for processing to be described later is stored in the ROM 108, it is once read into the RAM 109 and then executed by the CPU 107.

The operation unit 111 is a device for inputting instructions from the user, and includes, for example, buttons, a mode dial, a touch panel attached to the display unit 113, and the like. In the case of this embodiment, input for switching between HDR (high dynamic range) output and standard output is also performed using the operation unit 111.

The character generation unit 112 is a device for generating characters and graphics.

The display unit 113 is a device that displays images such as photographed images and GUI, and generally uses a CRT, a liquid crystal display, or the like. Alternatively, a known touch screen may be used. In that case, the input via the touch screen can also be handled as the input from the operation unit 111.

The moving object map acquisition unit 114 is configured to detect whether or not there is movement in the subject acquired by the imaging unit 101. The moving object map acquisition unit 114, for example, acquires imaging data in time series from the imaging unit 101, and generates object movement information for each pixel by averaging luminance changes between the imaging data over a plurality of pixels. If the average value of brightness changes is large, motion information is generated as a moving pixel, and if the average value of brightness changes is small, motion information is generated as a pixel without movement. The moving object map acquisition unit 114 generates the motion information generated in this way for each pixel in the image, and holds it as a moving object map. Details of the moving object map will be described later. Note that the moving object map may be acquired using not only the above method but also other methods. For example, between two pieces of imaging data in time series, the difference in pixel values at the same pixel position is compared with a predetermined threshold, and pixels with a large difference are treated as moving object areas, and pixels with small differences are treated as non-moving object areas. You can also generate a moving object map.

The brightness map acquisition unit 115 acquires a brightness map indicating the brightness of the subject for each pixel from the imaging data acquired by the imaging unit 101. For example, the brightness map acquisition unit 115 averages brightness information for each pixel obtained from the sensor for a plurality of pixels, and holds the averaged information as a brightness map of the image. Details of the brightness map will be described later.

The imaging system control unit 110 is a device for controlling the imaging system as instructed by the CPU 107. In this embodiment, shooting conditions such as shutter speed and ISO sensitivity set by the imaging system control unit 110 are set for each pixel in the imaging unit 101. The imaging system control unit 110 also performs controls such as focusing, opening the shutter, and adjusting the aperture.

Note that the imaging device includes various components other than those described above, but since they are not the main focus of the present invention, a description thereof will be omitted.

The photographing condition setting process described in this embodiment acquires a moving object map from the moving object map acquisition section 114, acquires a brightness map from the brightness map acquisition section 115, and controls the imaging system based on the moving object map and the brightness map. Photographing condition data for controlling section 110 is generated. The photographing condition setting process is executed by the CPU 107 loading a control program stored in the ROM 108 into the RAM 109 and executing it.

[About brightness map and moving object map]
The brightness map and moving object map used in this embodiment will be described below with reference to the schematic diagram of FIG. 2.

FIG. 2(a) shows an example of a photographed image, FIG. 2(b) shows an example of a brightness map, and FIG. 2(c) shows an example of a moving object map. Both the brightness map and the moving object map are information composed of fewer pixels than the number of pixels of the captured image. Furthermore, brightness maps and moving object maps are expressed using fewer bits than captured images; for example, while captured images are expressed using 12 bits, brightness maps and moving object maps are expressed using 8 bits. be done. In the brightness map, the higher the value, the higher the illuminance. The higher the value of the moving object map, the higher the probability that a moving object is included in the subject area.

[Overall flow of imaging processing]
The overall flow of imaging processing in the imaging device of this embodiment will be explained using the flowchart of FIG. 3. Here, when an instruction to start imaging with HDR output specified is input, for example, through the operation unit 111, the imaging process described below is started.

Note that the series of processes shown in the flowchart is performed by the CPU 107 loading a control program stored in the ROM 108 into the RAM 109 and executing it. Alternatively, some or all of the functions of the steps in the flowchart may be realized by hardware such as an ASIC or an electronic circuit. The symbol "S" in the description of each process means a step in the flowchart. The same applies to other flowcharts.

First, in S201, the CPU 107 uses the brightness map acquisition unit 115 to acquire a brightness map of the subject.

In S202, the CPU 107 uses the moving object map acquisition unit 114 to acquire a moving object map of the subject.

In S203, the CPU 107 sets shooting conditions when the imaging unit 101 performs shooting based on the brightness map obtained in S201 and the moving object map obtained in S202. That is, the CPU 107 functions as a photographing condition setting means. Details will be described later.

In S204, the CPU 107 sends the imaging condition data set in S203 to the imaging system control unit 110, drives the imaging unit 101 to perform imaging, and acquires imaging data.

In S205, the CPU 107 applies digital gain to the image data obtained in S204 in accordance with the exposure amount difference to match the brightness of the entire image. The imaging data obtained in S204 has different imaging conditions applied to each pixel, so in order to pseudo-unify these imaging conditions, a digital gain corresponding to the exposure amount difference is applied to each pixel. .

As explained above, the imaging process in the imaging device of this embodiment is performed.

[Shooting condition setting process]
Next, the photographing condition setting process shown in S203 of FIG. 3 will be explained using the flowchart of FIG. 4.

In S301, the CPU 107 inputs the moving object map acquired in S201 to the RAM 109.

In S302, the CPU 107 identifies an area corresponding to the moving object as a moving object area based on the moving object map input to the RAM 109. Specifically, the CPU 107 determines that a portion of the moving object map whose value is equal to or greater than a predetermined threshold is a moving object portion, and updates the value of the moving object map to 255, which is the maximum value. On the other hand, the CPU 107 determines that a portion of the moving object map whose value is smaller than a predetermined threshold is a non-moving object region, and updates the value of the moving object map to 0, which is the minimum value. In this way, the CPU 107 functions as a moving object area specifying means.

In S303, the CPU 107 inputs the brightness map acquired in S202 to the RAM 109.

In S304, the CPU 107 determines the basic exposure amount at the time of shooting. The basic exposure amount is determined based on, for example, the shutter speed, ISO sensitivity, and aperture value specified by the user. Note that the basic exposure amount may be determined by converting the average brightness into an exposure amount with reference to the brightness map obtained from the brightness map acquisition unit 115, or may be determined by other methods. .

In S305, the CPU 107 initializes the position of the imaging condition setting pixel. For example, the CPU 107 sets the upper left coordinates as the initial value. Thereafter, the photographing condition setting process for each pixel is performed sequentially toward the lower right.

In S306, the CPU 107 determines whether the value of the moving object map corresponding to the position of the target pixel on which the imaging condition setting process is performed (i.e., the pixel position to be processed) is a value indicating a moving object portion (i.e., 255). . If it is determined that it is a moving body part, the process advances to S307; otherwise, the process advances to S308.

In S307, the CPU 107 refers to the value of the illuminance distribution map corresponding to the pixel position to be processed, and sets photographing conditions with the ISO sensitivity changed. That is, for a moving object part, the ISO sensitivity is changed without changing the shutter speed. Therefore, even if there are areas with different brightness within a single moving object, the same shutter speed will be applied, so even if the moving object is blurred, the amount of blur will be the same, and this will prevent any discomfort from occurring. be able to. Details will be described later.

In S308, the CPU 107 refers to the value of the illuminance distribution map corresponding to the pixel position to be processed, and sets photographing conditions in which the shutter speed is changed. That is, for non-moving object parts, the shutter speed is changed without changing the ISO sensitivity. Details will be described later.

In S309, the CPU 107 determines whether the imaging condition setting process has been completed for all pixels. If the process has ended, the process ends; otherwise, the process advances to S310.

In S310, the CPU 107 moves the pixel position to be processed to the next pixel position. When processing for one pixel is completed, the process moves to the next pixel to the right. If the pixel is at the right end, it moves to the left end pixel in the row below.

[ISO sensitivity setting change process]
Next, the ISO sensitivity setting change process shown in S307 of FIG. 4 will be explained using the flowchart of FIG.

First, in S401, the CPU 107 refers to the brightness map stored in the RAM 109.

In S402, the CPU 107 refers to the value of the brightness map corresponding to the pixel position to be processed, and calculates the gain change amount from the difference in brightness from the basic exposure determined in S304 of FIG. Specifically, the CPU 107 can calculate the gain change amount using the following equation (1).
G=round(L(X,Y)/N(map)) (1)

Here, G is the amount of gain change, L (X, Y) is the value of the brightness map belonging to the same area as the target pixel, and N (map) is the amount of change in ISO sensitivity setting shown in Fig. 7(a). Indicates the number of elements in the compiled photographing condition change table. The photographing condition change table will be described later.

In S403, the CPU 107 refers to the imaging condition change table shown in FIG. 7A and determines imaging conditions based on the gain change amount determined in S402. Specifically, from the value of the gain change amount G calculated using equation (1), the range of brightness map values in the shooting condition change table is referred to, and the corresponding ISO sensitivity setting change amount is determined in S304. Add to the ISO sensitivity. For example, if the gain change amount G determined by equation (1) is 75, the setting is changed to increase the ISO sensitivity determined in S304 by one step with reference to the photographing condition change table.

[Shutter speed setting change process]
Next, the shutter speed setting change process shown in S308 of FIG. 4 will be explained using the flowchart of FIG. 6.

First, in S501, the CPU 107 refers to the brightness map stored in the RAM 109.

In S502, the CPU 107 refers to the value of the brightness map corresponding to the pixel position to be processed, and calculates the gain change amount from the difference in brightness from the basic exposure determined in S304 of FIG. Specifically, the CPU 107 can calculate the gain change amount using the following equation (2).
G'=round(L(X,Y)/N'(map)) (2)

Here, G' is the amount of gain change, L (X, Y) is the value of the brightness map belonging to the same area as the target pixel, and N' (map) is the value of the photographing condition change table shown in FIG. 7(b). Indicates the number of elements. The photographing condition change table will be described later.

In S503 , the CPU 107 refers to the photographing condition change table shown in FIG. 7(b) and determines photographing conditions based on the gain change amount determined in S502. Specifically, from the value of the gain change amount G' calculated using equation (2), the range of brightness map values in the photographing condition change table is referred to, and the corresponding shutter speed change amount is determined in S304. Add to the shutter speed. For example, if the gain change amount G' determined by equation (2) is 190, the setting is changed to shorten the shutter speed determined in S304 by one step with reference to the photographing condition change table.

[Photography condition change table]
FIG. 7 shows an example of the imaging condition change table used in S402 of FIG. 5 and S502 of FIG. 6. FIG. 7A shows a photographing condition change table for changing the ISO sensitivity, and shows the relationship between the brightness map value and the ISO value. FIG. 7B shows a photographing condition change table when changing the shutter speed, and shows the relationship between the shutter speed (Tv) value and the brightness map value.

As described above, according to the present embodiment, it is possible to generate an image corresponding to a scene with a wide dynamic range without causing a sense of discomfort in a moving subject portion.

(Second embodiment)
Next, a second embodiment of the present invention will be described. In this embodiment, exposure amount correction for a moving object is performed using both shutter speed and gain. When changing the exposure compensation according to the speed of the moving object that is the subject, when the moving object is very fast, or when the moving object is very bright or dark, the ISO sensitivity correction is used. This is because there are cases where the exposure amount of the moving object part cannot be adjusted sufficiently by only using the above-mentioned method.

First, the overall flow of the photographing process in this embodiment will be explained using the flowchart of FIG. 8. Further, FIG. 9 shows an example of a moving object map value update table in this embodiment, and FIG. 10 shows an example of a photographing condition change table in this embodiment. Note that in the description of this embodiment, the points that are different from the first embodiment will be mainly explained.

This embodiment differs from the first embodiment in the method of grouping moving object maps, the method of setting gain swing conditions, and the method of setting shutter speed.

In S602, the CPU 107 identifies an area corresponding to the moving object as a moving object area based on the moving object map input to the RAM 109. Specifically, the CPU 107 determines that a portion of the moving object map where the value of the moving object map is equal to or greater than a predetermined value, as shown in the table shown in FIG. Update based on. For example, if the value of the moving object map is 150, the value of the moving object map is updated to 128. Further, the CPU 107 determines that a portion of the moving object map whose value is smaller than a predetermined value is a non-moving object region, and updates the value of the moving object map to 0, which is the minimum value. Note that this value classification is not limited to the values shown in FIG. 9, but may be performed using other values.

In S607, the CPU 107 refers to the value of the illuminance distribution map corresponding to the pixel position to be processed, and calculates the average illuminance of pixels having the same moving object map value. Then, according to the table in FIG. 10(a), the shutter speeds of pixels having the same moving object map value are changed. At the same time, the illuminance map values of pixels having the same moving object map value are corrected. For example, if the illuminance of a certain moving object pixel is 100 and the average illuminance of the moving object part is 50, the shutter speed of the moving object part is increased by one step, and 50 is subtracted from the value of the illuminance map to change it to 50. do.

In S611, the CPU 107 refers to the table in FIG. 10(b) and sets imaging conditions with changed gains for the moving object portion. As a result of these operations, for moving object parts determined to move at the same speed, the shutter speed is set to the same value, and the gain is also changed according to the illuminance of the subject. Therefore, even if there are areas with different brightness within the moving object, the same shutter speed is applied to the moving object. Therefore, even if the moving object is shaken, the amount of shake remains the same, and it is possible to prevent the user from feeling uncomfortable. Furthermore, when trying to deal with a moving object that is entirely dark or a moving object that is entirely bright by changing only the gain as in the first embodiment, it is necessary to apply a large gain to the moving object. come out. In this embodiment, even in such a case, by changing the shutter speed for the entire moving object, the amount of blur of the moving object can be made the same. In addition, even if there are regions where the illuminance of a moving object differs, appropriate exposure conditions can be set by changing the gain.

As described above, according to the present embodiment, even when ISO sensitivity alone is not sufficient for adjusting the exposure amount of a moving object region, an image in which the exposure amount has been adjusted can be obtained.

Note that the present invention can also be applied to images in which a plurality of images are combined. In this case, when photographing a moving object, multiple images have the same shutter speed. Since these images are combined, differences in blur between moving object parts are eliminated, and an image with less discomfort can be obtained.

Furthermore, in the above-described embodiments, a case where a sensor that can set photographing conditions for each pixel is used is used as an example, but the present invention can also be applied to a case where a sensor that can set photographing conditions for each region is used. In that case, the photographing conditions are set so that the same shutter speed is set for the area including the moving object.

(Other embodiments)
The present invention provides a system or device with a program that implements one or more of the functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 Imaging section 102 A/D conversion section 103 Signal processing section 104 D/A conversion section 105 Encoder section 106 Media I/F section 107 CPU
108 ROM
109 RAM
110 Imaging system control section 111 Operation section 112 Character generation section 113 Display section 114 Moving object map acquisition section 115 Brightness map acquisition section

Claims (18)

An imaging device capable of capturing images by controlling imaging conditions for each pixel or region,
identification means for identifying a moving object area in the image based on movement information of the object;
acquisition means for acquiring brightness information of the subject;
The photographing conditions are set for each pixel or region so that the same shutter speed is applied to the pixels corresponding to the moving object region, and the ISO sensitivity according to the brightness information is applied. For pixels corresponding to non-moving object areas that are not areas, the shooting conditions are set for each pixel or area so that the same ISO sensitivity is applied and a shutter speed according to the brightness information is applied. means and
An imaging device comprising: The setting means determines a basic exposure amount when shooting the image, sets the shooting conditions according to the basic exposure amount, and then changes the shutter speed for each pixel or area corresponding to the moving object area. 2. The imaging apparatus according to claim 1, wherein the ISO sensitivity is determined according to the brightness information without determining the brightness information . The setting means determines an amount of change in ISO sensitivity according to the brightness information for each pixel or area corresponding to the moving object area , based on a difference between the brightness information and the basic exposure amount. The imaging device according to claim 2 . The setting means determines a basic exposure amount when shooting the image, sets the shooting conditions according to the basic exposure amount, and then adjusts the brightness for each pixel or area corresponding to the non- moving object area. The imaging device according to claim 1, wherein the shutter speed is determined according to the brightness information without changing the ISO sensitivity according to the information . The setting means may determine, for each pixel or region corresponding to the non-moving object region, an amount of change in shutter speed according to the brightness information , based on a difference between the brightness information and the basic exposure amount. The imaging device according to claim 4 , characterized in that: The imaging device according to any one of claims 1 to 5 , wherein the motion information is a moving object map generated by averaging luminance changes between time-series imaging data over a plurality of pixels. . 7. The imaging device according to claim 1 , wherein the brightness information is a brightness map generated by averaging the brightness of each pixel over a plurality of pixels. further comprising calculation means for calculating an average illuminance of the same subject based on the brightness information corresponding to the pixel position of the moving object area,
The imaging apparatus according to claim 1, wherein the setting means determines a shutter speed to be applied to each pixel in each moving object area based on the average illuminance. The imaging device according to claim 8 , wherein the setting means determines that a longer shutter speed is applied when there is a pixel with brighter illuminance than the average illuminance among the pixels in the moving object area. . The imaging device according to claim 8 or 9 , wherein the brightness information is an illuminance map. 11. The imaging device according to claim 10 , wherein the acquisition unit updates the value of a pixel having an illuminance brighter than the average illuminance among pixels in the moving object area in the illuminance map with the average illuminance. The imaging device according to claim 1, wherein the setting unit applies the same shutter speed to moving body parts that are determined to move at the same speed among areas corresponding to the moving body area . The imaging apparatus according to claim 1, wherein the specifying means specifies the moving object area according to the magnitude of movement based on the movement information. The setting means includes:
determining the basic exposure amount when photographing the image;
2. The imaging apparatus according to claim 1, wherein the photographing conditions are set for all pixels according to the basic exposure amount until the specifying means specifies the moving object area. The imaging device according to claim 1, wherein the photographing conditions are ISO sensitivity and shutter speed. The setting means is characterized in that, for at least some of the pixels corresponding to the non-moving object area, a shutter speed according to the brightness information that is different for each pixel according to the brightness information is applied. Item 1. The imaging device according to item 1. A program for causing a computer to function as the imaging device according to any one of claims 1 to 16 . A method for controlling an imaging device capable of capturing an image by controlling imaging conditions for each pixel or region, the method comprising:
a specifying step of specifying a moving object area in the image based on movement information of the subject;
an acquisition step of acquiring brightness information of the subject;
The photographing conditions are set for each pixel or region so that the same shutter speed is applied to the pixels corresponding to the moving object region, and the ISO sensitivity according to the brightness information is applied. For pixels corresponding to non-moving object areas that are not areas, the shooting conditions are set for each pixel or area so that the same ISO sensitivity is applied and a shutter speed according to the brightness information is applied. process and
A control method characterized by comprising:

Images