WO2019244686A1

WO2019244686A1 - Imaging device, control method for imaging device, and program

Info

Publication number: WO2019244686A1
Application number: PCT/JP2019/022851
Authority: WO
Inventors: 大輔入江; 寿人関根
Original assignee: キヤノン株式会社
Priority date: 2018-06-19
Filing date: 2019-06-10
Publication date: 2019-12-26

Abstract

This imaging device amplifies and thereafter digital-coverts signal voltage obtained by voltage conversion of a signal charge obtained by photoelectric conversion of an optical image of a subject, and comprises: an amplification circuit (104) that amplifies the signal voltage by two or more set amplification factors; a weight determination means (110) that determines a weight for respective signals digital-converted after being amplified on the basis of the two or more amplification factors; and a synthesis means (103) that synthesizes the two or more signals digital-converted after being amplified using the weight.

Description

Imaging device, imaging device control method, and program

The present invention relates to a technique for controlling an imaging device.

改善 In an imaging device, it is important to improve the S / N (SN ratio). A general imaging device has one amplifying means for one photoelectric conversion element, and amplifies an electric signal generated by the photoelectric conversion element. On the other hand, in order to improve the S / N, an imaging device that amplifies an electric signal generated by a photoelectric conversion element by two amplifying units is known (see Patent Literature 1, Patent Literature 2, Non-Patent Literature 1). . In such an imaging apparatus, one image is generated by appropriately selecting two amplified electric signals or a digital value obtained by AD (analog-digital) conversion of the two amplified electric signals according to luminance. Can improve the S / N.

JP 2016-129297 A US Patent Application Publication No. 2010/0177225

However, in the related arts (Patent Literature 1, Patent Literature 2, Non-Patent Literature 1), an amplification factor for an electric signal is determined before photographing. For this reason, for example, in a case where areas having different brightness in a shooting scene change rapidly, such as in outdoor shooting, there is a problem that if a preset amplification factor is used, a sufficient S / N improvement effect cannot be obtained. is there.

Therefore, an object of the present invention is to obtain a sufficient S / N improvement effect.

The image pickup apparatus of the present invention is an image pickup apparatus that performs digital conversion after amplifying a signal voltage obtained by voltage-converting a signal charge obtained by photoelectrically converting an optical image of a subject, wherein the signal voltage is set to two or more set respectively. An amplification circuit that amplifies the amplification factor according to the amplification factor; weight determination means that determines a weight for each of the digitally converted signals after amplification based on the two or more amplification factors; and the digital after the two or more amplification factors. Combining means for combining the converted signals using the weights.

According to the present invention, a sufficient S / N improvement effect can be obtained.

FIG. 2 is a diagram illustrating a configuration of an imaging device according to the first embodiment. 5 is a flowchart illustrating a flow of processing of the imaging device according to the first embodiment. 5 is a flowchart of an amplification factor determination process according to the first embodiment. FIG. 4 is an explanatory diagram of an amplification factor determination method according to the first embodiment. It is a flowchart of the amplification factor determination processing of 2nd Embodiment. It is explanatory drawing of the amplification factor determination method of 2nd Embodiment. It is explanatory drawing of the amplification factor determination method of 2nd Embodiment. It is a figure showing the composition of the imaging device of a 3rd embodiment. 13 is a flowchart illustrating a flow of processing of the imaging device according to the third embodiment. It is a figure showing the composition of the imaging device of a 4th embodiment. 15 is a flowchart illustrating a flow of a process of the imaging device according to the fourth embodiment. It is a flowchart of the amplification factor determination process of 4th Embodiment. It is explanatory drawing of the amplification factor determination method of 4th Embodiment. It is a flowchart of the amplification factor determination process of 5th Embodiment. It is explanatory drawing of the amplification factor determination method of 5th Embodiment. It is a figure showing the composition of the imaging device of a 6th embodiment. It is a flow chart which shows a flow of processing of an imaging device of a 6th embodiment. It is a flow chart which shows a flow of processing of an imaging device of an 8th embodiment. It is a flow chart which shows a flow of processing of an imaging device of a 9th embodiment. FIG. 11 is a diagram illustrating a configuration of an imaging device according to another embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not necessarily limited to the illustrated configurations.

<First embodiment>
FIG. 1 is a diagram illustrating a schematic configuration example of the imaging apparatus according to the first embodiment.

The imaging unit 100 includes an optical system 101, a photoelectric conversion element 102, an FD 103, and an analog processing circuit 106, and converts an optical image of a subject into image data. The optical system 101 includes a lens and an aperture, and forms an optical image from a subject or the like on an imaging surface of the photoelectric conversion element 102. The photoelectric conversion element 102 captures an optical image of a subject or the like by photoelectric conversion and converts it into a signal charge. The FD 103 is a floating diffusion amplifier that converts a signal charge output from the photoelectric conversion element 102 into a signal voltage by voltage conversion and outputs the signal voltage. The signal voltage output from the FD 103 is input to the analog processing circuit 106.

The analog processing circuit 106 includes the PGA 104 and the AD conversion circuit 105, and performs analog signal processing on the signal voltage input from the FD 103. The PGA 104 is a programmable gain amplifier, and is an amplifier circuit that amplifies the signal voltage input from the FD 103. The AD conversion circuit 105 performs analog-to-digital conversion for converting the signal voltage amplified by the PGA 104 into a digital value. Note that the analog processing circuit 106 includes a PGA 104 and an AD conversion circuit 105 that are divided into at least two systems. The two or more systems of the AD conversion circuit 105 output two or more systems of images based on digital values. , To the digital processing circuit 107.

The digital processing circuit 107 includes an image processing circuit 108, a luminance obtaining unit 109, and an amplification factor determining circuit 110, and performs digital signal processing on a captured image. The details of the digital processing circuit 107 will be described later.

The buffer 111 is a data buffer that stores a processing result in the luminance acquisition unit 109 and data being processed.

(4) The recording medium 113 is, for example, an SD card, a CF card (Compact Flash (registered trademark)), an HDD (hard disk drive), or the like that stores captured image data.

(4) The recording circuit 112 records the image data subjected to the digital signal processing by the digital processing circuit 107 on the recording medium 113, and reads out the image data from the recording medium 113 as necessary.

The system control unit 117 controls the entire imaging device.

The ROM 118 is a non-volatile memory that stores control data such as programs and various control parameters for the system control unit 117 to execute various controls as described below.

The RAM 116 is a volatile memory used when the system control unit 117 controls the imaging device.

The imaging control unit 114 is an imaging operation control unit that controls the operation of the imaging unit 100 in accordance with a control command from the system control unit 117.

(4) The gain control circuit 115 controls the gain of the signal voltage by changing the setting of the capacitance of the FD 103 and the setting of the gain of the PGA 104.

The operation unit 119 inputs an instruction from the outside of the imaging device (a connected device such as a user or a release) to the imaging device.

<Shooting operation flow of imaging device>
Hereinafter, the flow of the moving image capturing operation in the imaging device configured as described above will be described with reference to the configuration diagram illustrated in FIG. 1 and the flowchart illustrated in FIG. Each processing step in the flowchart of FIG. 2 is processing performed by each component of the imaging apparatus under the control of the system control unit 117. Further, in the description of the flowchart of FIG. 2, each processing step S201 to step S208 is abbreviated as S201 to S208. The same applies to other flowcharts described later.

In the present embodiment, as shown in FIG. 1, a configuration in which the analog processing circuit 106 includes two systems of the PGA 104 and the AD conversion circuit 105 is taken as an example, and the amplification factor is determined based on the average value and the standard deviation of the pixels. The processing will be described.

In S201 of FIG. 2, the imaging control unit 114 sets the imaging conditions (for example, shutter speed, aperture, ISO sensitivity) of the imaging device in accordance with an instruction input from the outside via the operation unit 119. When setting the imaging conditions, the imaging control unit 114 changes the setting of the aperture of the optical system 101 of the imaging unit 100 and the setting of the exposure time of the photoelectric conversion element 102.

In S202, the gain control circuit 115 sets the gain of the PGA 104 and the capacitance of the FD 103. This step is a pre-process performed only in the first frame of the moving image. When setting the amplification factor and the capacitance in S202, a value corresponding to the shooting condition or a setting value stored in the ROM 118 is used. Used.

In S203, the imaging control unit 114 starts the imaging operation of the imaging unit 100 in response to an imaging instruction input from the outside via the operation unit 119. Here, as a photographing operation, the imaging control unit 114 first drives the lens and aperture of the optical system 101 to form an optical image of the subject on the photoelectric conversion element 102. Further, the exposure time of the photoelectric conversion element 102 is controlled by a control signal from the imaging control unit 114, and the photoelectric conversion element 102 converts an optical image into a signal charge and outputs the signal charge to the FD 103. The signal charge output from the photoelectric conversion element 102 is converted into a signal voltage in the FD 103, and is then distributed to two paths (two systems) and sent to the analog processing circuit 106. The signal voltages input to the analog processing circuit 106 are respectively amplified by the two systems of PGAs 104, then converted into digital values by the two systems of AD conversion circuits 105, and output to the digital processing circuit 107. In the example of FIG. 1, since the PGA 104 and the AD conversion circuit 105 are provided for two systems, two images based on the two digital values from the AD conversion circuit 105 are output to the digital processing circuit 107. You.

In S204, the image processing circuit 108 performs an acquisition process of the two images output in S203 and an image synthesis process of outputting an image generated by synthesizing the two images. At this time, the image processing circuit 108 performs general image processing such as digital gain, white balance, and gamma correction on the acquired two images, and synthesizes the two images after the image processing. Perform processing. In the present embodiment, the processing method of the image synthesizing process differs depending on the amplification factors set for the two systems of images to be processed. For example, when the amplification factors set for the FD 103 and the PGA 104 in the imaging unit 100 are different between the two images to be processed, the image processing circuit 108 determines the image quality in an area where the luminance ranges of the two images overlap. A selection process is performed to selectively use good pixels. A pixel with good image quality is, for example, a pixel of an image for which a higher amplification factor is set. On the other hand, for example, when the amplification factors are the same for the two images to be processed, the image processing circuit 108 averages the pixel values of the pixels at the same position in the two images and uses the averaged pixel value. Perform processing. Then, images generated by the image synthesis processing by the image processing circuit 108 (hereinafter, referred to as synthesized images) are sequentially stored as frames constituting a moving image. Note that the moving image is converted into data of a predetermined recording format by the recording circuit 112, and then recorded and stored on the recording medium 113.

In S205, the luminance acquisition unit 109 acquires a representative value of the luminance of the image as luminance information. The acquisition of the luminance information is performed using any one of the two types of images processed by the image processing circuit 108. For example, the luminance acquisition unit 109 acquires luminance information from an image processed at a relatively low amplification rate among the two systems of images after the image processing. More specifically, the luminance obtaining unit 109 obtains a luminance value by performing a conversion process from a pixel value to a luminance value on a pixel value (sRGB value) obtainable from an image, and further obtains the luminance values. The average value, the standard deviation, and the maximum value are obtained as representative values, and they are used as luminance information. Then, the luminance acquiring unit 109 stores the acquired luminance information in the buffer 111. The conversion process from the pixel value to the luminance value can be realized by applying a general conversion matrix from the sRGB value to the luminance value for each pixel.

In S206, the amplification factor determination circuit 110 reads the average value, the standard deviation, and the maximum value of the luminance values acquired as the luminance information in S205 from the buffer 111, and based on the information, the amplification factor corresponding to the luminance of the subject. To determine. The details of the process of determining the amplification factor according to the luminance of the subject will be described later.

In S207, the gain control circuit 115 resets the gain in the imaging unit 100 based on the gain determined in S206. That is, the amplification factor control circuit 115 resets the capacitance of the FD 103 and the amplification factors of the two PGAs 104 based on the amplification factor determined in S206.

After that, in S208, the system control unit 117 determines the end of the shooting. In other words, if a shooting end instruction is input from the outside via the operation unit 119 while the processing from S203 to S208 is being performed, the system control unit 117 sends a shooting end instruction to the imaging control unit 114. Thus, the imaging control unit 114 ends the operation of the imaging unit 100. On the other hand, if the shooting is not to be ended because the shooting end instruction has not been input, the system control unit 117 returns the process to S203 and executes the processes from S203 to S207 again.

<How to determine the amplification factor>
Hereinafter, the process of determining the amplification factor in S206 will be described with reference to the flowchart in FIG. 3 and FIG. In the present embodiment, of the two systems, the gain with a relatively lower gain is referred to as a low gain, and the gain with a relatively higher gain is referred to as a high gain. In the present embodiment, a case will be described as an example where two values of a low gain having a relatively low amplification factor and a high gain having a relatively high amplification factor are determined. In the following description, an image obtained in the imaging unit 100 (FD 103 and PGA 104) with the low gain set is referred to as a low gain image, and an image obtained with the high gain set is referred to as a high gain. Called an image.

In step S301, the amplification factor determination circuit 110 acquires, from the buffer 111, the average value, the standard deviation, and the maximum value of the luminance values acquired as the luminance information in step S205.

In S302, the amplification factor determining circuit 110 determines a low gain based on the value obtained in S301.

Hereinafter, an example method for determining the low gain will be described with reference to FIG. FIG. 4 is a graph in which the horizontal axis represents the luminance value of the image, the vertical axis represents the frequency of appearance of the luminance value, and the distribution of the luminance value of the subject is represented by a normal distribution using the average value and the standard deviation calculated in S301. . In the present embodiment, the Low gain is set for the purpose of photographing the entire subject without overexposure. Therefore, the amplification factor determination circuit 110 sets the Low gain so that the entire normal distribution can be photographed as shown in the luminance range L of FIG. The amplification factor at this time can be expressed by equation (1).

In the equation (1), G _Low is a low gain, G _pre is an amplification factor already set for an image used for acquiring luminance information, μ is an average value of image luminance, and σ is a standard value of image luminance. The deviation, I _Max , represents the maximum possible luminance value of the image.

In S303, the amplification factor determination circuit 110 acquires the state of the “setting mode” of the priority luminance of the imaging device. The “setting mode” includes, for example, a low luminance priority setting mode, a medium luminance priority setting mode, and a high luminance priority setting mode, and can be set by a user or the like. Then, any one of the setting modes is selected at the time of setting the imaging conditions in S201.

In S304, the amplification factor determination circuit 110 determines a High gain based on the average value, the standard deviation, and the maximum value of the luminance values acquired in S301.

The following is an example of a method for determining the High gain. In the present embodiment, a case where two modes, a low brightness priority setting mode and a middle brightness priority setting mode, can be set will be described as an example. When the low-luminance priority setting mode is set, the amplification factor determination circuit 110 sets the High gain so that a luminance range indicated by a range H1 in FIG. 4 can be captured, for example. Further, when the mode is set to the medium luminance priority setting mode, the amplification factor determining circuit 110 sets the High gain so that a luminance range indicated by a range H2 in FIG. 4 can be photographed, for example. The High gains when photographing the respective luminance ranges of the ranges H1 and H2 can be expressed by Expressions (2) and (3), respectively.

G _{High1 in} Expression (2) represents a High gain when capturing the luminance range of the range H1, and G _{High2 in} Expression (3) represents a High gain when capturing the luminance range in the range H2. When the High gain calculated by Expression (2) or Expression (3) is expressed as G _High without distinction, the amplification factor determination circuit 110 determines, from among the amplification factors that can be set by the imaging device, a value closest to G _High. Is the amplification value of the High gain that is actually set. For example, when the imaging device can set the amplification factors of 1, 2, and 4 times, and the G _High calculated by the expression (2) or (3) is 2.5, the amplification factor determination circuit 110 Selects twice as the amplification value of the High gain that is actually set. If the luminance distribution of the subject is not wider than, for example, a predetermined luminance distribution, the same amplification factor is set for G _Low and G _High among the amplification factors that can be set by the imaging device.

In general, it is known that the characteristics of an output image change depending on the amplification factor. For this reason, for example, characteristics are different between a low gain image and a high gain image. Specifically, a low gain image has a relatively wide range of luminance that can be photographed, but has a low S / N (SN ratio) with respect to luminance, as compared with a high gain image. Conversely, a high-gain image has a relatively narrower luminance range than a low-gain image, but has a higher S / N with respect to luminance. Such a relationship is the same for an image in which the amplification rate is set based on the values of G _High1 and G _High2 . That is, in the low luminance priority setting mode, a relatively high S / N improvement effect can be obtained only in the low luminance region of the image. On the other hand, in the medium luminance priority setting mode, the S / N improvement effect is lower than in the low luminance priority setting mode, but the S / N in a wider luminance region can be improved.

In the above-described example, a method of determining two gains of a low gain and a high gain has been described. However, the imaging apparatus has, for example, three or more systems of the PGA 104 and the AD conversion circuit 105 and at least three or more gains. This embodiment is also applicable to the case of determining In this case, the amplification factor determination circuit 110 sets the same amplification factor as the above-mentioned High gain for the third and subsequent amplification factors. Further, at the time of the image synthesizing process in S203, the image processing circuit 108 performs the process of averaging the above-described pixel values at the same position between the images having the same amplification factor, and synthesizes the image with the Low gain image.

The above is the processing performed by the imaging device of the first embodiment. As described above, according to the present embodiment, the S / N is improved irrespective of the luminance of the subject by determining the amplification factor using the luminance information such as the average value and the standard deviation of the luminance of the subject. The effect can be obtained.

<Second embodiment>
In the above-described first embodiment, the amplification factor is determined based on the average value and the standard deviation of the luminance values of the image. On the other hand, in the second embodiment, the skewness and kurtosis of the luminance value are calculated as representative values of the luminance of the image, and the amplification factor is determined using these as luminance information. The configuration of the imaging apparatus according to the second embodiment is the same as the configuration shown in FIG. The flow of processing in the second embodiment differs from that of the first embodiment in S205 and S206 in FIG. In the first embodiment, in S205, an average value, a standard deviation, and a maximum value of luminance values are obtained, and these are stored in the buffer 111 as luminance information. In contrast, in the second embodiment, in S205, the skewness and kurtosis of the luminance value are obtained, and these are stored in the buffer 111 as luminance information.

Hereinafter, processing performed by the imaging apparatus according to the second embodiment will be described with reference to a flowchart illustrated in FIG. 5, focusing on differences from the first embodiment. The flowchart in FIG. 5 illustrates a process corresponding to S206 in FIG. Regarding the processing method of S206 in the second embodiment, of the two systems, two values of a low gain of a relatively lower amplification factor and a High gain of a relatively higher amplification factor are determined. The case will be described as an example.

In S501, the amplification factor determination circuit 110 obtains, from the buffer 111, the skewness and kurtosis of the brightness value obtained as the brightness information in S205 of the second embodiment.

<Skewness and kurtosis>
Here, the skewness and the kurtosis are indices representing characteristics of the distribution of data. The skewness is a value indicating how asymmetric the distribution of the data is, and the kurtosis is a value indicating how sharp the distribution of the data is compared to the normal distribution. The brightness acquisition unit 109 in the case of the second embodiment calculates the skewness S by the following equation (4), and calculates the kurtosis K by the equation (5).

Here, in Equations (4) and (5), S is skewness, K is kurtosis, n is the total number of image luminance data, x is the i-th luminance value, and σ is the image luminance standard deviation. , Μ are the average values of the luminance of the image. In the case of the second embodiment, the skewness and kurtosis of these luminance values are held in the buffer 111 as luminance information.

In S502, the amplification factor determination circuit 110 calculates a coefficient using a table described later based on the skewness and kurtosis of the luminance value acquired as the luminance information in S501. For convenience, the table here is called a coefficient table, the details of which will be described later. The coefficient table is held in the ROM 118, and the calculation of the coefficient in the coefficient table is performed in the digital processing circuit 107, for example.

In S503, the amplification factor determination circuit 110 determines the values of the low gain and the high gain. The amplification factor determination circuit 110 determines the amplification factor set for the image used for calculating the skewness and kurtosis as the luminance information in S205 and the coefficient calculated based on the skewness and kurtosis from the coefficient table in S502. Is obtained, and this value is used as the amplification factor.

<Coefficient table>
Hereinafter, the coefficient table will be described with reference to FIGS. 6A and 6B. FIG. 6A shows an example of a coefficient table for calculating a coefficient for a low gain, and FIG. 6B shows an example of a coefficient table for calculating a coefficient for a high gain. 6A and 6B, the horizontal axis represents the skewness and the vertical axis represents the kurtosis. As described above, the value of the skewness is an index of the bias of the brightness of the subject, and the value of the kurtosis is the bias of the brightness. It is an index of how to sharpen. For example, when the skewness is a negative value, the luminance of the subject is biased toward the high luminance side, and when the skewness value is a positive value, the luminance of the subject is biased toward the low luminance side. The greater the absolute value of the skewness, the greater the bias. The kurtosis becomes sharper as its value increases. The value in the coefficient table is a coefficient that is multiplied by the amplification factor set for the image used for calculating the skewness and kurtosis of the luminance value in S501. For example, if the amplification factor set for the image used for calculating the skewness and kurtosis is 2, and the value determined from the coefficient table is 8, the amplification factor determined here is 2 × 8 = 16. Become.

The coefficient table used for determining the amplification factor is not limited to the two coefficient tables of the low gain and the high gain, and may be created according to the number of amplification factors to be determined. Further, the coefficients described in the coefficient table are not limited to the values shown in FIGS. 6A and 6B.

In the present embodiment, a method of determining two gains of a low gain and a high gain has been described. However, the imaging apparatus has three or more PGAs 104 and AD conversion circuits 105, and has at least three or more gains. It may be decided. In such a case, the third and subsequent amplification factors are set to the same amplification factors as the High gain described above. In this case, at the time of the image synthesizing process in S204, the image processing circuit 108 performs the process of averaging the pixel values at the same position described above between the images having the same amplification factor, and performs the image synthesizing with the Low gain image. .

The above is the processing performed by the imaging device of the second embodiment. As described above, according to the present embodiment, the S / N is improved irrespective of the luminance of the subject by determining the amplification factor using the skewness and the kurtosis luminance information of the luminance value of the subject. The effect can be obtained.

<Third embodiment>
In the third embodiment, a method of determining an amplification factor according to the luminance of a subject with a configuration different from the imaging devices of the first embodiment and the second embodiment will be described. In the third embodiment, in addition to a memory (buffer 111) for holding a representative value of luminance values (average and standard deviation, skewness and kurtosis, etc.), a charge memory for holding signal charges is provided. A method will be described in which it is not necessary to determine an amplification factor in advance for an image before a frame. FIG. 7 is a configuration diagram illustrating an example of a configuration of an imaging device according to the third embodiment. FIG. 7 is different from the imaging apparatus in FIG. 1 in the configuration of the imaging unit 100. The imaging unit 100 is divided into, for example, two systems after the photoelectric conversion element 102, and a charge memory 700 is provided in front of each FD 103 divided into two systems. Is provided. The charge memory 700 has a function of holding signal charges generated in the photoelectric conversion element 102. The imaging device in FIG. 7 has a configuration in which the charge memory 700 is provided for only two systems. However, when the FD 103, the PGA 104, and the AD conversion circuit 105 are divided into three or more systems, only the number corresponding to the number of systems is provided. May be provided. In the configuration diagram of FIG. 7, the components corresponding to those in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted.

<Description of imaging operation in imaging apparatus of third embodiment>
Hereinafter, the flow of the photographing operation in the imaging device of the third embodiment will be described with reference to the configuration diagram of FIG. 7 and the flowchart illustrated in FIG. In this embodiment, an example will be described in which a charge memory 700 for two systems is provided and two amplification factors are determined.

In S801, the imaging control unit 114 sets the imaging conditions (shutter speed, aperture, and ISO sensitivity) of the imaging device according to an instruction input from the outside via the operation unit 119. In setting the imaging conditions, the imaging control unit 114 sets the aperture of the optical system 101 of the imaging unit 100, the exposure time of the photoelectric conversion element 102, and the set values of the capacitance of the FD 103 and the amplification factor of the PGA 104 in each system. change.

In S802, the gain control circuit 115 sets the capacitance of the FD 103 and the gain of the PGA 104 in each system. As the set value, a value corresponding to the photographing condition or a value described in the ROM 118 is used.

In S803, the imaging control unit 114 starts the imaging operation of the imaging unit 100 in response to an imaging instruction input from the outside via the operation unit 119. At this time, the imaging control unit 114 first drives the lens and the aperture of the optical system 101 to form an optical image of the subject on the photoelectric conversion element 102. Then, the signal charges transferred from the photoelectric conversion element 102 are distributed to two systems, sent to the charge memory 700, and held. Next, the imaging control unit 114 causes the signal charge held in the charge memory 700 on the path side for setting the low gain, which is the relatively lower amplification rate, of the two systems to be transferred to the FD 103. The FD 103 converts the transferred signal charge into a signal voltage and outputs the signal voltage to the analog processing circuit 106. The output signal voltage is amplified by the PGA 104 in the analog processing circuit 106, converted into a digital value by the AD conversion circuit 105, and output to the digital processing circuit 107.

In S804, the image processing circuit 108 acquires a low gain image. Specifically, the image processing circuit 108 performs general image processing such as digital gain, white balance, and gamma correction on the digital value acquired in step S803 as necessary, and acquires a low gain image. .

In S805, the luminance acquisition unit 109 acquires the luminance information of the Low gain image. Specifically, similarly to S205 described above, a conversion process from a pixel value to a luminance value is performed on the pixel value (sRGB value). Further, the luminance acquisition unit 109 calculates a representative value of the luminance value and stores the representative value in the buffer 111 as luminance information. As the representative value of the luminance value, the average value, the standard deviation, and the maximum value of the luminance value described in the first embodiment, and the skewness and kurtosis described in the second embodiment can be exemplified.

In step S806, the amplification factor determination circuit 110 acquires the representative value (luminance information) of the luminance value calculated in step S805 from the buffer 111, and determines the amplification factor according to the luminance of the subject based on the acquired representative value. I do. As a method of determining the amplification factor suitable for the luminance of the subject, one of the methods described in the first embodiment and the second embodiment is used. For example, when the average value, the standard deviation, and the maximum value of the luminance values described in the first embodiment are used as the representative values of the luminance values, the amplification factor determination method described in the first embodiment is used. . For example, when the skewness and the kurtosis described in the second embodiment are used, the method for determining the amplification factor described in the second embodiment is used. Note that the method of determining these two amplification factors is the same as described above, and a description thereof will be omitted.

In step S807, the amplification factor control circuit 115 resets the amplification factor (the capacitance of the FD 103 and the amplification factor of the PGA 104) in the path for setting the High gain of the two systems based on the amplification factor determined in S806. Reset).

In S808, the image processing circuit 108 acquires a High gain image. At this time, the gain control circuit 115 transfers the signal charge held by the charge memory 700 connected to the path for setting the High gain to the FD 103 and converts it into a signal voltage, and the gain is reset. The PGA 104 amplifies the signal voltage. Thereafter, the signal voltage amplified by the PGA 104 is converted to a digital value by the AD conversion circuit 105. Then, the image processing circuit 108 performs general image processing such as digital gain, white balance, and gamma correction on the input digital values as necessary. As a result, a High gain image amplified at the reset amplification factor is obtained.

In S809, the image processing circuit 108 performs a synthesis process of the Low gain image acquired in S804 and the High gain image acquired in S808. For example, when the amplification factors set in the imaging unit 100 are different between the two systems, the image processing circuit 108 selectively selects pixels of an image having a high S / N in a region where the luminance ranges of the two systems overlap. A selection process such as that used for is performed. On the other hand, when the amplification factors are the same in the two systems, the image processing circuit 108 performs an averaging process in which pixels at the same position in the two systems of images are averaged and used. After that, the combined image after the image combining processing by the image processing circuit 108 is converted into data of a predetermined recording format by the recording circuit 112, and then recorded and stored in the recording medium 113.

The above is the processing performed by the imaging device according to the third embodiment. As described above, according to the present embodiment, the charge memory for holding the signal charge is provided, and for example, the luminance of the subject can be obtained even if the amplification factor is not determined in advance using the image one frame before. The amplification factor can be determined based on the information. In the third embodiment as well, similar to the first and second embodiments, an S / N improvement effect can be obtained regardless of the luminance of the subject.

<Fourth embodiment>
In the first to third embodiments, an average value, a standard deviation, and a maximum value of luminance values of an image or representative values such as skewness and kurtosis are acquired as luminance information to determine an amplification factor. I have. An imaging apparatus according to a fourth embodiment described below includes an image memory 900 having a role of storing an image, obtains a luminance histogram from the image stored in the image memory 900, and uses the histogram having a low frequency in the image based on the histogram. Determine the area. Then, in the imaging device of the fourth embodiment, the amplification factor is determined according to the determination result of the luminance region. That is, in the fourth embodiment, a luminance distribution of a shooting scene is obtained in detail, a luminance region having a low frequency in an image is determined based on the luminance distribution, and an amplification factor is determined based on the determination result of the luminance region. I do. For this reason, in the imaging apparatus of the fourth embodiment, a more appropriate setting of the amplification factor for the shooting scene is performed, compared to the method of determining the amplification factor based on the representative values described in the first embodiment to the third embodiment. It can be carried out.

FIG. 9 is a configuration diagram illustrating an example of a configuration of an imaging device according to the fourth embodiment. The imaging device according to the fourth embodiment shown in FIG. 9 is different from the imaging device shown in FIG. 1 in that an image memory 900 is provided. The image memory 900 has a function of holding an image. The image memory 900 can hold an image acquired by the imaging unit 100, a luminance image acquired as luminance information in S1006 in FIG. In the configuration diagram of FIG. 9, the components corresponding to those in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted.

<Description of Photographing Operation in Imaging Device of Fourth Embodiment>
Hereinafter, the flow of the image capturing operation in the image capturing apparatus according to the fourth embodiment will be described with reference to the configuration diagram of FIG. 9 and the flowchart illustrated in FIG. Note that, in the fourth embodiment, as in the above-described embodiment, an example in which the amplification factor is determined for two systems will be described.

In S1001, similarly to S201 described above, the imaging control unit 114 sets the imaging conditions of the imaging device according to an instruction input from the outside via the operation unit 119.

In step S1002, similarly to step S202, the amplification factor control circuit 115 sets the amplification factor of the PGA 104 and the capacitance of the FD 103.

In S1003, as in S203, the imaging control unit 114 starts the imaging operation of the imaging unit 100 in response to an imaging instruction input from outside via the operation unit 119. In the same manner as described above, the signal charge transferred from the photoelectric conversion element 102 is converted into a signal voltage in the FD 103, then divided into two systems, and sent to the analog processing circuit 106. Then, as described above, in the analog processing circuit 106, the input signal voltage is amplified by the PGA 104, converted into a digital value by the AD conversion circuit 105, and output to the digital processing circuit 107.

In S1004, as in S204, the image processing circuit 108 performs two-system image acquisition processing and image synthesis processing. Then, the composite image after the processing by the image processing circuit 108 is converted into data of a predetermined recording format by the recording circuit 112 and then recorded on the recording medium 113.

In step S <b> 1005, the digital processing circuit 107 causes the image memory 900 to store the image with the lowest amplification factor set among the images acquired from the imaging unit 100.

In S1006, the brightness acquisition unit 109 acquires the brightness information of the image stored in the image memory 900. The luminance obtaining unit 109 performs a conversion process from a pixel value to a luminance value on a pixel value (sRGB value) obtainable from an image stored in the image memory 900. Then, the luminance acquisition unit 109 stores the image (hereinafter, referred to as a luminance image) obtained by performing the conversion process into the luminance value in the image memory 900 as luminance information. The conversion process from the pixel value to the luminance value is the same as described above, and is performed by applying a general conversion matrix from the sRGB value to the luminance value to each pixel.

In S1007, the amplification factor determination circuit 110 reads the luminance image acquired as the luminance information in S1006 from the image memory 900, and determines an amplification factor according to the luminance of the subject based on the luminance image. The details of the process of determining the amplification factor according to the luminance of the subject in the fourth embodiment will be described later.

In S1008, the gain control circuit 115 resets the gain in the imaging unit 100 based on the gain determined in S1007. That is, the amplification factor control circuit 115 resets the capacitance of the FD 103 and the amplification factors of the two PGAs 104 based on the amplification factor determined in S1007.

After that, in S1009, the system control unit 117 determines the end of the shooting. If a shooting end instruction is input from the outside via the operation unit 119 while the processing from S1003 to S1008 is being performed, the system control unit 117 sends a shooting end instruction to the imaging control unit 114 and The operation of is ended. On the other hand, if the shooting end instruction has not been input and the shooting is not to be ended, the system control unit 117 returns the process to S1003 and executes the processes from S1003 to S108 again.

<Amplification Ratio Determination Process in Fourth Embodiment>
Hereinafter, the amplification factor determination processing in S1007 will be described using the flowchart of FIG. 11 and FIG. Similarly to the above, in the present embodiment, as an example, a case where two values of a low gain of a relatively lower amplification factor and a High gain of a relatively higher amplification factor of two systems are determined will be described as an example. Give an explanation.

In S1101, the brightness acquisition unit 109 calculates a brightness histogram based on the brightness image stored as the brightness information in S1006, and stores the calculated histogram information in the buffer 111.

In S1102, the amplification factor determination circuit 110 calculates a value that is a candidate for an amplification factor that becomes a High gain (hereinafter, referred to as a candidate value).

Hereinafter, a method of calculating a candidate value will be described with reference to FIG.

First, as shown in FIG. 12, the amplification factor determination circuit 110 performs a threshold process on the frequency of the histogram, and changes the point where the luminance exceeds the threshold in a direction of increasing luminance from a part below the threshold (hereinafter, referred to as a point below). Inflection point). In the example of FIG. 12, inflection points IP1, IP2, and IP3 are obtained. The threshold value is set to a value that prevents erroneous detection of an inflection point due to an increase or decrease in the frequency of occurrence due to noise.

Next, the amplification factor determination circuit 110 sets an amplification factor that can photograph each luminance range from the luminance value “0” to the respective inflection points IP1, IP2, and IP3 acquired as described above as candidate values. . In the example of FIG. 12, the amplification factors that can be photographed in the ranges H1, H2, and L, which are the luminance ranges from the luminance value “0” to the inflection points IP1, IP2, and IP3, are set as candidate values. An amplification factor that enables the photographing of the luminance range from the luminance value “0” to the inflection point is calculated by Expression (6).

In the equation (6), I _C is a luminance value at an arbitrary inflection point, I _Max is a maximum luminance value that the image can take, G _pre is an amplification factor set for the image used for the histogram calculation, and G _C is an arbitrary value. This is the amplification factor when capturing the luminance range up to the inflection point. Amplification factor decision circuit 110 performs the calculation of the gain G _C for each inflection point, and the candidate value the value of each gain G _C obtained.

In S1103, the amplification factor determining circuit 110 determines the value of the low gain. For example, the Low gain is set to a value corresponding to the inflection point having the highest luminance (inflection point IP3 in FIG. 12) among a plurality of inflection points so that the entire luminance range of the subject can be photographed. When the number of inflection points of the histogram is zero or the frequency of the histogram exceeds the threshold value in the luminance range after the inflection point with the highest luminance, the Low gain is determined by the image used for the histogram calculation (Low gain in the present embodiment). Image) is one step lower than the amplification factor. As the amplification factor, for example, values of 2, 3, 4, and 5 can be set. When the amplification factor set to the image used for calculating the histogram is 4, the Low gain is 1 It is set to three times the lower value. At this time, when the amplification factor of the image used for the calculation of the histogram is twice the lower limit of the set value, the Low gain is twice the lower limit.

In S1104, the amplification factor determination circuit 110 calculates an image quality evaluation value for each amplification factor candidate value.

Hereinafter, the calculation process of the evaluation value of the image quality will be described. First, the amplification factor determination circuit 110 sets any one of the at least two candidate values calculated in S1102, which is different from the candidate value used when obtaining the Low gain, as the High gain. . When two or more candidate values do not exist, the High gain takes the same value as the Low gain. After that, the amplification factor determining circuit 110 proceeds to the process of S1008.

Next, the amplification factor determination circuit 110 calculates the image quality of the composite image when the setting is performed in this manner. In the case of the present embodiment, S / N is used as the evaluation value of the image quality. The S / N of the composite image can be expressed by equation (7).

In equation (7), S / N _Mix is the S / N of the composite image, I is the luminance value, N (I) is the frequency of the luminance value I, and I _High is the maximum luminance value of the High gain. In the equation (7), G _High is a high gain, G _Low is a low gain, σ (G _High , I) is a noise amount at the luminance value I at the time of the high gain, and σ (G _Low , I) is a noise amount at the time of the low gain. This is the amount of noise at the luminance value I.

Next, the amplification factor determining circuit 110 similarly calculates the S / N _Mix from the equation (7) when another candidate value is set to the High gain, and performs this for each candidate value. In the present embodiment, S / N is used as the image quality evaluation value, but the image quality evaluation value in the present invention is not limited to this.

In S1105, the amplification factor determination circuit 110 determines the value of the High gain. For example, the amplification factor determination circuit 110 sets a candidate value when the image quality of the combined image is the best, based on the image quality evaluation value (S / N _Mix in this embodiment) for each candidate value calculated in S1104, as a High gain. .

In the present embodiment, a method of determining two gains of a low gain and a high gain has been described. However, the imaging unit 100 has three or more systems of the PGA 104 and the AD conversion circuit 105 and determines at least three or more gains. In some cases. In such a case, the third and subsequent amplification factors are set to the same amplification factors as the High gain described above. Further, at the time of the image synthesizing process in S1004, the image processing circuit 108 first performs the process of averaging the pixel values at the same position described above between the images for which the same amplification factor is set, and then performs the image synthesis with the Low gain image. Do.

The above is the processing performed in the fourth embodiment. As described above, according to the fourth embodiment, the amplification factor is determined in consideration of a portion where the frequency of the histogram is low. On the other hand, it is possible to set a more appropriate amplification factor.

<Fifth embodiment>
In the fifth embodiment, the amplification factor is determined based on the histogram of the image and the importance of the peak of the histogram. The configuration of the imaging apparatus according to the fifth embodiment is the same as the configuration shown in FIG. The flow of processing in the fifth embodiment differs from that of the fourth embodiment in S1007 in FIG. Here, the fifth embodiment will be described focusing on the differences from the fourth embodiment. Hereinafter, a flow of a process performed by the imaging device according to the fifth embodiment will be described with reference to a flowchart illustrated in FIG. 13 and FIG. 9. Also in the fifth embodiment, a case will be described in which two values of a low gain having a relatively low amplification factor and a high gain having a relatively high amplification factor are determined as in the above-described embodiment.

In S1301, the luminance acquisition unit 109 calculates a luminance histogram from the luminance image acquired as the luminance information in S1006, and stores the result in the buffer 111.

In S1302, the amplification factor determination circuit 110 divides the histogram into regions in the luminance direction. In step S1302, the amplification factor determination circuit 110 first performs threshold processing on the frequency of the histogram as shown in FIG. 14, and changes from a portion exceeding the threshold to a portion below the threshold in a direction in which the luminance increases. Get the inflection point. In the example of FIG. 14, inflection points IP1, IP2, and IP3 are obtained. The threshold value is set to a value that prevents erroneous detection of an inflection point due to an increase or decrease in frequency caused by noise, as described with reference to FIG. Next, the amplification factor determination circuit 110 determines the range from the luminance value “0” to the inflection point with the lowest luminance value (region R1 in FIG. 14) and the range between two consecutive inflection points (FIG. The region R2 and the region R3) are defined as one region.

In S1303, the amplification factor determining circuit 110 determines the value of the low gain. As in the above-described step S1103, for example, the low gain is set to the inflection point having the highest luminance (inflection point IP3 in FIG. 14) among a plurality of inflection points so that the entire luminance range of the subject can be photographed. Take the combined value. As described above, when the number of inflection points of the histogram is zero or the frequency of the histogram exceeds the threshold value in the luminance range after the inflection point having the highest luminance, the Low gain is set to the image used for calculating the histogram (Low gain image). ) Is one step lower than the amplification factor.

In S1304, the amplification factor determination circuit 110 sets the importance for each of the regions divided in S1302. In the case of the present embodiment, it is assumed that the index of importance uses the maximum value. The amplification factor determination circuit 110 obtains the maximum value of the frequency of the histogram for each area, and defines the importance in the descending order of the maximum value. For example, the amplification factor determining circuit 110 determines that the region having the largest maximum value has the highest importance as "1" and the region having the second largest value has the second highest importance. The importance is defined as the degree “2”. The index of importance is not limited to the maximum value. For example, the total number of frequencies in the area may be used as an index of importance.

In S1305, the amplification factor determining circuit 110 determines the value of the High gain. The High gain is set to a value such that an image can be captured up to the region of the highest importance defined in S1304. Specifically, at the inflection point when the region having the importance “1” is set, the amplification factor determination circuit 110 determines the amplification factor such that the luminance range up to the inflection point with higher luminance can be captured. And The amplification factor at this time is calculated using equation (6) described in the fourth embodiment. If two or more inflection points do not exist, the High gain takes the same value as the Low gain, and the process proceeds to S1008.

In the present embodiment, a method of determining two gains of a low gain and a high gain has been described. However, the imaging unit 100 has three or more systems of the PGA 104 and the AD conversion circuit 105 and determines at least three or more gains. In some cases. In this case, the third and subsequent amplification factors are set to the same amplification factors as the High gain described above. Then, at the time of the image synthesizing process in S1004, the image processing circuit 108 performs an averaging process between the images having the same amplification factor, and synthesizes the image with the Low gain image.

The above is the processing performed in the fifth embodiment. In the fifth embodiment, the importance is calculated for each of the divided areas of the histogram, and the amplification rate is determined based on the importance. On the other hand, it is possible to set a more appropriate amplification factor.

<Sixth embodiment>
In the sixth embodiment, a method of determining the amplification factor according to the luminance of the subject with a configuration different from the imaging devices of the fourth and fifth embodiments will be described.

FIG. 15 is a configuration diagram illustrating an example of a configuration of an imaging device according to the sixth embodiment. In the case of FIG. 15, the configuration inside the imaging unit 100 is different from the imaging device of FIG. 9, and is divided into two systems at the subsequent stage of the photoelectric conversion element 102 as in the configuration example of FIG. A charge memory 700 is provided in a stage preceding the FD 103. Although the imaging device in FIG. 15 has a configuration in which the charge memory 700 is provided for only two systems, it may be divided into three or more systems. In the configuration diagram of FIG. 15, the components corresponding to those in FIGS. 1, 7, and 9 are denoted by the same reference numerals, and description thereof will be omitted.

<Description of Photographing Operation in Imaging Device of Sixth Embodiment>
Hereinafter, the flow of the imaging operation in the imaging device of the sixth embodiment will be described with reference to the configuration diagram of FIG. 15 and the flowchart illustrated in FIG. In the present embodiment, an example will be described in which the imaging apparatus has two sets of charge memories 700 and determines two amplification factors.

In step S1601, similarly to S801 described above, the imaging control unit 114 sets the imaging conditions of the imaging device according to an instruction input from outside the imaging device via the operation unit 119.

In S1602, similarly to S802 described above, the gain control circuit 115 sets the capacitance of the FD 103 and the gain of the PGA 104 in each system.

In S1603, similarly to S802 described above, the imaging control unit 114 starts the imaging operation of the imaging unit 100 in response to an imaging instruction input from the outside via the operation unit 119. Then, the signal charges transferred from the photoelectric conversion element 102 are distributed to two systems, sent to the charge memory 700, and held. Further, in the same manner as described above, the imaging control unit 114 causes the signal charge held in the charge memory 700 on the path side for setting a low gain, which is a relatively low amplification rate, of the two systems to be transferred to the FD 103. The FD 103 converts the transferred signal charge into a signal voltage and outputs the signal voltage to the analog processing circuit 106. The output signal voltage is amplified by the PGA 104 in the analog processing circuit 106, converted to a digital value by the AD conversion circuit 105, and output to the digital processing circuit 107.

In step S1604, similarly to step S804, the image processing circuit 108 performs general image processing as necessary to obtain a low gain image.

In S1605, the digital processing circuit 107 causes the image memory 900 to store the Low image acquired in S1604.

In step S1606, the brightness acquisition unit 109 acquires the brightness information of the image stored in the image memory 900. That is, the luminance obtaining unit 109 performs a conversion process from a pixel value to a luminance value on a pixel value (sRGB value) obtainable from the image stored in the image memory 900. Then, the brightness acquisition unit 109 stores the brightness image obtained by performing the conversion process into the brightness value in the image memory 900 as brightness information.

In S1607, the amplification factor determination circuit 110 reads the luminance image acquired as the luminance information in S1606 from the image memory 900, and determines an amplification factor suitable for the luminance of the subject based on the luminance image. The process of determining the amplification factor suitable for the brightness of the subject uses one of the amplification factor determination processes described in the fourth and fifth embodiments. Since these two amplification factor determination processes have been described above, the description here is omitted.

In S1608, the gain control circuit 115 resets the gain in the path for setting the High gain based on the gain determined in S1607. That is, the amplification factor control circuit 115 resets the capacitance of the two FDs 103 and the amplification factor of the PGA 104 based on the amplification factors determined in S1607.

In S1609, the image processing circuit 108 acquires a High gain image. At this time, the amplification factor control circuit 115 transfers the signal charge held by the charge memory 700 connected to the path for setting the High gain to the FD 103 to convert the signal charge into a signal voltage, and further, the PGA 104 amplifies the signal voltage. Thereafter, the data is converted into a digital value by the AD conversion circuit 105. Then, the image processing circuit 108 performs the above-described general image processing on the input digital value as necessary, and acquires a High gain image.

In S1610, the image processing circuit 108 performs a combining process of the Low gain image acquired in S1604 and the High gain image acquired in S1609. Here, when the amplification factors set in the imaging unit 100 are different between the two systems, the image processing circuit 108 determines that the pixels of the high-quality image (for example, A selection process is performed to selectively use pixels (a pixel for which a high amplification factor is set). On the other hand, when the amplification factors are the same in the two systems, the image processing circuit 108 performs an averaging process in which pixels at the same position in the two systems of images are averaged and used. Then, the combined image after the image combining processing by the image processing circuit 108 is converted into data of a predetermined recording format by the recording circuit 112, and then recorded and stored on the recording medium 113.

The above is the processing performed by the imaging device of the sixth embodiment. The imaging apparatus according to the sixth embodiment has a charge memory for holding signal charges, and performs the amplification factor determination processing of any of the fourth embodiment and the fifth embodiment described above. Even if it is not determined, the amplification factor can be determined based on the luminance information of the subject.

<Seventh embodiment>
In the synthesizing process according to the first to sixth embodiments described above, the image processing circuit 108 selects a pixel having a high amplification factor when the amplification factor is different in an area where the luminance range overlaps, and performs amplification. If the rates are the same, they are averaged and output. On the other hand, in the seventh embodiment, the image processing circuit 108 synthesizes using the weight based on the amplification factor.

合成 The following describes a combining method using weights based on amplification factors. Since the configuration diagram is the same as that of the first embodiment, illustration and description are omitted. In the case of the seventh embodiment, the combining process using the weight based on the amplification factor is performed in steps such as S204 described above.

In the seventh embodiment, the high gain of the gain determined by the gain determination circuit 110 is G _High , the low gain is G _Low , and the composite weight of the High gain image I _High and the Low gain image I _Low is w, The saturated pixel value of the low gain image is defined as Max. In this case, the composite image I _{Mix at} the (x, y) coordinates of the High gain image I _High and the Low gain image I _Low can be expressed by Expression (8).

The standard deviation of the light shot noise is σ _S , the standard deviation of the dark current noise is σ _d , the read noise generated when the amplification factor is High is σ _High , and the read noise generated when the amplification factor is Low is Let it be σ _Low . Standard deviation sigma _Mix noise of the combined image _{I Mix} in this case can be represented by the formula (9).

Then, the composite weight w that minimizes the standard deviation σ _Mix can be expressed as in Expression (10).

The image processing circuit 108 performs synthesis by the synthesis method represented by Expression (8) using the synthesis weight w obtained by Expression (10). In Equation (10), the composite weight w increases as the ratio (G _High / G _Low ) between the high gain gain G _High and the low gain gain G _Low approaches 1, and the gains are equal. That is, in the case of 1.0, the maximum w = 0.5. At this time, the composite image I _Mix is a result of averaging the High gain image I _High and the Low gain image I _Low .

The above is the processing performed in the seventh embodiment. In the seventh embodiment, a high gain image and a low gain image are combined using a weight in which the read noise amount is added to the amplification rates of the high gain and the low gain. Thus, according to the seventh embodiment, the amount of noise after combination can be reduced as compared with the case where the combination processing described in the first embodiment to the sixth embodiment is performed.

<Eighth embodiment>
In the first embodiment described above, the gains of the high gain and the low gain are determined based on the gain of the previous frame. On the other hand, in the eighth embodiment, the gain of the high gain and the gain of the low gain are determined at the time of the preliminary shooting, and the gain of the low gain is fixed and the gain of the high gain alone is changed at the time of the main shooting. An example will be described. Since the configuration diagram is the same as that of the first embodiment, illustration and description are omitted.

<Description of Photographing Operation in Imaging Device of Eighth Embodiment>
Hereinafter, the flow of the imaging operation in the imaging device of the eighth embodiment will be described with reference to the flowchart shown in FIG.

In S1701 in FIG. 17, the imaging control unit 114 sets the imaging conditions of the imaging device according to an instruction input from the outside via the operation unit 119, as in S201 described above.

In S1702, the amplification factor control circuit 115 sets the amplification factor of the PGA 104 and the capacitance of the FD 103 as in S202 described above.

In S1703, the imaging control unit 114 starts the preliminary imaging operation of the imaging unit 100 in response to an imaging instruction input from the outside via the operation unit 119.

In S1704, the image processing circuit 108 performs an acquisition process of the two images output in the preliminary photographing operation in S1703, and performs an image synthesis process of outputting an image generated by synthesizing the two images.

In step S1705, the luminance acquisition unit 109 acquires a representative value of luminance as luminance information and stores it in the buffer for the preliminary captured image synthesized in step S1704, as in step S205.

In step S1706, the amplification factor determination circuit 110 reads the average value, standard deviation, and maximum value of the luminance values acquired as the luminance information in step S1705 from the buffer 111, and, based on the information, reads out the luminance of the subject based on the information in step S206. Determine the amplification factor according to.

In step S1707, the gain control circuit 115 resets the gain in the imaging unit 100 based on the gain determined in step S1706, as in step S207.

In step S1708, the system control unit 117 determines the start of the main shooting. The system control unit 117 proceeds to step S1709 if the actual shooting is to be started, and returns to step S1703 if the actual shooting is not to be started, and executes the processing of steps S1703 to S1708 again.

In step S1709, the imaging control unit 114 controls the imaging unit 100 to start the main imaging operation.

In the next step S1710, the image processing circuit 108 performs an acquisition process of the two systems of images acquired in the main photographing operation in S1709, and an image synthesis process of outputting an image generated by combining the two systems of images. Then, the synthesized images obtained by the image synthesis processing by the image processing circuit 108 are sequentially stored as frames constituting a moving image.

In S1711, the brightness acquisition unit 109 acquires the representative value of the brightness of the captured image synthesized in S1710 as brightness information, as in S205.

In step S1712, the amplification factor determination circuit 110 reads the luminance information acquired and stored in step S1705 from the buffer 111, and determines an amplification factor according to the luminance of the subject based on the information. At this time, the amplification factor determination circuit 110 fixes the amplification factor of the low gain, and determines only the amplification factor of the high gain.

Then, in S1713, the amplification factor control circuit 115 resets the amplification factor of the High gain determined in S1712.

Then, in step S1715, the system control unit 117 determines whether to end shooting. If the imaging is not to be ended, the system control unit 117 returns the processing to S1709, and executes the processing from S1709 to S1715 again. On the other hand, the system control unit 117 ends the processing of the flowchart in FIG.

The above is the processing performed in the eighth embodiment. In the eighth embodiment, the amplification rate of the High gain is controlled during the main photographing. Thus, according to the eighth embodiment, when the luminance distribution of the subject changes with time, it is possible to reduce the change in the luminance of the composite image due to the change in the amplification factor of each of the High gain and the Low gain.

<Ninth embodiment>
In the eighth embodiment, the gains of the high gain and the low gain are determined in the preliminary photographing, and only the high gain is reset in the main photographing. In the ninth embodiment, an example is described in which a single image capturing operation, that is, a still image capturing operation is performed using the amplification factor determined in the preliminary image capturing operation on the assumption that the image capturing operation is a still image. Since the configuration diagram is the same as that of the first embodiment, illustration and description are omitted.

<Description of Photographing Operation in Imaging Device of Ninth Embodiment>
Hereinafter, the flow of the imaging operation in the imaging device of the ninth embodiment will be described with reference to the flowchart shown in FIG.

In S1801 of FIG. 18, the imaging control unit 114 sets the imaging conditions of the imaging device according to an instruction input from the outside via the operation unit 119, as in S201.

In S1802, the amplification factor control circuit 115 sets the amplification factor of the PGA 104 and the capacitance of the FD 103 as in S202.

In S1803, in response to a shooting instruction input from the outside via the operation unit 119, the imaging control unit 114 starts the preliminary shooting operation of the imaging unit 100, as in S1703.

In S1804, similarly to S1704, the image processing circuit 108 performs an acquisition process of the two images output in S1803 and an image synthesis process of outputting an image generated by synthesizing the two images.

In S1805, as in S1705, the brightness acquisition unit 109 acquires the representative value of the brightness of the preliminary captured image synthesized in S1804 as brightness information and stores the brightness information in the buffer 111.

In S1806, similarly to S1706, the amplification factor determination circuit 110 reads the luminance information from the buffer 111 in S1805, and determines an amplification factor according to the luminance of the subject based on the information.

In S1807, similarly to S1707, the gain control circuit 115 resets the gain in the imaging unit 100 based on the gain determined in S1806.

In S1808, the system control unit 117 determines whether to start the actual shooting of the still image. Then, the system control unit 117 advances the processing to S1809 when starting the main shooting, returns the processing to S1803 when not starting the main shooting, and executes the processing from S1803 to S1808 again.

In step S1809, the imaging control unit 114 causes the imaging unit 100 to start a main imaging operation. In step S1810, the image processing circuit 108 performs an image combining process, and stores the combined image.

The above is the processing performed in the ninth embodiment. In the ninth embodiment, the gains of the high gain and the low gain are determined by the preliminary photographing, and the still image can be photographed with the determined gains.

<Other embodiments>
In the above-described first to ninth embodiments, an example is described in which the analog processing circuit 106 has two systems of the PGA 104 and the AD conversion circuit 105. However, as another embodiment, as shown in FIG. A configuration having only the PGA 104 and the AD conversion circuit 105 may be employed. In FIG. 19, the same components as those in the first to eighth configurations described above are denoted by the same reference numerals as those described above, and description thereof is omitted.

In the configuration shown in FIG. 19, only one PGA 104 and AD conversion circuit 105 are used in a time-division manner. In the case of the configuration in FIG. 19, the system control unit 117 drives the FD 103, the PGA 104, and the AD conversion circuit 105 by setting one of the High gain and the Low gain as the first drive control. Next, as the second drive control, the system control unit 117 similarly sets the FD 103, the PGA 104, and the AD conversion circuit 105 with the setting of the high gain and the low gain that is not used in the first drive control. Drive. Thereby, also in the configuration of FIG. 19, a High gain image and a Low gain image are obtained.

The functions of the digital processing circuit 107 in each of the above-described embodiments may be realized only by a hardware configuration, or may be realized by a software configuration by a CPU or the like executing a program. Further, a part may be realized by a hardware configuration and the rest may be realized by a software configuration. The program for this software configuration is not limited to the case where it is prepared in advance, but may be obtained from a recording medium such as an external memory (not shown), or may be obtained via a network (not shown).

In the above-described embodiment, a digital camera is assumed as an example of the imaging device. However, various devices capable of capturing an image, such as a monitoring camera, an industrial camera, a vehicle-mounted camera, and a medical camera, such as a smartphone and a tablet terminal, are used. Is also applicable.

The program for realizing one or more functions in the signal processing according to the present invention can be supplied to a system or an apparatus via a network or a storage medium, and is read and executed by one or more processors of a computer of the system or the apparatus. It can be realized by doing. Further, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions.

The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be interpreted in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features.

The present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to make the scope of the present invention public.

This application claims priority based on Japanese Patent Application No. 2018-115919 filed on Jun. 19, 2018 and Japanese Patent Application No. 2019-077034 filed on Apr. 15, 2019, The entire contents of the description are incorporated herein.

Claims

An image pickup device that converts a signal voltage obtained by voltage-converting a signal charge obtained by photoelectrically converting an optical image of a subject into a digital signal after amplifying the signal voltage,
An amplifier circuit for amplifying the signal voltage by two or more amplification factors,
Weight determining means for determining a weight for each of the digitally converted signals after amplification based on two or more different amplification factors in the amplifier circuit,
Synthesizing means for synthesizing the digitally converted signal after the two or more amplifications using the weights,
An imaging device comprising:
2. The imaging apparatus according to claim 1, further comprising an amplification factor determining unit that determines an amplification factor corresponding to at least one of the two or more amplification factors in accordance with luminance information of the subject. 3.
The imaging device according to claim 2, wherein the amplification circuit includes two or more amplification circuits.
The signal voltage is amplified by one amplifier circuit of the two or more amplifier circuits, from the digitally converted signal, comprising a luminance acquisition unit that acquires the luminance information,
The amplification factor determination unit determines an amplification factor according to the luminance information for an amplification circuit different from an amplification circuit that amplifies the signal voltage in the signal used for acquiring the luminance information. Item 4. The imaging device according to Item 3.
The luminance obtaining means obtains the luminance information from the digitally converted signal in which the signal voltage is amplified by an amplifier circuit having a relatively low amplification factor set among the two or more amplifier circuits,
5. The amplification factor determining unit according to claim 4, wherein an amplification factor corresponding to the luminance information is determined for an amplification circuit having a relatively high amplification factor set among the two or more amplification circuits. An imaging device according to item 1.
The luminance obtaining means may calculate the average value and the standard deviation of the luminance of the image based on the digitally converted signal, or at least one of the skewness and the kurtosis of the luminance of the image based on the digitally converted signal, using the luminance information. The imaging device according to claim 4, wherein the image is acquired as:
The luminance obtaining means obtains an average value and a standard deviation of luminance of an image based on the digitally converted signal, as the luminance information,
The imaging apparatus according to claim 4, wherein the amplification factor determination unit determines an amplification factor of the amplification circuit based on the luminance information and a setting mode for setting a priority luminance range.
The luminance acquisition unit acquires at least one of skewness and kurtosis of luminance of an image based on the digitally converted signal as the luminance information,
The imaging apparatus according to claim 4, wherein the amplification factor determination unit determines the amplification factor of the amplification circuit based on the luminance information and a table describing the amplification factor.
Having at least one or more charge memories for storing the signal charges,
The imaging apparatus according to claim 4, wherein the luminance acquisition unit acquires the luminance information based on a signal voltage obtained by voltage-converting the signal charge output from the charge memory.
Having storage means for storing an image based on the digitally converted signal in an image memory,
The imaging apparatus according to claim 4, wherein the brightness obtaining unit obtains a brightness histogram from the stored image.
The amplification factor determining means determines a portion having a low frequency in the histogram by threshold processing, and determines the amplification factor based on a result of the determination and an evaluation value of image quality. Item 11. The imaging device according to Item 10.
When a plurality of candidate amplification factors are obtained based on a result of the determination by the threshold processing, the amplification factor determination unit determines the image quality evaluation value from among the plurality of candidate amplification factors. The imaging apparatus according to claim 11, wherein the amplification factor to be determined is selected.
The imaging apparatus according to claim 11, wherein the amplification factor determination unit acquires S / N as the evaluation value of the image quality.
The amplification factor determining means determines a less frequent part in the histogram by threshold processing, divides the histogram based on luminance based on the result of the determination, and determines the importance for each of the divided areas. The imaging apparatus according to claim 10, wherein the amplification factor is determined based on the importance.
The imaging apparatus according to claim 14, wherein the amplification factor determination unit acquires a maximum value of the frequency of the histogram or a total number of frequencies as the importance.
2. The synthesizing unit according to claim 1, wherein the synthesizing unit synthesizes using two or more amplification factors and a weight determined according to an amount of readout noise generated when the photoelectrically converted signal is digitally converted. The imaging device according to any one of claims 15 to 15.
A signal voltage obtained by voltage conversion of a signal charge obtained by photoelectrically converting an optical image of a subject, and a control method of an imaging apparatus that performs digital conversion after amplification.
Amplifying the signal voltage by two or more amplification factors;
A weight determination step of determining a weight for each of the digitally converted signals after amplification based on the two or more amplification factors,
A synthesizing step of synthesizing the digitally converted signal after the two or more amplifications using the weights,
A method for controlling an imaging device, comprising:
A program that causes a computer to execute a control method of an imaging device that performs digital conversion after amplifying a signal voltage obtained by voltage-converting a signal charge obtained by photoelectrically converting an optical image of a subject,
Amplifying the signal voltage by two or more amplification factors,
A weight determination step of determining a weight for each of the digitally converted signals after amplification based on the two or more amplification factors,
A synthesizing step of synthesizing the digitally converted signal after the two or more amplifications using the weights,
A program that causes a computer to execute.