WO2017212870A1 - Signal processing device, signal processing method, and imaging device - Google Patents

Abstract

This signal processing device is provided with: a luminance detection unit for detecting, on the basis of an imaging signal, the luminance of each pixel for each of a plurality of detection frames set by dividing an imaging screen in a vertical scanning direction; a period detection unit for detecting the frame periodicity of a flicker component in each of the detection frames by using detected luminance values obtained by the luminance detection unit for a plurality of frames; and a determination unit for determining whether a flicker is present on the basis of the result of frame periodicity detection by the period detection unit.

Description

Signal processing apparatus, signal processing method, and imaging apparatus

The present disclosure relates to a signal processing device, a signal processing method, and an imaging device.

In an image pickup apparatus including an image pickup device called an image sensor, when shooting is performed in a light source environment in which brightness varies periodically, the brightness of the periodic brightness in the vertical scanning direction is displayed on the display screen based on the image pickup signal. Variations appear. This periodic variation in luminance is visually recognized as a so-called horizontal stripe (flicker stripe) on the display screen, which causes a deterioration in the image quality of the captured image.

In order to suppress image quality deterioration due to this flicker (flicker stripe), it is necessary to detect the occurrence of flicker. Conventionally, occurrence of flicker is detected by detecting the period of flicker fringes in one screen (within one imaging frame) caused by the readout time of the imaging signal in the imaging device and the blinking cycle of the light source whose brightness varies periodically. It was made to detect (for example, refer patent document 1).

JP 2003-189129 A

As described above, the flicker detection according to the prior art described in Patent Document 1 is to detect the period of flicker fringes within one screen (within one imaging frame). Therefore, when flicker fringes in one screen do not occur, the conventional technology described in Patent Document 1 cannot detect the occurrence of flicker.

Therefore, an object of the present disclosure is to provide a signal processing device, a signal processing method, and an imaging device that can detect the occurrence of flicker even when flicker fringes in one screen do not occur. .

In order to achieve the above object, a signal processing device of the present disclosure is provided.
For a plurality of detection frames set by dividing the imaging screen in the vertical scanning direction, a luminance detection unit that detects the luminance of each pixel for each detection frame based on the imaging signal,
A period detection unit that detects the frame periodicity of each detection frame for flicker components using the luminance detection values obtained by the luminance detection unit for a plurality of frames; and
A determination unit that determines the presence or absence of flicker based on the detection result of the frame periodicity by the cycle detection unit,
Is provided.

In order to achieve the above object, a signal processing method of the present disclosure includes:
For a plurality of detection frames set by dividing the imaging screen in the vertical scanning direction, the luminance of each pixel is detected for each detection frame based on the imaging signal,
Detect the frame periodicity of each detection frame for flicker components using the luminance detection value for multiple frames,
The presence or absence of flicker is determined based on the detection result of the frame periodicity.

In order to achieve the above object, an imaging apparatus of the present disclosure is provided.
An imaging pixel unit, and
A signal processing circuit for processing an imaging signal output from the imaging pixel unit;
With
The signal processing circuit
For a plurality of detection frames set by dividing the imaging screen in the vertical scanning direction, a luminance detection unit that detects the luminance of each pixel for each detection frame based on the imaging signal,
A period detection unit that detects the frame periodicity of each detection frame for flicker components using the luminance detection values obtained by the luminance detection unit for a plurality of frames; and
A determination unit that determines the presence or absence of flicker based on the detection result of the frame periodicity by the cycle detection unit.

In the signal processing device, the signal processing method, or the imaging device having the above-described configuration, flicker is obtained by using the luminance detection values obtained by detecting the luminance of each pixel for each detection frame for a plurality of detection frames. The frame periodicity of each detection frame can be detected for the component. The presence / absence of a flicker component can be determined from the detection result of the frame periodicity.

According to the present disclosure, by detecting the frame periodicity of each detection frame for flicker components and determining the presence or absence of flicker components from the detection result, even if flicker fringes in one screen do not occur, flicker Can be detected.

It should be noted that the effect described here is not necessarily limited, and may be any effect described in the present specification. Moreover, the effect described in this specification is an illustration to the last, Comprising: It is not limited to this, There may be an additional effect.

FIG. 1A is a conceptual diagram of an operation of the rolling shutter system, and FIG. 1B is an explanatory diagram regarding a relationship between a pixel drive frame rate, a light source frequency, and a flicker cycle. FIG. 2 is an exploded perspective view schematically illustrating the configuration of the image pickup device having a laminated structure. FIG. 3 is a circuit diagram illustrating an example of the configuration of the imaging pixel unit. FIG. 4A is a conceptual diagram of the operation of the pseudo global shutter system, and FIG. 4B is a conceptual diagram of the operation of the global shutter system. FIG. 5 is a block diagram illustrating an example of the configuration of the signal processing device according to the first embodiment of the present disclosure. FIG. 6 is a block diagram showing an example of the configuration of an N frame cycle flicker detector that constitutes the cycle detection unit. FIG. 7 is an explanatory diagram of the normalization process. FIG. 8 is a flowchart showing a flow of N frame periodicity detection processing in the N frame high frequency detection processing unit. FIG. 9 is a flowchart showing the flow of the screen ratio determination process in the screen ratio determination unit. FIG. 10 is a diagram illustrating a processing example of variation removal when M = 8 and variation removal determination threshold = 5. FIG. 11 is a block diagram illustrating an exemplary configuration of a signal processing device according to the second embodiment of the present disclosure. FIG. 12 is a block diagram illustrating an example of a system configuration of the imaging apparatus according to the present disclosure.

Hereinafter, modes for carrying out the technology of the present disclosure (hereinafter referred to as “embodiments”) will be described in detail with reference to the drawings. The technology of the present disclosure is not limited to the embodiments, and various numerical values in the embodiments are examples. In the following description, the same reference numerals are used for the same elements or elements having the same function, and redundant description is omitted. The description will be given in the following order.
1. 1. General description of signal processing apparatus, signal processing method, and imaging apparatus of the present disclosure About Flicker 3. Regarding an image sensor having a laminated structure 4. Pseudo global shutter system First embodiment 5-1. System configuration 5-2. N frame period flicker detector 5-2-1. Normalization processing 5-2-2. N frame high frequency detection 5-2-3. N-frame periodicity detection 5-2-4. Screen ratio determination 5-2-5. Unevenness (removal of chattering)
6). Second embodiment 7. 7. Imaging device of the present disclosure Modification 9 Composition of the present disclosure

<Description on Signal Processing Device, Signal Processing Method, and Imaging Device of the Present Disclosure>
In the signal processing device, the signal processing method, and the imaging device of the present disclosure, with respect to the luminance detection value, the luminance integrated value obtained by integrating the luminance information of each pixel for each detection frame, or the luminance integrated value as one pixel. It can be set as the structure which is the luminance average value averaged by the unit.

In the signal processing device, the signal processing method, and the imaging device of the present disclosure including the preferred configuration described above, a plurality of frames are included in the past N frames including the frame of interest (N is the driving frequency of the imaging device and the blinking of the light source). Value determined by frequency). Then, the period detection unit obtains the maximum value and the minimum value in the luminance detection values of N frames, performs normalization processing based on the difference between the maximum value and the minimum value, and evaluates the frame periodicity of each detection frame for flicker components. It can be. At this time, when the difference between the maximum value and the minimum value in the luminance detection values of N frames is less than a predetermined minimum amplitude threshold value, it is preferable to set the evaluation value of the frame periodicity to 0.

Furthermore, in the signal processing device, the signal processing method, and the imaging device of the present disclosure including the preferable configuration described above, the period detection unit may include a high-frequency detection processing unit and a periodicity detection processing unit. it can. The high frequency detection processing unit detects whether or not a high frequency component exists in the N frame. The periodicity detection processing unit detects the periodicity of the flicker component in the N frame in units of frames based on the difference in the luminance detection value between the N frames.

Furthermore, in the signal processing device, the signal processing method, and the imaging device of the present disclosure including the preferred configuration described above, the period detection unit is based on the detection results of the high-frequency detection processing unit and the periodicity detection processing unit. When there are high frequency components in N frames and the flicker components have periodicity in units of frames, the flicker components in the N frame cycle can be detected.

Furthermore, in the signal processing device, the signal processing method, and the imaging device of the present disclosure including the preferred configuration described above, the luminance detection value is subjected to discrete Fourier transform for the high frequency detection processing unit, and imaging is performed from the obtained frequency information. The presence or absence of a flicker component can be determined based on whether or not a high-frequency component exists in the signal. The periodicity detection processing unit is configured to determine that the flicker component in the N frame has periodicity in units of frames when the difference in luminance detection value between the N frames is smaller than a predetermined periodicity determination threshold. can do.

Furthermore, in the signal processing device, the signal processing method, and the imaging device of the present disclosure including the preferable configuration described above, the determination unit is based on the detection result of the frame periodicity of each detection frame by the cycle detection unit. It can be configured to determine the presence or absence of flicker on the entire screen. Further, the determination unit counts the detection result of the frame periodicity of each detection frame by the cycle detection unit, calculates the flicker detection ratio by dividing the total count by the total number of detection frames, and sets the flicker detection ratio to a predetermined value. It can be configured to determine the presence or absence of flicker by comparing with the area ratio determination threshold.

Furthermore, in the signal processing device, the signal processing method, and the imaging device of the present disclosure including the preferred configuration described above, the determination result of the presence / absence of flicker in each frame by the determination unit is accumulated, and variation in the determination result is eliminated. And it can be set as the structure provided with the variation removal part made into a final determination result. Then, the variation removal unit accumulates the determination results for the past M frames by the determination unit, and when the number of flicker determinations is equal to or greater than a predetermined variation removal determination threshold, the final determination result is configured to have flicker. it can.

Furthermore, in the signal processing device, the signal processing method, and the imaging device of the present disclosure including the preferred configuration described above, a long-time accumulation imaging signal obtained by accumulating signal charges for a long time for one pixel, A short-time accumulation imaging signal obtained by accumulating signal charges for a short time can be input to the luminance detection unit. At this time, the luminance detection unit can be configured to detect the luminance of each pixel for each detection frame for each of the long-time accumulation imaging signal and the short-time accumulation imaging signal and to give the luminance detection value to the period detection unit. .

Alternatively, in the imaging device of the present disclosure, the imaging pixel unit may be configured by a CMOS sensor. In addition, a pixel substrate on which an imaging pixel unit is formed and a circuit substrate on which a signal processing circuit is formed are stacked, and an imaging signal output from the imaging pixel unit is provided between the pixel substrate and the circuit substrate. A structure in which a substrate on which a memory portion for temporarily storing is formed is provided.

Further, in the imaging device of the present disclosure including the above-described preferable configuration, the imaging pixel unit to the memory unit can be read out with a readout time shorter than the readout time in the rolling shutter system in a light source environment in which the brightness periodically varies. The imaging signal can be read out. At this time, it is preferable to read the imaging signal from the imaging pixel unit to the memory unit with a readout time shorter than the blinking cycle of the light source. Further, from the imaging pixel unit, a signal processing circuit for a long-time accumulation imaging signal obtained by accumulating signal charges for a long time and a short-time accumulation imaging signal obtained by accumulating signal charges for a short time for one pixel. It can be set as the structure supplied to.

<About Flicker>
First, flicker that contributes to image quality degradation of a photographed image will be described. In an XY address type image pickup device represented by a CMOS (Complementary Metal Oxide Semiconductor) image sensor, as a shutter system, generally, from one line (horizontal line) to several lines are made into one block, and each block is divided. A rolling shutter system that acquires an imaging signal is employed. A conceptual diagram of the operation of the rolling shutter system is shown in FIG. 1A. In FIG. 1A, the solid line indicates the exposure start time, the broken line indicates the readout start time of the imaging signal, and the shaded area indicates the exposure time.

In a rolling shutter type imaging device, when shooting is performed in a light source environment in which the brightness varies periodically, luminance is periodically displayed in the vertical scanning direction (pixel row arrangement direction) on the display screen based on the imaging signal. Flicker is generated in which the fluctuations are visually recognized as horizontal stripes (flicker stripes). For example, when a light source such as a fluorescent lamp is turned on with a commercial power supply having a frequency of 50 Hz, the brightness periodically fluctuates (flickers) at a frequency twice the power supply frequency, that is, 100 Hz.

Under the above light source environment, for example, when the cycle of the shooting frame is 1/30 second (30 fps), the brightness phase of the light source is aligned in three frame cycles, so the luminance of each frame varies every three frame cycles. In addition, in the rolling shutter type imaging device, as shown in FIG. 1, since the exposure start time (signal charge accumulation start time) is different for each line, the luminance fluctuation periodically in the vertical scanning direction on the display screen. Appear as flicker stripes.

FIG. 1B shows the relationship between the frame rate, the light source frequency, and the flicker cycle. Specifically, under a light source environment of 50 Hz, when the frame rate is 15 fps, 30 fps, and 60 fps, the flicker period is 3 frame periods. Further, under a light source environment of 60 Hz, when the frame rate is 12.5 fps, 25 fps, and 50 fps, the flicker cycle becomes 5 frame cycles.

As described above, in a rolling shutter type imaging device, flicker stripes (hereinafter referred to as “in-plane line flicker”) in units of horizontal lines within one screen (within one imaging frame) depending on the readout time of the imaging signal and the blinking cycle of the light source. May occur). Therefore, the occurrence of flicker can be detected by detecting the period of in-plane line flicker. However, this flicker detection technique cannot detect the occurrence of flicker in the case of an image sensor that does not generate in-plane line flicker.

<Regarding an image sensor with a laminated structure>
As an image sensor that does not generate in-plane line flicker, an image sensor having a laminated structure can be exemplified. However, the present invention is not limited to the image pickup device having a laminated structure. FIG. 2 shows an outline of the configuration of the image pickup element having a laminated structure.

The image pickup device 10 having a laminated structure according to this example has a configuration in which at least three semiconductor substrates (semiconductor chips) 11, 12, and 13 are laminated. The first-layer semiconductor substrate 11 is a pixel substrate on which the imaging pixel unit 21 is formed. Specifically, on the semiconductor substrate 11, an imaging pixel unit 21 is configured in which pixels including photoelectric conversion elements and various pixel transistors are arranged in a matrix. Details of the configuration of the imaging pixel unit 21 will be described later.

The second-layer semiconductor substrate 12 is a memory substrate on which a memory unit 22 that temporarily stores an analog imaging signal output from the imaging pixel unit 21 of the first-layer semiconductor substrate 11 is formed. For example, a high-speed, low-power, large-capacity memory is preferably used as the memory unit 22 that temporarily stores the imaging signal.

The third-layer semiconductor substrate 13 is a circuit board on which signal processing units such as an A (analog) / D (digital) conversion unit 23 and a logic circuit 24 are formed. The A / D conversion unit 23 converts an analog imaging signal read from the memory unit 22 of the second-layer semiconductor substrate 12 into a digital imaging signal. The logic circuit 24 performs various signal processing on the digital imaging signal.

The A / D converter 23 can be a known A / D converter. Specifically, as the A / D conversion unit 23, a single slope type A / D conversion unit, a successive approximation type A / D conversion unit, or a delta-sigma modulation type (ΔΣ modulation type) A / D conversion unit is exemplified. can do. However, the A / D converter 23 is not limited to these.

As described above, the image pickup device 10 having a stacked structure in which at least three semiconductor substrates 11, 12, and 13 are stacked has a size (area) enough to form the image pickup pixel portion 21 as the first semiconductor substrate 11. It's fine. Accordingly, the size (area) of the first-layer semiconductor substrate 11 and, consequently, the size of the entire chip can be reduced. Furthermore, a process suitable for pixel creation can be applied to the first semiconductor substrate 11, and a process suitable for circuit creation can be applied to the second and third semiconductor substrates 12 and 13, respectively. There is also an advantage that the process can be optimized in the manufacture of 10.

In the imaging device 10 described above, an imaging signal read from the imaging pixel unit 21 is converted into digital data by the A / D conversion unit 23, and then controlled by a memory controller (not shown). Data is written to the memory unit 22 at high speed. Then, the digital data written in the memory unit 22 is read out to the signal processing unit such as the logic circuit 24 at a low speed under the control of the memory controller. As a result, an image pickup signal is instantaneously read from the image pickup pixel unit 21 to the memory unit 22 and is slowly read out from the memory unit 22 to perform signal processing, thereby obtaining a high-quality image with little distortion.

Here, the configuration of the imaging pixel unit 21 will be described with a specific example. An example of the configuration of the imaging pixel unit 21 is shown in FIG. The imaging pixel unit 21 according to this example is configured as a CMOS sensor in which pixels 30 including photoelectric conversion elements and various pixel transistors are arranged in a two-dimensional matrix in the row direction and the column direction.

The pixel 30 includes, for example, a photodiode 31 as a photoelectric conversion element, and includes, for example, four transistors, which are a transfer transistor 32, a reset transistor 33, an amplification transistor 34, and a selection transistor 35, as pixel transistors.

Here, as the four pixel transistors 32 to 35, for example, N-channel transistors are used. However, the conductivity type combinations of the transfer transistor 32, the reset transistor 33, the amplification transistor 34, and the selection transistor 35 illustrated here are merely examples, and are not limited to these combinations. That is, a combination using a P-channel transistor can be used as necessary.

A transfer signal TRG, a reset signal RST, and a selection signal SEL that are drive signals for driving the pixel 30 are appropriately supplied to the pixel 30 from a row selection unit (not shown). That is, the transfer signal TRG is applied to the gate electrode of the transfer transistor 32, the reset signal RST is applied to the gate electrode of the reset transistor 33, and the selection signal SEL is applied to the gate electrode of the selection transistor 35.

The photodiode 31 has an anode electrode connected to a low-potential side power source (for example, ground), and photoelectrically converts received light (incident light) into photocharge (here, photoelectrons) having a charge amount corresponding to the amount of light. Then, the photocharge is accumulated. The cathode electrode of the photodiode 31 is electrically connected to the gate electrode of the amplification transistor 34 via the transfer transistor 32. A node electrically connected to the gate electrode of the amplification transistor 34 is a floating diffusion (floating diffusion region) 36.

The transfer transistor 32 is connected between the cathode electrode of the photodiode 31 and the floating diffusion 36. A high level (for example, V _DD level) transfer signal TRG is applied to the gate electrode of the transfer transistor 32. In response to the transfer signal TRG, the transfer transistor 32 becomes conductive, and the photoelectric charge photoelectrically converted by the photodiode 31 is transferred to the floating diffusion 36.

The reset transistor 33 has a drain electrode connected to the pixel power supply V _DD and a source electrode connected to the floating diffusion 36. A high level reset signal RST is applied to the gate electrode of the reset transistor 33. In response to the reset signal RST, the reset transistor 33 becomes conductive and resets the floating diffusion 36.

The amplification transistor 34 has a gate electrode connected to the floating diffusion 36 and a drain electrode connected to the pixel power source V _DD . The amplification transistor 34 outputs the potential of the floating diffusion 36 after being reset by the reset transistor 33 as a reset signal (reset level). Further, the amplification transistor 34 outputs the potential of the floating diffusion 36 after the signal charge is transferred by the transfer transistor 32 as a light accumulation signal (signal level).

The selection transistor 35 has, for example, a drain electrode connected to the source electrode of the amplification transistor 34 and a source electrode connected to the signal line 25. A high level selection signal SEL is supplied to the gate electrode of the selection transistor 35. In response to the selection signal SEL, the selection transistor 35 is turned on, and the pixel 30 is selected and a signal output from the amplification transistor 34 is read out to the signal line 25.

Here, the circuit configuration in which the selection transistor 35 is connected between the source electrode of the amplification transistor 34 and the signal line 25 is illustrated, but the selection transistor 35 is connected between the pixel power supply V _DD and the drain electrode of the amplification transistor 34. It is also possible to adopt a circuit configuration.

Further, the pixel 30 is not limited to the pixel configuration including the four transistors 32 to 35 described above. For example, a pixel configuration including three transistors in which the function of the selection transistor 35 is provided in the amplification transistor 34, or a pixel configuration in which a transistor after the floating diffusion 36 is shared between a plurality of photoelectric conversion elements (between pixels). There is no limitation on the configuration of the pixel circuit.

<Pseudo global shutter system>
In the image pickup device 10 having the above-described stacked structure, high-speed reading of the image pickup signal from the image pickup pixel unit 21 to the memory unit 22 is performed with a read time shorter than the read time in the above-described rolling shutter method, for example, brightness is periodically changed. In a changing light source environment, the time is shorter than the blinking cycle of the light source. Then, by performing the rolling shutter at high speed, the characteristics of the global shutter are approximated. Accordingly, in this specification, a readout time shorter than the readout time in the rolling shutter system, that is, a shutter system that reads out an imaging signal at a higher speed than the rolling shutter system is referred to as a pseudo global shutter system.

A conceptual diagram of the operation of the pseudo global shutter system is shown in FIG. 4A. In the case of the image pickup device 10 that reads out an image pickup signal from the image pickup pixel unit 21 to the memory unit 22 at a high speed by this pseudo global shutter method, the in-plane line flicker does not occur and flickers blinking in units of one screen (one image pickup frame). Hereinafter, “surface flicker” may be described). In the case of the pseudo global shutter system in which the surface flicker occurs, it is difficult to detect the flicker in the flicker detection system for the rolling shutter system in which the in-plane line flicker occurs.

Incidentally, FIG. 4B shows a conceptual diagram of the operation of the global shutter system generally employed in a charge transfer type imaging device represented by a CCD (Charge Coupled Device) image sensor. Even in the case of this global shutter system, in-plane line flicker does not occur, and surface flicker that blinks in units of one screen occurs. Specifically, in the global shutter system, a flicker image blinking at a high frequency is obtained by a combination of the driving frequency of the image sensor and the blinking frequency of the light source.

As an example, when the driving frequency of the image sensor is 60 Hz and the blinking frequency of the light source is 50 Hz, a flicker image having a three-frame cycle is obtained. Using this property, in the global shutter method, the imaging screen is divided in the vertical scanning direction to set a plurality of detection frames, and using the luminance detection value of each detection frame, the number and interval of waves are calculated as theoretical calculation values. By comparison, flicker was detected.

In the pseudo global shutter system, as in the global shutter system, a surface flicker that blinks in units of one screen occurs, but a certain luminance difference occurs on the screen. Therefore, since regular blinking does not occur when viewed on the entire imaging screen, it is possible to accurately detect flicker periodicity generated in the pseudo global shutter method using a detection method targeting the global shutter method. Have difficulty.

Therefore, in the present disclosure, in an XY address type image pickup device typified by a CMOS image sensor, for example, an image pickup device having a laminated structure, in-plane line flicker does not occur and flicker occurs even when surface flicker occurs. We propose a technique that can accurately detect the periodicity of the. However, the technology of the present disclosure is not limited to application to an image pickup device having a laminated structure, and does not exclude application to a charge transfer type image pickup device typified by a CCD image sensor. Hereinafter, specific embodiments of the technology of the present disclosure will be described.

<First Embodiment>
FIG. 5 is a block diagram illustrating an example of a configuration of the signal processing device (signal processing circuit) according to the first embodiment of the present disclosure. In the present embodiment, the imaging screen 40 corresponding to the imaging region (pixel region) of the imaging pixel unit 21 is divided into a plurality of detection frames 41 ₁ to 41 by dividing the imaging screen 40 in the vertical scanning direction (direction orthogonal to the pixel row / horizontal line). ₆ is set. The number of detection frames to be set is arbitrary, but here it is set to 6 for simplification of the drawing. Actually, it is preferable to set a large number of detection frames such as 16 or 32.

[System configuration]
As illustrated in FIG. 5, the signal processing device 50 according to the present embodiment includes a luminance detection unit 51, a period detection unit 52, a screen ratio determination unit 53, and a variation removal unit 54. The flicker frame period is detected by using the luminance detection value of each detection frame for a plurality of frames. Then, based on the detection results obtained from the detection frames 41 ₁ to 41 _6, it is determined whether or not flicker has occurred in the entire screen. Preferably, the determination results are accumulated in units of frames, and variations in the determination results are removed to obtain a final determination result.

The luminance detection unit 51 includes six luminance detectors 51 ₁ to 51 ₆ corresponding to, for example, six detection frames 41 ₁ to 41 ₆ of the imaging screen 40, and the six luminance detectors 51 ₁ to 51 _6. , The luminance of each pixel 30 is detected for each detection frame based on the imaging signal. More specifically, the luminance detection unit 51 detects the luminance of each pixel 30 so that the change of the luminance detection value for each of the detection frames 41 ₁ to 41 ₆ is a period change equivalent to the global shutter method. The luminance detection value may be a luminance integrated value obtained by integrating the luminance information of each pixel 30 for each of the detection frames 41 ₁ to 41 ₆ , or a luminance average value obtained by averaging the luminance integrated value for each pixel. May be.

The period detection unit 52 includes six N frame period flicker detectors 52 ₁ to 52 ₆ corresponding to the six luminance detectors 51 ₁ to 51 _6, and each detection frame 41 _{1 is} used for flicker components using a plurality of frames. A periodicity (or high frequency) detection process in units of ˜41 ₆ frames is performed. By performing this detection process, the occurrence of flicker can be detected. Here, the plurality of frames are the past N frames including the frame of interest. N is a value determined by the drive frequency of the image sensor and the blinking frequency of the light source. Details of the period detector 52 will be described later.

The screen ratio determination unit 53 determines the presence or absence of flicker in the entire imaging screen 40 based on the detection result of periodicity (or high frequency) in units of frames by the period detection unit 52. The variation removal unit 54 accumulates the determination result of the presence / absence of the flicker component in each frame by the screen ratio determination unit 53, and removes the variation of the determination result to obtain the final determination result. Details of the screen ratio determination unit 53 and the variation removal unit 54 will be described later.

[N frame period flicker detector]
An example of the configuration of the N frame cycle flicker detectors 52 ₁ to 52 ₆ constituting the cycle detector 52 is shown in FIG. When a subject is imaged with a global shutter imaging device in a light source environment where the brightness varies periodically, the captured image (output image) has a high-frequency luminance change with a periodicity in units of frames. .

The N frame period flicker detectors 52 ₁ to 52 ₆ perform high frequency detection and periodicity detection in units of frames by utilizing the characteristics of having high frequency and periodicity in units of frames, and detecting flicker components of N frame periods. Determine presence or absence. Specifically, the N frame period flicker detectors 52 ₁ to 52 ₆ include a normalization processing unit 511, an N frame high frequency detection processing unit 512, an N frame periodicity detection processing unit 513, and a logical product unit 514. ing. Here, the case where the luminance average value is used as the luminance detection value will be described as an example, but the same applies to the case where the luminance integral value is used.

(Normalization processing)
As illustrated in FIG. 7, the normalization processing unit 511 obtains the maximum value Max and the minimum value Min in the luminance detection values of the past N frames including the frame of interest. Here, the case where N = 3, that is, the case where the drive frequency of the image sensor is 60 Hz and the blinking frequency of the light source is 50 Hz is illustrated. After obtaining the maximum value Max and the minimum value Min, the normalization processing unit 511 performs normalization processing based on the difference between the maximum value Max and the minimum value Min (Max−Min), and each detection frame 41 ₁ to 41 ₆ for the flicker component. Evaluate the frame periodicity. By this normalization processing, even when the flicker due to flicker is very small, for example, when the exposure time is long, the periodicity of flicker can be detected with high accuracy.

However, when the difference (Max−Min) between the maximum value Max and the minimum value Min in the luminance detection value of N frames is less than a predetermined minimum amplitude threshold, the normalization processing unit 511 determines that the luminance is not blinking due to flicker. The evaluation value of frame periodicity is set to 0. Thereby, since flickering of luminance that is not caused by flicker can be excluded, the periodicity of flicker can be detected with higher accuracy.

An evaluation formula of the frame of interest i in the normalization process by the normalization processing unit 511 is shown in the following formula (1).

In Expression (1), X represents the luminance detection value of the frame of interest i, and Max represents the maximum luminance detection value in the past N frames. Min represents the minimum luminance detection value in the past N frames, and a represents a reference amplitude coefficient.

(N frame high frequency detection)
The N-frame high-frequency detection processing unit 512 performs N-point discrete Fourier transform (DFT) on the luminance average value (or luminance integral value) acquired from the luminance detection unit 51, and generates N frames from the obtained frequency information. The presence / absence of a flicker component is determined depending on whether a high frequency component is present therein.

The general formula of the discrete Fourier transform is shown in the following formula (2).

When Expression (2) is expanded for m = 1 and normalized by the number of frames N, the square of the absolute value is taken to obtain the following Expression (3).

The N frame high frequency detection processing unit 512 sets the result of the expression (3) as an evaluation value as to whether or not a high frequency component exists in the N frame. Then, when the result (evaluation value) of the expression (3) is larger than a predetermined high frequency determination threshold, the N frame high frequency detection processing unit 512 has a high frequency component in the N frame, and the flicker component within the detection frame Judge that there is.

(N-frame periodicity detection)
The N-frame high-frequency detection processing unit 512 can detect high-frequency blinking by discrete Fourier transform, but does not consider the periodicity of flicker components in units of frames. Therefore, there is a concern that a change in the subject is erroneously detected as flicker. Therefore, an N frame periodicity detection processing unit 513 is provided. The N frame periodicity detection processing unit 513 determines that the flicker component in the N frame has periodicity in units of frames when the difference in the luminance detection value between the N frames is smaller than a predetermined periodicity determination threshold.

FIG. 8 shows a flow of N frame periodicity detection processing in the N frame periodicity detection processing unit 513. The N frame periodicity detection processing unit 513 detects a difference in luminance detection value between N frames (step S11), and then determines whether or not the difference between N frames is smaller than a predetermined periodicity determination threshold. (Step S12).

Then, when the difference between N frames is smaller than the predetermined periodicity determination threshold value (S12 YES), the N frame periodicity detection processing unit 513 sets the periodicity determination result to true (true) (step S13). . Further, when the difference between N frames is equal to or greater than a predetermined periodicity determination threshold (NO in S12), the N-frame high-frequency detection processing unit 512 sets the periodicity determination result to no periodicity (false) (step S14). .

The logical product unit 514 calculates the logical product of the detection results of the N frame high frequency detection processing unit 512 and the N frame periodicity detection processing unit 513. Thereby, the period detection unit 52 has a high frequency component in the N frame and flicker components in units of frames based on the detection results of the N frame high frequency detection processing unit 512 and the N frame periodicity detection processing unit 513. When it has periodicity, the flicker component of the N frame period is detected.

(Screen ratio judgment)
The screen ratio determination unit 53 determines the presence or absence of flicker in the entire imaging screen 40 based on the detection result of the frame periodicity of the detection frames 41 ₁ to 41 ₆ by the cycle detection unit 52. More specifically, the screen ratio determination unit 53 counts the detection results of the frame periodicity of the detection frames 41 ₁ to 41 ₆ by the cycle detection unit 52, divides the total count by the total number of detection frames, and flickers. The detection ratio is calculated, and the presence or absence of flicker is determined by comparing the flicker detection ratio with a predetermined area ratio determination threshold.

The flow of the screen ratio determination process in the screen ratio determination unit 53 is shown in FIG. The screen ratio determination unit 53 first calculates the number of flicker detection frames (step S21). The number of flicker detection frames can be calculated as (number of flicker detection frames) = Σ (detection results of the detection frames 41 ₁ to 41 ₆ ). Next, the screen ratio determination unit 53 calculates a flicker detection ratio (step S22). The flicker detection ratio can be calculated as (flicker detection ratio) = (number of flicker detection frames) / (total number of detection frames).

Next, the screen ratio determination unit 53 determines whether or not the flicker detection ratio exceeds a predetermined area ratio determination threshold (step S23), and if flicker detection ratio> area ratio determination threshold (YES in S23). The flicker detection result is assumed to be flicker present (true) (step S24). Further, when the flicker detection ratio ≦ the area ratio determination threshold value (NO in S23), the screen ratio determination unit 53 sets the flicker detection result to “no flicker” (false) (step S25).

[Removal of variation]
The variation removal unit 54 performs variation removal (chattering removal) of the determination result of the presence or absence of the flicker component by the screen ratio determination unit 53. Specifically, the variation removal unit 54 accumulates the determination results of the screen ratio determination unit 53 for the past M frames (M can be arbitrarily set), and the number of determinations with flicker is equal to or greater than a predetermined variation removal determination threshold. In this case, the final judgment result is flicker. As a result, it is possible to prevent erroneous detection of flicker due to changes in the subject.

FIG. 10 shows an example of variation removal processing when M = 8 and variation removal determination threshold = 5. In case 1, the number of flicker determinations is 4 when the flicker frequency of the light source is 50 Hz, the number of flicker determinations is 0 when the light source flicker frequency is 60 Hz, and the number of determinations without flicker is 4 It is an example. In the case 1, since the number of flicker determinations when the blinking frequency of the light source is 50 Hz and 60 Hz is less than the variation removal determination threshold, the final determination result is set to no flicker detection.

In case 2, the number of flicker determinations is 5 when the light source flicker frequency is 50 Hz, the flicker determination number is 0 when the light source flicker frequency is 60 Hz, and the number of flicker determinations is 3 It is an example. In the case 2 case, since the number of flicker determinations when the flicker frequency of the light source is 50 Hz is equal to or greater than the variation removal determination threshold, the final determination result is that flicker is detected when the flicker frequency of the light source is 50 Hz.

In case 3, the number of flicker determinations is 3 when the light source flicker frequency is 50 Hz, the number of flicker determinations is 3 when the light source flicker frequency is 60 Hz, and the number of flicker determinations is 2 It is an example. In the case 3 case, since the number of flicker determinations when the blinking frequency of the light source is 50 Hz and 60 Hz is less than the variation removal determination threshold, the final determination result is set to no flicker detection.

In case 4, the number of flicker determinations is 3 when the light source flicker frequency is 50 Hz, the number of flicker determinations is 5 when the light source flicker frequency is 60 Hz, and the number of flicker determinations is 0. It is an example. In the case 4 case, since the number of flicker determinations when the flicker frequency of the light source is 60 Hz is equal to or greater than the variation removal determination threshold, the final determination result is flicker detection when the flicker frequency of the light source is 60 Hz.

As described above, in the signal processing device 50 according to the first embodiment, for the plurality of detection frames 41 ₁ to 41 ₆ , the luminance detection values obtained by detecting the luminance of each pixel for each detection frame are obtained for a plurality of frames. By using it, the frame periodicity of each detection frame is detected for the flicker component. The presence or absence of flicker is determined from the detection result of the frame periodicity. Thus, even when flicker fringes within one screen, that is, when in-plane line flicker does not occur, occurrence of flicker can be detected with high accuracy.

In particular, the maximum value Max and the minimum value Min in the luminance detection values of the past N frames including the frame of interest are obtained, normalization processing is performed using the difference (Max-Min), and the frames of the detection frames 41 ₁ to 41 ₆ with respect to the flicker component Periodicity is evaluated. By this normalization processing, even when the flicker due to flicker is very small, for example, when the exposure time is long, the periodicity of flicker can be detected with high accuracy.

Also, when the difference (Max-Min) falls below a predetermined minimum amplitude threshold, it is determined that the luminance is not flickering due to flicker, and the evaluation value of the frame periodicity is set to zero. Thereby, since flickering of luminance that is not caused by flicker can be excluded, the periodicity of flicker can be detected with higher accuracy. Further, screen ratio determination results for the past M frames are accumulated, and the final determination result is set to flicker when the number of flicker determinations is equal to or greater than a predetermined variation removal determination threshold. As a result, it is possible to prevent erroneous detection of flicker due to changes in the subject.

Second Embodiment
The second embodiment is an example when applied to an imaging pixel unit (imaging device) having a wide dynamic range (HDR). An example of the configuration of a signal processing device (signal processing circuit) according to the second embodiment is shown in FIG.

From the imaging pixel unit 21 (see FIG. 3), in order to widen the dynamic range, a long-time accumulation imaging signal obtained by accumulating signal charges for a long time and a signal charge for a short time are accumulated for one pixel. And a short-time accumulated imaging signal obtained in this way. Then, by performing predetermined processing using the long-time accumulated image signal and the short-time accumulated image signal, an image signal having a contrast with respect to a wide range of incident light amounts can be obtained.

In the signal processing device 50 according to the second embodiment, the luminance detection unit 51 is provided for each of the detection frames 41 ₁ to 41 _{6 for} each of the long-time accumulation imaging signal and the short-time accumulation imaging signal input from the imaging pixel unit 21. The luminance of the pixel is detected, and the luminance detection value is given to the period detection unit 52. That is, the luminance detection unit 51 includes a long-time accumulation luminance detector 51 _L that processes a long-time accumulation imaging signal and a short-time accumulation luminance detector 51 _S that processes a short-time accumulation imaging signal. The function of the long-time accumulation luminance detector 51 _L and short accumulation luminance detector 51 _S is the same as the brightness detector 51 _{1-51 6} in the first embodiment.

The period detector 52 also has a long-time accumulation flicker detector 52 _L and a short-time accumulation flicker detector 52 _S corresponding to the luminance detector 51. The function of the long-time accumulation flicker detector 52 _L and short storage flicker detector 52 _S is the same as the N-frame period flicker detector 52 _{1-52 6} in the first embodiment. In FIG. 11, the screen ratio determination unit 53 and the variation removal unit 54 of the first embodiment are not shown.

Various processes can be considered for the long-time accumulation flicker detection result by the long-time accumulation flicker detector 52 _L and the short-time accumulation flicker detection result by the short-time accumulation flicker detector 52 _S. As an example, when a flicker detection result is output only from the short-time accumulation flicker detector 52 _S , a process of performing control so that flicker does not occur only on the short-time accumulation side can be exemplified.

Even in the signal processing device 50 according to the second embodiment described above, the same operations and effects as those of the signal processing device 50 according to the first embodiment can be obtained. That is, even when flicker stripes within one screen, that is, in-plane line flicker does not occur, the occurrence of flicker can be detected with high accuracy. Further, even when flicker due to flicker is small, it is possible to accurately detect flicker periodicity and to prevent false detection of flicker due to changes in the subject.

<Imaging Device of the Present Disclosure>
An example of the system configuration of the imaging apparatus of the present disclosure is illustrated in FIG. The imaging device 1 according to the present disclosure includes a sensor unit 60, an analog signal processing circuit 70, and a digital signal processing circuit 80. The sensor unit 60 corresponds to the imaging pixel unit 21 in FIG. The analog signal processing circuit 70 includes a signal amplification unit 71 and an A / D conversion unit 72. The signal amplifying unit 71 performs processing for amplifying an analog imaging signal obtained by the imaging pixel unit 21. The A / D conversion unit 72 corresponds to the A / D conversion unit 23 in FIG. 2, and performs a process of converting an analog imaging signal output from the signal amplification unit 71 into a digital imaging signal.

The digital signal processing circuit 80 includes the function of the signal processing device (circuit) 50 according to the first embodiment or the second embodiment, that is, the function of detecting flicker, and the imaging pixel unit 21 and the signal amplification based on the detection result. And a function of controlling the A / D converter 72. Specifically, the digital signal processing circuit 80 includes a luminance detection unit 81, a flicker detection unit 82, an exposure control unit 83, a drive circuit control unit 84, and a drive circuit unit 85.

In the digital signal processing circuit 80, the luminance detection unit 81 includes the luminance detection unit 51 of the first embodiment or the second embodiment, and a plurality of detection frames 41 ₁ to 41 ₆ of the imaging screen 40 (see FIG. 5). Then, the luminance of each pixel is detected for each detection frame based on the imaging signal, and the luminance detection value is given to the flicker detection unit 82. The flicker detection unit 82 includes the period detection unit 52, the screen ratio determination unit 53, and the variation removal unit 54 of the first or second embodiment, and flicker based on the luminance detection value provided from the luminance detection unit 81. The presence / absence of presence / absence is detected and the detection result is given to the exposure control unit 83.

The exposure control unit 83 sets the signal charge accumulation time of the imaging pixel unit 21 and the gain value of the signal amplification unit 71 based on the detection result of the flicker detection unit 82. Specifically, for example, when the detection result of the flicker detection unit 82 is flicker, the exposure control unit 83 sets an accumulation time of n / 100 seconds for a 50 Hz light source, and n / 120 for a 60 Hz light source. Set the accumulation time in seconds. The drive circuit control unit 84 controls the imaging pixel unit 21, the signal amplification unit 71, and the A / D conversion unit 72 via the drive circuit unit 85 based on the accumulation time and gain value set by the exposure control unit 83. To drive.

As described above, according to the signal processing device 50 according to the first embodiment or the second embodiment, even if flicker fringes in one screen, that is, in-plane line flicker does not occur, occurrence of flicker is accurate. Can be detected well. Therefore, according to the imaging device 1 of the present disclosure including the digital signal processing circuit 80 having the function of the signal processing device 50 according to the first embodiment or the second embodiment, the flicker is detected with high accuracy of occurrence of flicker. It is possible to obtain a high-quality image without any image.

The system configuration shown in FIG. 12 assumes a system configuration in which the digital signal processing circuit 80 is built in together with the analog signal processing circuit 70 inside the image sensor including the sensor unit 60, but is limited to this system configuration. It is not a thing. That is, a system configuration in which the digital signal processing circuit 80 is provided outside the imaging device including the sensor unit 60 and the analog signal processing circuit 70 can also be adopted.

The imaging device 1 of the present disclosure can be used as an imaging device such as a digital still camera or a video camera. Furthermore, the imaging device 1 of the present disclosure can be used as an imaging unit in various electronic devices having an imaging function such as a mobile phone and a smartphone in addition to the camera.

<Modification>
Although the present disclosure has been described based on the preferred embodiments, the present disclosure is not limited to these embodiments. The configuration and structure of the image sensor described in each of the above embodiments are examples, and can be changed as appropriate. For example, the case where the present invention is applied to a pseudo-global shutter type imaging device (imaging pixel unit) has been described as an example, but the present invention can also be applied to a global shutter type imaging device. In addition, if the number of detection frame divisions is set to be larger than that in the case of the pseudo global shutter system, the present invention can also be applied to a rolling shutter system image sensor.

Further, in each of the above embodiments, the imaging of a three-layer structure in which the pixel substrate on which the imaging pixel unit is formed, the memory substrate on which the memory unit is formed, and the circuit substrate on which the signal processing circuit is formed is stacked. Although the case where it applied to the element was mentioned as an example and demonstrated, it is not restricted to application to a laminated structure of 3 layers. For example, the present invention can be applied to a two-layer structure in which the memory substrate is omitted and the memory portion is formed on the circuit board. Furthermore, the present invention is not limited to a stacked structure, and can be applied to a so-called flat structure in which a memory portion and a signal processing circuit are formed together with an imaging pixel portion on a pixel substrate.

In addition, the technology of the present disclosure can be applied regardless of the format of the image sensor such as a monochrome image sensor or a Bayer array image sensor. Furthermore, in each of the above-described embodiments, the case where luminance is detected has been exemplified. However, in order to simplify the configuration of the color-capable imaging device, for example, a configuration in which only information on green pixels is detected may be employed. It is.

<Configuration of the present disclosure>
In addition, this indication can also take the following structures.
[1] A luminance detection unit that detects the luminance of each pixel for each detection frame based on the imaging signal for a plurality of detection frames set by dividing the imaging screen in the vertical scanning direction;
A period detection unit that detects the frame periodicity of each detection frame for flicker components using the luminance detection values obtained by the luminance detection unit for a plurality of frames; and
A determination unit that determines the presence or absence of flicker based on the detection result of the frame periodicity by the cycle detection unit,
A signal processing apparatus comprising:
[2] The luminance detection value is a luminance integrated value obtained by integrating the luminance information of each pixel for each detection frame, or a luminance average value obtained by averaging the luminance integrated value in units of one pixel.
The signal processing device according to [1] above.
[3] The plurality of frames are the past N frames (N is a value determined by the driving frequency of the image sensor and the blinking frequency of the light source) including the frame of interest.
The period detection unit obtains a maximum value and a minimum value in luminance detection values of N frames, performs a normalization process based on a difference between the maximum value and the minimum value, and evaluates the frame periodicity of each detection frame with respect to a flicker component.
The signal processing device according to [1] or [2].
[4] The period detection unit sets the evaluation value of the frame periodicity to 0 when the difference between the maximum value and the minimum value in the luminance detection values of N frames is lower than a predetermined minimum amplitude threshold value.
The signal processing device according to [3] above.
[5] The period detector
A high-frequency detection processing unit that detects whether or not a high-frequency component exists in the N frame; and
A periodicity detection processing unit that detects the periodicity of the flicker component in the N frame in units of frames based on a difference in luminance detection values between the N frames;
The signal processing device according to any one of [1] to [4].
[6] The period detection unit is configured such that, based on the detection results of the high-frequency detection processing unit and the periodicity detection processing unit, a high-frequency component exists in N frames and the flicker component has periodicity in units of frames. Detect flicker component of frame period,
The signal processing device according to [5] above.
[7] The high frequency detection processing unit performs discrete Fourier transform on the luminance detection value, and determines the presence or absence of a flicker component based on whether or not a high frequency component exists in the imaging signal from the obtained frequency information.
The signal processing device according to [5] or [6] above.
[8] The periodicity detection processing unit determines that the flicker component in the N frame has periodicity in units of frames when the difference in luminance detection value between the N frames is smaller than a predetermined periodicity determination threshold.
The signal processing device according to [5] or [6] above.
[9] The determination unit determines the presence or absence of a flicker component as a whole screen based on the detection result of the frame periodicity of each detection frame by the cycle detection unit.
The signal processing device according to any one of [1] to [8].
[10] The determination unit counts the detection result of the frame periodicity of each detection frame by the cycle detection unit, calculates the flicker detection ratio by dividing the total count by the total number of detection frames, and sets the flicker detection ratio to a predetermined value. The presence or absence of flicker is determined by comparing with the area ratio determination threshold of
The signal processing device according to [9] above.
[11] A variation removal unit that accumulates the determination results of the presence or absence of flicker in each frame by the determination unit and removes variations in the determination results to obtain a final determination result.
The signal processing device according to any one of [1] to [10].
[12] The variation removal unit accumulates the determination results for the past M frames by the determination unit, and sets the final determination result to flicker when the number of flicker determinations is equal to or greater than a predetermined variation removal determination threshold.
The signal processing device according to [11] above.
[13] When a long-time accumulation imaging signal obtained by accumulating signal charges for a long time and a short-time accumulation imaging signal obtained by accumulating signal charges for a short time are input to the luminance detection unit for one pixel ,
The luminance detection unit detects the luminance of each pixel for each detection frame for each of the long-time accumulation imaging signal and the short-time accumulation imaging signal, and gives the luminance detection value to the period detection unit.
The signal processing device according to any one of [1] to [12].
[14] For a plurality of detection frames set by dividing the imaging screen in the vertical scanning direction, the luminance of each pixel is detected for each detection frame based on the imaging signal;
Detect the frame periodicity of each detection frame for flicker components using the luminance detection value for multiple frames,
The presence or absence of flicker is determined based on the detection result of the frame periodicity.
Signal processing method.
[15] An imaging pixel unit, and
A signal processing circuit for processing an imaging signal output from the imaging pixel unit;
With
The signal processing circuit
For a plurality of detection frames set by dividing the imaging screen in the vertical scanning direction, a luminance detection unit that detects the luminance of each pixel for each detection frame based on the imaging signal,
A period detection unit that detects the frame periodicity of each detection frame for flicker components using the luminance detection values obtained by the luminance detection unit for a plurality of frames; and
A determination unit that determines the presence or absence of flicker based on the detection result of the frame periodicity by the cycle detection unit;
Imaging device.
[16] The imaging pixel unit is composed of a CMOS sensor.
The imaging device according to [15] above.
[17] A pixel substrate on which an imaging pixel portion is formed, and
The circuit board on which the signal processing circuit is formed is laminated,
Between the pixel substrate and the circuit board, there is provided a memory substrate in which a memory unit for temporarily storing an imaging signal output from the imaging pixel unit is formed.
The imaging device according to [16] above.
[18] Under a light source environment in which brightness periodically varies, an imaging signal is read from the imaging pixel unit to the memory unit with a readout time shorter than the readout time in the rolling shutter system.
The imaging device according to [17] above.
[19] Read an imaging signal from the imaging pixel unit to the memory unit with a readout time shorter than the blinking cycle of the light source.
The imaging device according to [18] above.
[20] The imaging pixel unit supplies, for each pixel, a long-time accumulation imaging signal with a long signal charge accumulation time and a short-time accumulation imaging signal with a short signal charge accumulation time to the signal processing circuit.
The imaging device according to any one of [15] to [19].

DESCRIPTION OF SYMBOLS 1 ... Imaging device, 10 ... Imaging element, 11, 12, 13 ... Semiconductor substrate, 21 ... Imaging pixel part, 22 ... Memory part, 23 ... A / D conversion part, 24 ... logic circuit, 25 ... signal line, 30 ... pixel, 31 ... photodiode (photoelectric conversion element), 32 ... transfer transistor, 33 ... reset transistor, 34 ... Amplifying transistor, 35 ... select transistor, 40 ... imaging screen, 41 ₁ to 41 ₆ ... detection frame, 50 ... signal processing device (signal processing circuit), 51 ... luminance detection unit, 52 ... Period detection unit, 53 ... Screen ratio determination unit, 54 ... Variance removal unit, 60 ... Sensor unit, 70 ... Analog signal processing circuit, 80 ... Digital signal processing circuit, 511 ... Normalization processing unit, 512 ... N Over beam frequency detection processing section, 513 ... N frame periodicity detection processing section, 514 ... logical unit

Claims (20)

1. For a plurality of detection frames set by dividing the imaging screen in the vertical scanning direction, a luminance detection unit that detects the luminance of each pixel for each detection frame based on the imaging signal,
   A period detection unit that detects the frame periodicity of each detection frame for flicker components using the luminance detection values obtained by the luminance detection unit for a plurality of frames; and
   A determination unit that determines the presence or absence of flicker based on the detection result of the frame periodicity by the cycle detection unit,
   A signal processing apparatus comprising:
2. The luminance detection value is a luminance integrated value obtained by integrating the luminance information of each pixel for each detection frame, or a luminance average value obtained by averaging the luminance integrated value in units of one pixel.
   The signal processing apparatus according to claim 1.
3. The plurality of frames are the past N frames including the frame of interest (N is a value determined by the drive frequency of the image sensor and the blinking frequency of the light source),
   The period detection unit obtains a maximum value and a minimum value in luminance detection values of N frames, performs a normalization process based on a difference between the maximum value and the minimum value, and evaluates the frame periodicity of each detection frame with respect to a flicker component.
   The signal processing apparatus according to claim 1.
4. The period detection unit sets the evaluation value of the frame periodicity to 0 when the difference between the maximum value and the minimum value in the luminance detection values of N frames is lower than a predetermined minimum amplitude threshold value.
   The signal processing apparatus according to claim 3.
5. The period detector
   A high-frequency detection processing unit that detects whether or not a high-frequency component exists in the N frame; and
   A periodicity detection processing unit that detects the periodicity of the flicker component in the N frame in units of frames based on a difference in luminance detection values between the N frames;
   The signal processing apparatus according to claim 1.
6. Based on the detection results of the high-frequency detection processing unit and the periodicity detection processing unit, the period detection unit has a high-frequency component in N frames and the flicker component has periodicity in units of frames. Detect flicker components,
   The signal processing apparatus according to claim 5.
7. The high frequency detection processing unit performs discrete Fourier transform on the luminance detection value, and determines the presence or absence of a flicker component based on whether or not a high frequency component exists in the imaging signal from the obtained frequency information.
   The signal processing apparatus according to claim 5.
8. The periodicity detection processing unit determines that the flicker component in the N frame has periodicity in units of frames when the difference in the luminance detection value between the N frames is smaller than a predetermined periodicity determination threshold.
   The signal processing apparatus according to claim 5.
9. The determination unit determines the presence or absence of a flicker component as a whole screen based on the detection result of the frame periodicity of each detection frame by the cycle detection unit.
   The signal processing apparatus according to claim 1.
10. The determination unit counts the detection result of the frame periodicity of each detection frame by the period detection unit, calculates the flicker detection ratio by dividing the total count by the total number of detection frames, and calculates the flicker detection ratio to a predetermined area ratio. The presence or absence of flicker is determined by comparing with a determination threshold value.
    The signal processing apparatus according to claim 9.
11. The determination unit accumulates the determination result of the presence or absence of flicker in each frame, and includes a variation removal unit that removes variation in the determination result and sets it as a final determination result.
    The signal processing apparatus according to claim 1.
12. The variation removal unit accumulates the determination results for the past M frames by the determination unit, and sets the final determination result to flicker when the number of determinations with flicker is equal to or greater than a predetermined variation removal determination threshold.
    The signal processing apparatus according to claim 11.
13. When a long-time accumulation imaging signal obtained by accumulating signal charges for a long time and a short-time accumulation imaging signal obtained by accumulating signal charges for a short time are input to the luminance detection unit for one pixel,
    The luminance detection unit detects the luminance of each pixel for each detection frame for each of the long-time accumulation imaging signal and the short-time accumulation imaging signal, and gives the luminance detection value to the period detection unit.
    The signal processing apparatus according to claim 1.
14. For a plurality of detection frames set by dividing the imaging screen in the vertical scanning direction, the luminance of each pixel is detected for each detection frame based on the imaging signal,
    Detect the frame periodicity of each detection frame for flicker components using the luminance detection value for multiple frames,
    The presence or absence of flicker is determined based on the detection result of the frame periodicity.
    Signal processing method.
15. An imaging pixel unit, and
    A signal processing circuit for processing an imaging signal output from the imaging pixel unit;
    With
    The signal processing circuit
    For a plurality of detection frames set by dividing the imaging screen in the vertical scanning direction, a luminance detection unit that detects the luminance of each pixel for each detection frame based on the imaging signal,
    A period detection unit that detects the frame periodicity of each detection frame for flicker components using the luminance detection values obtained by the luminance detection unit for a plurality of frames; and
    A determination unit that determines the presence or absence of flicker based on the detection result of the frame periodicity by the cycle detection unit;
    Imaging device.
16. The imaging pixel unit is composed of a CMOS sensor.
    The imaging device according to claim 15.
17. A pixel substrate on which an imaging pixel unit is formed; and
    The circuit board on which the signal processing circuit is formed is laminated,
    Between the pixel substrate and the circuit board, there is provided a memory substrate in which a memory unit for temporarily storing an imaging signal output from the imaging pixel unit is formed.
    The imaging device according to claim 16.
18. In a light source environment in which brightness varies periodically, an imaging signal is read from the imaging pixel unit to the memory unit with a readout time shorter than the readout time in the rolling shutter method.
    The imaging device according to claim 17.
19. Reading the imaging signal from the imaging pixel unit to the memory unit with a readout time shorter than the blinking cycle of the light source,
    The imaging device according to claim 18.
20. The imaging pixel unit supplies a signal processing circuit with a long-time accumulation imaging signal obtained by accumulating signal charges for a long time and a short-time accumulation imaging signal obtained by accumulating signal charges for a short time for one pixel. ,
    The imaging device according to claim 15.

Images