JP2013051523A - Flicker detection device, flicker detection method, control program, readable recording medium, solid-state imaging device, multi-eye imaging device, and electronic information apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately perform flicker determination by enabling effective detection of flicker noise by reducing the influence of a subject image while suppressing an increase in chip size or the number of components such as a frame memory.SOLUTION: A flicker detection device 1 for detecting the occurrence of a flicker from each image signal of a plurality of image sensors comprises: image processing means 3 that performs image processing on two simultaneously-photographed image signals to exclude image information other than a flicker component; and flicker detection means 4 that extracts flicker information from an image signal after the exclusion processing of image information other than the flicker component.

Description

  The present invention relates to a flicker detection apparatus that detects occurrence of flicker from images from a plurality of sensors corresponding to a plurality of viewpoints, a flicker detection method using the same, and solid-state imaging that can correct flicker detected by the flicker detection apparatus A multi-lens imaging device capable of imaging each image signal acquired from two or more solid-state imaging devices represented by CMOS (Complementary Metal Oxide Semiconductor) that photoelectrically converts image light from a subject Control program for causing a computer to execute a flicker detection method, a computer-readable readable recording medium on which the control program is recorded, a digital video camera and a digital still using the solid-state imaging device or the multi-eye imaging device as an imaging unit Digital camera Cameras and, image input camera, such as a vehicle-mounted camera, a scanner, a facsimile, an electronic information device, such as a camera-equipped mobile phone device.

  In a conventional solid-state imaging device, for example, a fluorescent lamp blinks at 120 Hz with respect to a commercial frequency of 60 Hz, and the blinking is not transmitted to the human eye, but in a solid-state imaging device, for example, 30 frames per second, There are several hundred lines in one frame, and exposure is performed for each line. For this reason, the exposure timing sequentially shifts in time series for each line, so that a situation occurs in which the image is bright at a certain timing and dark at a certain timing. This results in a hazy, hazy image. For this reason, a signal processing technique for detecting flicker caused by blinking and preventing the influence of flicker on an image is required.

  Since the commercial frequency is 60 Hz in the Kansai area and the commercial frequency is 50 Hz in the Kanto area, if you do not know whether the shutter speed is an integral multiple of 1/120 sec or an integral multiple of 1/100 sec, this flicker is It cannot be removed. For example, in the Kansai area, if the shutter speed is set to an integral multiple of 1/120 sec, it is the blinking cycle of the fluorescent lamp, so both the brightness and the darkness are included. It becomes constant. In order to realize this, Patent Document 1 discloses a flicker detection method for detecting and correcting whether the blinking cycle is 1/120 sec or 1/100 sec.

  FIG. 11 is a block diagram for explaining a schematic configuration example of a conventional flicker correction apparatus disclosed in Patent Document 1. In FIG.

  As shown in FIG. 11, the conventional flicker correction apparatus 100 has an input terminal 101, and an image signal Sig 1 is input from the input terminal 101. The image signal Sig1 is obtained by imaging a subject with a camera using a solid-state imaging device such as an electronic camera or a scanner. For example, under a fluorescent lamp illumination that blinks in synchronization with an AC power supply cycle, the CMOS image sensor When a subject is imaged by a camera using an XY address scanning type solid-state imaging device as described above, an image signal Sig1 including a flicker component is input from the input terminal 101.

  The image signal Sig1 input from the input terminal 1 is supplied to the image memory 102, and the image signal Sig1 for one frame or one field is stored in the image memory 102 as image data. The image signal Sig1 is also supplied to the image average value calculation unit 103.

  The image average value calculation means 103 calculates the average value of the image signal Sig1 for each horizontal line from the input image signal Sig1, and outputs the image average value Sig2 to the flicker data extraction means 104 and 105, respectively. To do. In this image average value calculating means 103, it is possible to calculate the average value of the image signal for each of a plurality of lines, and it is also possible to calculate the image average value Sig2 from the image signal Sig1 in the vertical direction.

  Further, the flicker frequency calculation means 106 calculates the flicker frequency using the AC power supply frequency and the frame frequency or field frequency of the image sensor. The flicker frequency calculating means 106 outputs the flicker frequency Sig3 when the AC power supply frequency is 50 Hz to the flicker data extracting means 104, and calculates the flicker frequency Sig4 when the AC power supply frequency is 60 Hz to calculate the flicker data extraction means 106. To 105.

  The flicker data extraction unit 104 performs a discrete Fourier transform (frequency conversion) on the image average value Sig2 input from the image average value calculation unit 103, thereby obtaining a spectrum amount Sig5 of the flicker frequency Sig3 input from the flicker frequency calculation unit 106. , Spectral amounts Sig5 ′ and S5 ″ of frequencies in the vicinity of the flicker frequency Sig3 are respectively calculated and output to the flicker determination means 107.

  Similarly, the flicker data extraction unit 105 performs discrete Fourier transform (frequency conversion) on the image average value Sig2 input from the image average value calculation unit 103, thereby obtaining the flicker frequency Sig4 input from the flicker frequency calculation unit 106. The spectrum amount Sig6 and the spectrum amounts Sig6 ′ and Sig6 ″ of frequencies near the flicker frequency Sig4 are respectively calculated and output to the flicker determination means 107.

  The flicker determination means 107 determines the presence / absence of the flicker phenomenon using the spectral amounts Sig5, Sig5 ′ and Sig5 ″, determines the presence / absence of the flicker phenomenon using the spectral amounts Sig6, Sig6 ′ and Sig6 ″, and the determination result Is output to the flicker correction means 108 as flicker information Sig8.

  When it is determined that there is a flicker phenomenon, the AC power supply frequency is determined by determining the presence or absence of the flicker phenomenon every time the AC power supply frequency is 50 Hz and 60 Hz, and the result is flicker information. It outputs to the switch SW109 as Sig7. The switch SW109 is switched according to the flicker information Sig7, and the spectral amounts Sig5, Sig5 ′ and Sig5 ″, or the spectral amounts Sig6, Sig6 ′ and the frequency according to the AC power supply frequency (50 Hz or 60 Hz) determined to cause the flicker phenomenon. Sig6 ″ is output to the flicker correction amount calculation means 110. If it is determined that there is no flicker phenomenon, flicker information Sig8 indicating that there is no flicker phenomenon is input to the flicker correction means 108 and no correction is performed.

  The flicker correction amount calculation unit 110 calculates the correction amount Sig9 by performing inverse discrete Fourier transform on the spectrum Sig5 or Sig6 input from the switch SW109, and outputs the calculated correction amount Sig9 to the flicker correction unit 108.

  The flicker correction unit 108 adds the flicker correction amount Sig9 input from the flicker correction amount calculation unit 110 to the image data including the flicker component stored in the image memory 102, and outputs after offsetting the flicker component. The output image signal Sig10 is output from the terminal 111.

  Next, Patent Document 2 discloses a conventional image processing apparatus that obtains an image without flicker by capturing and comparing a plurality of images having different times to detect and remove a flicker portion and synthesize an image. Yes.

  FIG. 12 is a flowchart for explaining a signal processing method for obtaining an image signal from which flicker is removed in the conventional image processing apparatus disclosed in Patent Document 2.

  As shown in FIG. 12, first, in step S201, two images having different times are captured as image signals.

  Next, in step S202, the absolute value of the difference in value is calculated for each pixel of the two captured images, and the pixel portion whose value is equal to or greater than a predetermined value is determined as “invalid portion. Create an intermediate image marked as The portion that is not determined as an invalid portion is a valid image, and is left as an “valid portion” in the intermediate image. Thereafter, the processes in the next steps S203 to S206 are repeated until the effective screen is filled with the entire image.

  In the process of step S203, it is determined whether or not there is an “invalid part” in the intermediate image. If there is no “invalid part” (NO), the final acquired image is output in step S207. If there is an “invalid portion” (YES), the process proceeds to the next step S204.

  In step S204, one image is captured again.

  Next, in the process of step S205, this image and the previous image are compared in the same manner as in the process of step S202, and the “invalid portion” of the new image is determined.

  Subsequently, in the process of step S206, if the effective part of this new image is applied to the "invalid part" of the intermediate image that has been determined in step S203 whether or not there is an "invalid part", the effective part is used. Embed in the intermediate image to obtain a new intermediate image.

  Thereafter, by repeating the processes of steps S203 to S206, when there is finally no invalid portion from the intermediate image, the intermediate image without this “invalid portion” is output as the final captured image in step S207, and the processing ends.

JP 2003-198932 A JP 11-32238 A

  However, since the conventional flicker correction apparatus 100 disclosed in Patent Document 1 uses only the image signal Sig1 input from one input terminal 101, when an image similar to flicker noise is input, or flicker There is a problem that it is difficult to detect the occurrence of flicker when the noise is very small.

  In short, as shown in FIG. 1311, the flicker data extraction means 104 and 105 of the conventional flicker correction apparatus 100 performs flicker by subjecting the image average value Sig2 input from the image average value calculation means 103 to discrete Fourier transform (frequency conversion). Since flicker determination is performed by the flicker determination means 107 by extracting the components, the flicker detection accuracy is often low and erroneously recognized due to the influence of the subject and the background image. When the solid-state image sensor is a CMOS sensor, flicker appears as horizontal stripes in the image, so the frequency component in the vertical direction of the screen is detected, and the presence of flicker is detected by the presence of a peak near the fluorescent lamp lighting frequency. Is possible. However, since the frequency is detected while being superimposed on the normal image, it is strongly influenced by the subject and the background image. In particular, in the case of an image with a strong change in strength in the vertical direction, such as a horizontal stripe image, misrecognition is particularly likely to occur.

  In the configuration of flicker detection and flicker removal disclosed in Patent Document 2 above, a plurality of images having different times are overlapped to detect and remove flicker, and an image is synthesized, so that a separate frame memory is required and the cost is high. At the same time, the number of components such as a frame memory and the chip size are increased, and there is a problem that a correct image cannot be configured for a moving image because the correlation between images between frames is small.

  The present invention solves the above-described conventional problems, and a flicker detection apparatus capable of suppressing flicker noise and effectively detecting flicker noise while suppressing an increase in the number of components such as a frame memory and a chip size, and accurately performing flicker determination Flicker detection method using the same, control program for causing a computer to execute the flicker detection method, a computer-readable readable recording medium on which the control program is recorded, a solid-state imaging device using the flicker detection device, and the flicker Multi-view imaging device capable of generating moving image including stereoscopic vision and still image / image using detection device, mobile phone with camera using this solid-state imaging device or multi-view imaging device as image input device in imaging section An object is to provide an electronic information device such as a device.

  The flicker detection apparatus of the present invention is a flicker detection apparatus that detects the occurrence of flicker from each image signal of a plurality of image sensors, and performs image processing on two image signals that have been shot simultaneously to obtain image information other than the flicker component. And image processing means for performing at least one of eliminating and emphasizing the flicker frequency component, and flicker detection means for extracting flicker information from the image signal after image information exclusion processing other than the flicker component. Yes, and the above objective is achieved.

  Preferably, the image processing means in the flicker detection device of the present invention includes difference processing means for performing difference processing on image information of two screens from the first image pickup element and the second image pickup element among the plurality of image pickup elements. .

  Further preferably, the image processing means in the flicker detection device of the present invention is configured to shift the vertical synchronization signal from the second image sensor with respect to the vertical synchronization signal from the first image sensor by a half cycle of the flicker frequency. It has a synchronizing signal shift means.

  Further preferably, the image processing means in the flicker detection device of the present invention is configured to shift the vertical synchronization signal from the second image sensor with respect to the vertical synchronization signal from the first image sensor by a half cycle of the flicker frequency. Synchronization signal shifting means, and difference processing means for differentially processing the image information of two screens from the first and second imaging elements in which the vertical synchronization signals are shifted from each other by a half cycle of the flicker frequency.

  Further preferably, the flicker detection means in the flicker detection apparatus of the present invention is an image average value calculation means for calculating image average value information for each horizontal line or a plurality of horizontal lines of one field of the image signal after the difference processing. Have

  Further preferably, the flicker detection means in the flicker detection apparatus of the present invention further comprises frequency conversion means for frequency-converting the calculated image average value information to extract a flicker frequency, and the flicker frequency is used as the flicker information. Extract.

  Further preferably, the frequency conversion means in the flicker detection apparatus of the present invention detects flicker information of the flicker frequency by performing discrete Fourier transform on the calculated image average value information.

  The solid-state imaging device of the present invention corrects each image signal of the plurality of imaging devices based on the flicker detection device of the present invention and the flicker information output from the flicker detection device, and Flicker correction means for outputting each image signal from which the flicker frequency component in each image signal has been removed, thereby achieving the above object.

  The solid-state imaging device according to the present invention includes the flicker detection device according to the present invention and sensor drive control for setting an exposure time during which flicker does not occur to each of the plurality of imaging elements based on the flicker information output from the flicker detection device. Means for achieving the above object.

  The multi-lens image pickup device of the present invention can pick up each image signal acquired from two or more image pickup elements of the solid-state image pickup device of the present invention, thereby achieving the above object.

  An electronic information device according to the present invention uses the solid-state imaging device according to the present invention or the multi-eye imaging device according to the present invention as an image input device in an imaging unit, thereby achieving the above object.

   The flicker detection method of the present invention is a flicker detection method for detecting the occurrence of flicker from each image signal of a plurality of image sensors, in which an image processing means performs image processing on two image signals photographed at the same time to produce a flicker component. An image processing step that performs at least one of eliminating image information other than the above and emphasizing the flicker frequency component, and a flicker detection unit extracts flicker information from the image signal after the image information exclusion processing other than the flicker component. And a flicker detection step for extracting, thereby achieving the above object.

  Preferably, in the image processing step in the flicker detection method of the present invention, the difference processing means performs difference processing on the image information of two screens from the first image sensor and the second image sensor among the plurality of image sensors. It has a differential processing step.

  Further preferably, in the image processing step in the flicker detection method of the present invention, the vertical synchronization signal shift means converts the vertical synchronization signal from the second image sensor to the flicker frequency with respect to the vertical synchronization signal from the first image sensor. A vertical synchronizing signal shift step of shifting by a half cycle.

  Further preferably, in the image processing step in the flicker detection method of the present invention, the vertical synchronization signal shift means converts the vertical synchronization signal from the second image sensor to the flicker frequency with respect to the vertical synchronization signal from the first image sensor. The vertical synchronization signal shift step for shifting the half-cycle of the first image sensor and the second image sensor from which the difference processing means mutually shifts the vertical synchronization signal by the half-cycle of the flicker frequency. And a differential processing step for differential processing.

  Further preferably, in the flicker detection step in the flicker detection method of the present invention, the image average value calculation means calculates image average value information for each horizontal line or a plurality of horizontal lines of one field of the image signal after the difference processing. And an image average value calculating step for extracting flicker information based on the image average value information.

  Further preferably, the flicker detection step in the flicker detection method of the present invention further includes a frequency calculation step in which the frequency calculation means extracts the flicker information by converting the frequency of the calculated image average value information.

  Further preferably, the frequency calculation step in the flicker detection method of the present invention detects flicker information of the flicker frequency by performing discrete Fourier transform on the calculated image average value information.

  The control program of the present invention is for causing a computer to execute each step of the flicker detection method of the present invention, thereby achieving the above object.

  The readable recording medium of the present invention is a computer-readable recording medium on which the control program of the present invention is recorded, whereby the above object is achieved.

  With the above configuration, the operation of the present invention will be described below.

  In the present invention, in a flicker detection device that detects the occurrence of flicker from each image signal of a plurality of image sensors, image information other than the flicker component is removed by performing image processing on two image signals that have been shot simultaneously, and flicker Image processing means for performing at least one of enhancing the frequency component, and flicker detection means for extracting flicker information from the image signal after image information excluding the flicker component is excluded.

  As a result, at least one of removing image information other than the flicker component and emphasizing the flicker frequency component by performing image processing on the two image signals photographed at the same time, the influence of the subject and the background image is exerted. It is possible to reduce or emphasize the flicker frequency component to suppress an increase in the number of components such as a frame memory and a chip size as in the prior art, and to detect flicker noise effectively and perform flicker determination with high accuracy.

  As described above, according to the present invention, in order to perform at least one of removing image information other than the flicker component and enhancing the flicker frequency component by performing image processing on two image signals captured simultaneously, In addition, the influence of the background image can be reduced and the flicker frequency component can be emphasized to suppress the increase in the number of components such as the frame memory and the chip size, and the flicker noise can be detected effectively, and the flicker determination can be performed with high accuracy.

It is a block diagram which shows the principal part structural example of the flicker detection apparatus of Embodiment 1 of this invention. It is a block diagram which shows the principal part structural example of the flicker detection means of FIG. It is a figure for demonstrating the flicker detection means of FIG. It is a block diagram which shows the principal part structural example of the image processing means of FIG. It is a figure for demonstrating the vertical synchronizing signal shift means of the image processing means of FIG. It is a figure for demonstrating the difference processing means of the image processing means of FIG. It is a block diagram which shows the principal part structural example of the solid-state imaging device using the flicker detection apparatus of Embodiment 1 of this invention. It is a block diagram which shows the other example of the principal part structure example of the solid-state imaging device using the flicker detection apparatus of Embodiment 1 of this invention. It is a block diagram which shows the principal part structural example of the flicker detection apparatus of Embodiment 2 of this invention. FIG. 9 is a block diagram illustrating a schematic configuration example of an electronic information device using, as an imaging unit, a solid-state imaging device or a multi-eye imaging device using the flicker detection device according to Embodiment 1 or 2 of the present invention as Embodiment 3 of the present invention. . It is a block diagram for demonstrating the example of schematic structure of the conventional flicker correction apparatus currently disclosed by patent document 1. FIG. 10 is a flowchart for explaining a processing method for obtaining an image from which flicker has been removed in a conventional image processing apparatus disclosed in Patent Document 2; It is a figure for demonstrating the subject in the conventional image processing apparatus of FIG.

  Hereinafter, a flicker detection apparatus and a flicker detection method of the present invention, a first embodiment of a solid-state imaging apparatus using them, a flicker detection apparatus and a flicker detection method of this software configuration, and a second embodiment of a solid-state imaging apparatus using them are described. Embodiment 3 of an electronic information device such as a mobile phone device with a camera using the solid-state imaging device or the other-eye imaging device of Embodiments 1 and 2 as an image input device in an imaging unit will be described with reference to the drawings. However, it explains in detail.

(Embodiment 1)
FIG. 1 is a block diagram illustrating a configuration example of a main part of the flicker detection apparatus according to the first embodiment of the present invention.

  As shown in FIG. 1, the flicker detection apparatus 1 according to the first embodiment performs image processing on each of two image signals simultaneously captured by at least two image sensors to eliminate image information other than the flicker component. 3 and flicker detection means 4 for extracting flicker information from the image signal after eliminating the adverse effect of the image information, and detecting the occurrence of flicker from each image signal from the plurality of image sensors.

  That is, the flicker detection apparatus 1 according to the first embodiment has a plurality of input terminals 2, and a plurality of image signals Sig 11 are input from the input terminals 2, respectively. Each input image signal Sig11 is obtained by imaging a subject and a background image by a camera using a plurality of imaging elements such as an electronic camera, a scanner, etc., for example, fluorescent lamp illumination that blinks in synchronization with an AC power supply cycle Below, when a subject and a background image are imaged with a camera using an XY address scanning type solid-state imaging device such as a CMOS image sensor, each image signal Sig11 of image information including a flicker component is Input from the input terminal 2 respectively.

  The plurality of image signals Sig11 input from each input terminal 2 excludes image information from two image signals Sig11 of the plurality of image signals Sig11 that are simultaneously shot so that the flicker detection can be easily performed in the image processing unit 3. Signal processing is performed, and this image information is eliminated, and an image signal Sig 12 that facilitates flicker detection is output to the flicker detection means 4.

  The image signal Sig12 is input to the flicker detection unit 4, and the flicker detection unit 4 extracts flicker information based on the image signal Sig12 and outputs flicker information Sig13. The flicker information Sig13 can be used to remove flicker from the plurality of image signals Sig11.

  FIG. 2 is a block diagram showing a configuration example of a main part of the flicker detection unit 4 in FIG. 1, and FIG. 3 is a diagram for explaining the flicker detection unit 4 in FIG.

  As shown in FIG. 2, the flicker detection means 4 calculates the average image information for the image signal Sig12 from which the image information is eliminated and the flicker detection becomes easy for each horizontal line or a plurality of lines of one field. A value calculating unit 41; and a frequency calculating unit 42 for extracting the flicker frequency information by performing frequency conversion on the calculated image average value information.

  As shown in FIG. 3, the image average value calculation means 41 of the flicker detection means 4 calculates an image average value for each horizontal line or a plurality of lines of one image (one field) of the image signal Sig12 from which the image information is excluded, The frequency calculating means 42 performs discrete Fourier transform on the calculated image average value information to detect the flicker frequency as flicker information. In this way, flicker frequency flicker information Sig13 as a determination result of the flicker phenomenon is detected based on a change in image average value information (change in signal intensity) for each horizontal line or for a plurality of lines on the time axis in the vertical direction. .

  In order to correctly determine the flicker information Sig13 of the flicker frequency, it is preferable that the flicker component included in the image average value Sig12 is large and the image components other than the flicker component are small. Here, in order to reduce the image components other than the flicker component, as described above, the image processing unit 3 performs image processing on the two image signals Sig11 photographed at the same time to eliminate image information other than the flicker component. ing.

  That is, since a single image signal photographed by a monocular imaging device as in the prior art cannot distinguish between a flicker component and other image components, the image signal Sig11 must be determined as it is. In a multi-lens image pickup device or a solid-state image pickup device having elements, a plurality of image signals Sig11 photographed at the same time (the same image) are input. Therefore, the image information of the plurality of image signals Sig11 is processed and image information other than flicker components By eliminating this, it is possible to keep image components other than the flicker component small.

  FIG. 4 is a block diagram showing a configuration example of a main part of the image processing means 3 in FIG.

  In FIG. 4, the image processing means 3 converts the vertical synchronization signal from the second solid-state image sensor for a half cycle of the commercial frequency (flicker frequency) with respect to the vertical synchronization signal from the first solid-state image sensor among the plurality of image sensors. Vertical synchronizing signal shift means 31 that shifts by a half cycle) and difference processing means 32 that performs differential processing on each of the two image signals from the first solid-state imaging device and the second solid-state imaging device.

  In this example, the difference processing means 32 performs difference processing on the two image signals, with the image processing that excludes image information other than the flicker component from the two image signals simultaneously captured.

  By reducing the image component other than the flicker component by the difference processing means 32, the flicker determination can be performed with high accuracy. Further, by controlling the vertical synchronization signal shift means 31 so that the flicker component of the difference processing means 32 is increased, flicker determination can be performed with high accuracy. Hereinafter, a detailed description will be given with reference to FIGS. 5 and 6.

  FIG. 5 is a diagram for explaining the vertical synchronizing signal shift means 31 of the image processing means 3 of FIG.

  In FIG. 5, for example, under a 50 Hz fluorescent lamp, the flickering of the fluorescent lamp flicker is repeated every 1/100 seconds, so the vertical synchronization signal Sig21 from the first solid-state image sensor and the vertical synchronization intention signal from the second solid-state image sensor. By shifting Sig22 by 1/200 second, the phase of the flicker component of the image signal Sig11 from the first solid-state image sensor and the image signal Sig11 from the second solid-state image sensor can be shifted by 180 degrees (half cycle of frequency 100 Hz). it can. Alternatively, for example, under a 60 Hz fluorescent lamp, since the flickering of the fluorescent lamp flickers every 1/120 seconds, the vertical synchronization signal Sig21 from the first solid-state image sensor and the vertical synchronization intention signal Sig22 from the second solid-state image sensor are obtained. By shifting 1/240 seconds, the phases of the flicker components of the image signal Sig11 from the first solid-state image sensor and the image signal Sig11 from the second solid-state image sensor can be shifted by 180 degrees (half cycle of frequency 120 Hz). In this way, the vertical synchronization signal shift means 31 shifts the vertical synchronization signal Sig22 from the second solid-state image sensor by the half cycle of the commercial frequency with respect to the vertical synchronization signal Sig21 from the first solid-state image sensor. The flicker component can be emphasized after subsequent difference processing.

  In the vertical synchronizing signal shift means 31, the trademark frequency is initially set to a shift amount of 50 Hz or 60 Hz, and the trademark frequency is 50 Hz or 60 Hz according to the flicker information Sig13 from the flicker detection means 4. Depending on whether or not the vertical synchronization signal Sig22 of the second solid-state image sensor 12 is shifted by 1/200 second or 1/240 second with respect to the vertical synchronization signal Sig21 of one first solid-state image sensor 11 You may make it switch.

  FIG. 6 is a diagram for explaining the difference processing means 32 of the image processing means 3 in FIG.

  In FIG. 6, the difference detection unit 32 of the image processing unit 3 obtains the difference between the image signal Sig11a from the first solid-state image sensor and the image signal Sig11b from the second solid-state image sensor, so that the subject and the background image ( The adverse effect of (image information) can be reduced. In the image signal Sig12 after the difference processing, as described above, before the difference processing, the flicker components are emphasized by shifting the vertical synchronization signals of each other by a half cycle of the commercial frequency, and the image components other than the flicker are subtracted from each other. It is small. Thereby, flicker determination can be performed with high accuracy. In this case, in order to further suppress the influence of the image, the difference processing can be performed by matching the positions of the subjects of the image signal Sig11a and the image signal Sig11b.

  In the flicker detection unit 4, the flicker component is extracted by the image average value calculation unit 41 and the frequency calculation unit 42 using the image signal Sig 12 after the difference processing in which the flicker component is emphasized and the influence of the subject and the background image is removed. As a result, the flicker occurrence detection accuracy is significantly increased as compared with the conventional case where only the monocular image signal Sig11 is used.

  In this way, by taking the difference between the two images whose horizontal stripes are shifted, the influence of the subject can be weakened and the horizontal stripes of flicker can be enhanced. In particular, in the case of a 3D camera (multi-lens image pickup apparatus), the images of the two cameras A and B have almost no deviation in the vertical direction. Therefore, taking the difference greatly reduces the influence of the subject in the vertical direction. be able to.

  FIG. 7 is a block diagram illustrating a configuration example of a main part of a solid-state imaging device using the flicker detection device 1 according to the first embodiment of the present invention.

  As shown in FIG. 7, the solid-state imaging device 10 </ b> A according to the first embodiment includes a flicker detection device 1 having the image processing unit 3 and the flicker detection unit 4 in FIG. 1, and flicker frequency information output from the flicker detection device 1. Based on the flicker information Sig13, correction is performed for each of the plurality of image signals Sig11, and each of the image signals Sig14 from which the flicker frequency components in each of the image signals Sig11 are removed is output to each output terminal 6. Means 5.

  The flicker correction unit 5 multiplies each of the plurality of image signals Sig11 by the flicker correction amount calculated based on the flicker frequency and the frame frequency from the flicker information Sig13 to remove the flicker components. In this way, by multiplying the image signal Sig11 by a waveform having a phase opposite to that of the flicker component, correction for removing the flicker component can be performed.

  FIG. 8 is a block diagram illustrating another example of the configuration example of the main part of the solid-state imaging device using the flicker detection device 1 according to the first embodiment of the present invention.

  As shown in FIG. 8, the solid-state imaging device 10B according to the first embodiment includes the flicker detection device 1 having the image processing unit 3 and the flicker detection unit 4 in FIG. 1 and the flicker information Sig13 output by the flicker detection device 1. Originally, it is provided with sensor drive control means 14 that prevents flicker components from being generated in the image signal Sig11 by setting the exposure time during which flicker does not occur in the solid-state imaging devices 11 to 13, respectively.

  As described above, according to the first embodiment, the flicker detection apparatus 1 that detects the occurrence of flicker from the image signals of the plurality of image sensors performs image processing (difference processing) on the two image signals that have been shot at the same time. The image processing unit 3 excludes image information other than the components, and the flicker detection unit 4 extracts the flicker information from the image signal after the image information exclusion process other than the flicker components.

  By taking the difference between the image signals of the two-lens imaging device (compound-eye camera) whose operation timing is shifted by a half cycle of the flicker frequency, the two image signals photographed at the same time are subjected to image processing and other than the flicker component In order to eliminate the image information, the influence of the subject and the background image is reduced, and by extracting the flicker component in the state where the flicker component is emphasized, the erroneous recognition can be greatly reduced. As a result, the flicker component is emphasized by shifting the flicker frequency by a half cycle, and the image components other than the flicker are subtracted from each other to make them smaller. Therefore, flicker noise can be detected effectively and flicker determination can be performed with high accuracy.

  In the first embodiment, the flicker detection apparatus 1 that detects the occurrence of flicker from each image signal of a plurality of image sensors has been described, but the same applies to a flicker detection method using the same.

  That is, in the flicker detection method using the flicker detection apparatus 1, the image processing means 3 performs image processing on two image signals Sig11 photographed at the same time to eliminate image information other than the flicker component, and flicker detection. The means 4 may include a flicker detection step of extracting flicker information (flicker frequency component) from the image signal Sig12 after image information other than the flicker component is excluded.

  The image processing step includes a vertical synchronization signal shift step in which the vertical synchronization signal shift means 31 shifts the vertical synchronization signal from the second image sensor with respect to the vertical synchronization signal from the first image sensor by a half cycle of the flicker frequency; The difference processing means 32 includes a difference processing step for performing difference processing on two image signals from the first image pickup element and the second image pickup element among the plurality of image pickup elements.

  The flicker detection step includes an image average value calculation step in which the image average value calculation means 41 calculates image average value information for each horizontal line or a plurality of horizontal lines in one field, and a frequency conversion means. 42 has a frequency calculation step of extracting the flicker information by frequency-converting the calculated image average value information.

  In the first embodiment, the case where at least the difference processing step is performed among the vertical synchronization signal shift step and the difference processing step has been described. However, the present invention is not limited to this, and the case where only the vertical synchronization signal shift step is performed is also flicker. The object of the present invention can be achieved, in which noise can be detected effectively and flicker determination can be performed with high accuracy. In this case, in the image processing step, the image processing means 3 may perform image processing that emphasizes the flicker frequency component.

  In the first embodiment, the image processing unit 3 shifts the vertical synchronization signal Sig22 from the second solid-state image sensor by the half cycle of the flicker frequency with respect to the vertical synchronization signal Sig21 from the first solid-state image sensor. The signal shift means 31 and the difference processing means for differentially processing the two image signals Sig11a and Sig11b from the first solid-state image sensor and the second solid-state image sensor obtained by shifting the vertical synchronization signals Sig21 and Sig22 by a half cycle of the flicker frequency. 32, at least the difference processing by the difference processing means 32 has been described. However, the present invention is not limited to this, and the image processing means 3 receives the vertical synchronization signal Sig21 from the first solid-state imaging element from the second solid-state imaging element. Only vertical synchronizing signal shift means 31 for shifting the vertical synchronizing signal Sig22 by a half cycle of the flicker frequency is provided. It can have. As described above, the image processing unit 3 includes only the difference processing unit 32 that performs the difference processing on the two image signals Sig11a and Sig11b from the first solid-state imaging device and the second solid-state imaging device among the plurality of solid-state imaging devices. It may be.

  In the image processing step of the flicker detection method in this case, the difference processing means 32 performs difference processing on the two image signals Sig11a and Sig11b from the first solid-state imaging device and the second solid-state imaging device among the plurality of solid-state imaging devices. It has a differential processing step. Further, in the image processing step, the vertical synchronization signal shift means 31 performs vertical shift in which the vertical synchronization signal Sig22 from the second solid-state image sensor is shifted by the half cycle of the flicker frequency with respect to the vertical synchronization signal Sig21 from the first solid-state image sensor. A synchronization signal shifting step.

(Embodiment 2)
In the first embodiment, the hardware configuration of the flicker detection apparatus 1 has been described. In the second embodiment, the software configuration of the flicker detection apparatus will be described.

  FIG. 9 is a block diagram illustrating a configuration example of a main part of the flicker detection apparatus according to the second embodiment of the present invention.

  In FIG. 9, the flicker detection apparatus 1A of the second embodiment is configured by a computer system, and a CPU 21 (central processing unit) as a control means for performing flicker detection control, and a temporary working memory when the CPU 21 is activated. It has a RAM 22 as storage means, and a ROM 23 as a computer-readable readable recording medium (storage means) on which flicker detection control programs for operating the CPU 21 and various data used therefor are recorded.

  The CPU 21 detects a flicker from each image signal Sig11 input from a plurality of image sensors based on a flicker detection control program read from the ROM 23 into the RAM 22 and various data used for the control program. In 1A, image processing means 3A that performs image processing on two image signals Sig11 photographed at the same time to eliminate information other than the flicker component, and flicker information (flicker frequency component) from the image signal Sig12 after information exclusion processing other than the flicker component. ) To extract the function of the flicker detection means 4A.

  The image processing means 3A includes a vertical synchronization signal shift means 31A for shifting the vertical synchronization signal from the second image sensor by a half cycle of the flicker frequency with respect to the vertical synchronization signal from the first image sensor, and among the plurality of image sensors. Difference processing means 32A for differentially processing the two image signals Sig11 installed simultaneously from the first image sensor and the second image sensor.

  The flicker detection unit 4A includes an image average value calculation unit 41A that calculates image average value information for each horizontal line or a plurality of horizontal lines in one field, and frequency conversion of the calculated image average value information. And a frequency calculating means 42A for extracting flicker information.

  The ROM 23 as a readable recording medium may be composed of a hard disk, a formable optical disk, a magneto-optical disk, a magnetic disk, an IC memory, and the like. This control program and its data are stored in the ROM, but this control program and its data may be downloaded to the ROM 23 from another readable recording medium or via wireless, wired or the Internet.

  As described above, according to the second embodiment, flicker noise can be effectively detected by suppressing the increase in the number of components such as a frame memory and the chip size, and the influence of the subject image can be reduced, and flicker determination can be performed with high accuracy. it can.

  Note that the flicker detection apparatus 1A of the second embodiment can also be applied to the solid-state imaging devices 10A and 10B, similarly to the flicker detection apparatus 1 of the first embodiment. Moreover, these solid-state imaging devices 10A or 10B can also be applied to a multi-eye imaging device 10C described later that enables imaging of each image signal acquired from two or more solid-state imaging devices.

  In the second embodiment, the flicker detection apparatus 1A that detects the occurrence of flicker from each image signal of a plurality of image sensors has been described, but the same applies to a flicker detection method using the same.

  That is, in the flicker detection method using the flicker detection apparatus 1A, the image processing unit 3A performs image processing on the two image signals Sig11 photographed at the same time to eliminate image information other than the flicker component, and flicker detection. The means 4A may include a flicker detection step of extracting flicker information (flicker frequency component) from the image signal Sig12 after image information other than the flicker component is excluded.

  The image processing step includes a vertical synchronization signal shift step in which the vertical synchronization signal shift unit 31A shifts the vertical synchronization signal from the second image sensor by a half cycle of the flicker frequency with respect to the vertical synchronization signal from the first image sensor. The difference processing means 32A includes a difference processing step for performing difference processing on two image signals from the first image pickup element and the second image pickup element among the plurality of image pickup elements.

  In the flicker detection step, the image average value calculating unit 41A calculates an image average value information for each horizontal line or a plurality of horizontal lines of one field, and a frequency converting unit. 42A has a frequency calculation step of extracting the flicker information by frequency-converting the calculated image average value information.

  In the second embodiment, the case where the vertical synchronization signal shift process and the difference processing process are performed has been described. However, the present invention is not limited to this, and the case where only the vertical synchronization signal shift process is performed or only the difference processing process is performed. Thus, the object of the present invention can be achieved, in which flicker noise can be detected effectively and flicker determination can be performed with high accuracy. In this case, in the image processing step, the image processing unit 3A may perform image processing that emphasizes the flicker frequency component.

(Embodiment 3)
FIG. 10 shows, as Embodiment 3 of the present invention, an electronic device using the solid-state imaging device 10A or 10B or the multi-eye imaging device 10C using the flicker detection device 1 or 1A of Embodiments 1 and 2 of the present invention as an imaging unit. It is a block diagram which shows the schematic structural example of an information apparatus.

  In FIG. 10, the electronic information device 90 of the third embodiment obtains a color image signal by performing signal processing on the image signal, and the solid-state imaging device 10A or 10B using the flicker detection device 1 or 1A of the first or second embodiment. Or a multi-view imaging device 10C, a memory unit 91 such as a recording medium that enables data recording after the color image signal from the 10A or 10B or multi-view imaging device is subjected to predetermined signal processing for recording, and this solid A display unit 92 such as a liquid crystal display device that can display a color image signal from the imaging device 10A or 10B or the multi-eye imaging device on a display screen such as a liquid crystal display screen after performing predetermined signal processing for display; Communication processing is possible after color signal signals from the solid-state imaging device 10A or 10B or the multi-lens imaging device are subjected to predetermined signal processing for communication. A communication unit 93 such as a receiving device, and an image output unit such as a printer that can perform print processing after performing predetermined print signal processing for color image signals from the solid-state imaging device 10A or 10B or the multi-eye imaging device for printing. 94. The electronic information device 90 is not limited to this, and in addition to the solid-state imaging device 10A or 10B or the multi-eye imaging device, an image output such as a memory unit 91, a display unit 92, a communication unit 93, and a printer. You may have at least any one of the parts 94.

  As described above, the electronic information device 90 includes, for example, a digital camera such as a digital video camera and a digital still camera, an in-vehicle camera such as a surveillance camera, a door phone camera, and an in-vehicle rear surveillance camera, and a video phone camera. An electronic device having an image input device such as an image input camera, a scanner device, a facsimile device, a camera-equipped mobile phone device, and a portable terminal device (PDA) is conceivable.

  Therefore, according to the third embodiment, based on the color image signal from the solid-state imaging device 10A or 10B or the multi-eye imaging device, this can be displayed on the display screen, or the image can be displayed on the paper. Print out (print) favorably by the output unit 94, communicate this favorably as wired communication or wirelessly as communication data, perform a predetermined data compression process in the memory unit 91, and store it satisfactorily, Various data processing can be performed satisfactorily.

  In the first and second embodiments, the flicker detection apparatus 1 or 1A includes the image processing unit 3 or 3A that performs image processing on two image signals that are simultaneously captured to eliminate image information other than the flicker component, and the flicker component. And flicker detection means 4 or 4A for extracting flicker information from the image signal after the information exclusion process. The image processing means 3 or 3A is a vertical synchronization signal shift means 31 or 31A for shifting the vertical synchronization signal from the second image sensor by a half cycle of the flicker frequency with respect to the vertical synchronization signal from the first image sensor, Although a case has been described in which at least the difference processing means 32 or 32A of the difference processing means 32 or 32A for performing differential processing on two image signals from the first image pickup element and the second image pickup element among the plurality of image pickup elements is provided. However, the present invention is not limited thereto, and the flicker detection apparatus of the present invention includes image processing means for performing image processing so as to facilitate flicker detection from image signals of a plurality of solid-state image sensors, and flicker detection for detecting flicker frequency components from the image signals. Means. In this case, the solid-state imaging device of the present invention may include the flicker detection device of the present invention, a plurality of solid-state imaging devices, and flicker correction means for removing the generated flicker by signal processing. The solid-state imaging device of the present invention includes the flicker detection device of the present invention, a plurality of solid-state imaging devices, and sensor drive control means for controlling the operation of the solid-state imaging device based on the flicker information determined by the flicker detection device. You may have. Furthermore, it may be configured by a sensor control device that controls the solid-state image sensor to operate at a timing suitable for flicker detection. Further, it may be configured by a sensor control device that controls an image output position suitable for flicker detection with respect to the solid-state imaging device. Image cutting means for cutting out an image in a range suitable for flicker detection may be provided for the image signal from the solid-state imaging device.

  As mentioned above, although this invention has been illustrated using preferable Embodiment 1-3 of this invention, this invention should not be limited and limited to this Embodiment 1-3. It is understood that the scope of the present invention should be construed only by the claims. It is understood that those skilled in the art can implement an equivalent range based on the description of the present invention and the common general technical knowledge from the description of specific preferred embodiments 1 to 3 of the present invention. Patents, patent applications, and documents cited herein should be incorporated by reference in their entirety, as if the contents themselves were specifically described herein. Understood.

  The present invention relates to a flicker detection apparatus that detects occurrence of flicker from images from a plurality of sensors corresponding to a plurality of viewpoints, a flicker detection method using the same, and solid-state imaging that can correct flicker detected by the flicker detection apparatus A multi-lens imaging device capable of imaging each image signal acquired from two or more solid-state imaging devices represented by CMOS (Complementary Metal Oxide Semiconductor) that photoelectrically converts image light from a subject Control program for causing a computer to execute a flicker detection method, a computer-readable readable recording medium on which the control program is recorded, a digital video camera and a digital still using the solid-state imaging device or the multi-eye imaging device as an imaging unit Digital camera In the field of electronic information devices such as image input cameras such as digital cameras, in-vehicle cameras, scanners, facsimiles, and camera-equipped mobile phone devices, two image signals taken simultaneously are processed to eliminate information other than the flicker component. To reduce the influence of the subject and the background image or to emphasize the flicker frequency component to suppress the increase in the number of components such as the frame memory and the chip size, In addition, flicker noise can be detected effectively and flicker determination can be performed with high accuracy.

DESCRIPTION OF SYMBOLS 1, 1A Flicker detection apparatus 2 Input terminal 3, 3A Image processing means 31, 31A Vertical synchronizing signal shift means 32, 32A Difference processing means 4, 4A Flicker detection means 41, 41A Image average value calculation means 42, 42A Frequency calculation means 5 Flicker correction means 6 Output terminals 10A, 10B Solid-state imaging device 10C Multi-eye imaging device 14 Sensor drive control means 21 CPU 21 (control means)
22 RAM
23 ROM
Sig11 input image signal Sig11a Image signal from the first solid-state image sensor Sig11b Image signal from the second solid-state image sensor Sig12 Image signal after differential processing Sig13 Flicker information Sig14 Each image signal from which the flicker component is removed Sig21, 22 Vertical synchronization signal 90 Electronic information equipment 91 Memory unit 92 Display unit 93 Communication unit 94 Image output unit

Claims (20)

A flicker detection device that detects the occurrence of flicker from each image signal of a plurality of image sensors,
Image processing means for performing image processing on two simultaneously captured image signals to eliminate image information other than the flicker component and at least one of enhancing the flicker frequency component;
A flicker detection apparatus comprising flicker detection means for extracting flicker information from an image signal after image information exclusion processing other than the flicker component.   The flicker detection apparatus according to claim 1, wherein the image processing unit includes a difference processing unit that performs difference processing on image information of two screens from the first imaging element and the second imaging element among the plurality of imaging elements.   The said image processing means has a vertical-synchronization-signal shift means which shifts the vertical-synchronization signal from the said 2nd image pick-up element only by the half period of a flicker frequency with respect to the vertical-synchronization signal from the said 1st image pick-up element. Flicker detection device.   The image processing means includes a vertical synchronization signal shift means for shifting the vertical synchronization signal from the second image sensor by a half cycle of the flicker frequency with respect to the vertical synchronization signal from the first image sensor, and the vertical synchronization signal. The flicker detection apparatus according to claim 1, further comprising difference processing means for performing difference processing on the image information of two screens from the first imaging element and the second imaging element that are shifted by a half cycle of the flicker frequency. The flicker detection means includes
The flicker detection apparatus according to claim 1, further comprising image average value calculation means for calculating image average value information for each horizontal line or a plurality of horizontal lines in one field of the image signal after the difference processing. The flicker detection means includes
6. The flicker detection apparatus according to claim 5, further comprising frequency conversion means for converting the calculated image average value information to extract a flicker frequency, and extracting the flicker frequency as the flicker information.   The flicker detection apparatus according to claim 6, wherein the frequency conversion unit detects flicker information of a flicker frequency by performing discrete Fourier transform on the calculated image average value information.   Based on the flicker detection device according to any one of claims 1 to 7 and flicker information output from the flicker detection device, each of the image signals of the plurality of image sensors is corrected, A solid-state imaging device having flicker correction means for outputting each image signal from which a flicker frequency component in the image signal is removed.   The flicker detection device according to any one of claims 1 to 7, and sensor drive control means for setting an exposure time during which no flicker occurs to each of the plurality of image pickup devices based on the flicker information output from the flicker detection device. A solid-state imaging device.   A multi-lens imaging device capable of imaging each image signal acquired from two or more imaging elements of the solid-state imaging device according to claim 8 or 9.   An electronic information device using the solid-state imaging device according to claim 8 or 9 or the multi-eye imaging device according to claim 10 as an image input device in an imaging unit. A flicker detection method for detecting occurrence of flicker from each image signal of a plurality of image sensors,
An image processing step in which the image processing means performs at least one of image processing of two image signals captured simultaneously to eliminate image information other than the flicker component and enhancement of the flicker frequency component;
A flicker detection method, wherein the flicker detection means includes a flicker detection step of extracting flicker information from an image signal after image information exclusion processing other than the flicker component.   13. The image processing step according to claim 12, wherein the image processing step includes a difference processing step in which difference processing means performs difference processing on image information of two screens from the first image pickup device and the second image pickup device of the plurality of image pickup devices. Flicker detection method.   In the image processing step, the vertical synchronization signal shift unit shifts the vertical synchronization signal from the second image sensor by a half cycle of the flicker frequency with respect to the vertical synchronization signal from the first image sensor. The flicker detection method according to claim 12, comprising:   In the image processing step, the vertical synchronization signal shift unit shifts the vertical synchronization signal from the second image sensor by a half cycle of the flicker frequency with respect to the vertical synchronization signal from the first image sensor. And a difference processing step, wherein the difference processing means performs a difference process on the image information of the two screens from the first image sensor and the second image sensor in which the vertical synchronization signals are shifted from each other by a half cycle of the flicker frequency. 12. The flicker detection method according to 12. The flicker detection step includes
The image average value calculating means has an image average value calculating step of calculating image average value information for each horizontal line or a plurality of horizontal lines of one field of the image signal after the difference processing, and based on the image average value information The flicker detection method according to claim 12, wherein flicker information is extracted. The flicker detection step includes
The flicker detection method according to claim 16, further comprising a frequency calculation step in which frequency calculation means performs frequency conversion on the calculated image average value information to extract the flicker information.   The flicker detection method according to claim 17, wherein the frequency calculation step detects flicker information of a flicker frequency by performing discrete Fourier transform on the calculated image average value information.   A control program for causing a computer to execute each step of the flicker detection method according to claim 12.   A computer-readable readable recording medium on which the control program according to claim 19 is recorded.