JP6576028B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing technique for an image photographed under a light source in which a periodic light amount change occurs such as under a fluorescent lamp.

  When a subject is recorded as a moving image under a light source that generates flicker, such as under a fluorescent lamp, by an imaging device using an XY address type solid-state imaging device such as a CMOS image sensor, a light and dark stripe pattern may be recorded in the image. In order to suppress such deterioration in image quality, Patent Document 1 proposes a technique for detecting a blinking component of a light source from a periodic luminance change on an image and canceling the blinking component.

JP 2004-222228 A

  In the technique described in Patent Document 1, a stripe pattern in an image is approximated by a sine wave as a gain of a pixel value for each line and detected and corrected.

  However, the deviation from the actual flickering characteristic and the ideal characteristic of the sine wave differs depending on the type of illumination, especially in white fluorescent lamps, etc. The degree of deviation from is different. For this reason, it is difficult to perform correction suitable for the actual blinking characteristic by the technique described in Patent Document 1. Furthermore, when correction is performed using a correction value that is not suitable for the actual flickering characteristics, there is a problem in that a specific color component is partially corrected and overcorrected, and coloring is observed in the image.

  Therefore, an object of the present invention is to make it possible to correct a striped pattern generated in an image taken under a light source in which a periodic light quantity change occurs while suppressing coloring.

In order to achieve the above object, an image processing apparatus according to the present invention is based on an extraction unit that extracts a flicker component for each color component of an input image, and a flicker component for each color component extracted by the extraction unit. Correction value calculation means for calculating a correction value for flicker correction for each color component of the input image, correction means for performing flicker correction for each color component of the input image, and the extracted Reliability calculation means for calculating the reliability of the correction value for each color component calculated by the correction value calculation means based on the flicker component for each color component and the shutter speed at the time of acquiring the input image When have said correction means, said correction value of the reliability calculated by the reliability calculation unit for the color component below a predetermined threshold value, the reliability of the correction value of the predetermined threshold value Than the color component not less than characterized by weakening the correction degree of the flicker correction.

  ADVANTAGE OF THE INVENTION According to this invention, the striped pattern which arises in the image image | photographed under the light source which a periodic light quantity change produces can be corrected, suppressing coloring.

It is a block diagram which shows the structure of the image processing apparatus concerning this embodiment. It is a block diagram explaining the detail of the digital signal processing circuit of the image processing apparatus concerning this embodiment. It is a figure which shows the image which image | photographed under the light source which a periodic light quantity change produces, and the stripe pattern has produced. It is a figure which shows the flowchart of the correction value correction process in the correction value correction part. It is a figure which shows the flicker component of G component of the image image | photographed under the white fluorescent lamp, and B component. It is a figure which shows distribution on the color space of the predetermined | prescribed row | line | column of the some image obtained by image | photographing the object of a predetermined | prescribed color continuously under a white fluorescent lamp.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating an example of a schematic configuration of a camera 100 that is an image processing apparatus according to an embodiment of the present invention. The camera 100 includes, for example, a digital still camera, a digital video camera, a smartphone, a tablet terminal, and the like.

  In FIG. 1, reference numeral 1 denotes a first lens group located on the subject side with respect to the diaphragm 4, and 10 denotes a second lens group disposed on the imaging element 16 side (the main body side of the imaging apparatus) with respect to the diaphragm 4. Yes. That is, light before passing through the aperture of the diaphragm 4 is incident on the first lens group 1, and light after passing through the aperture of the diaphragm 4 is incident on the second lens group. The first lens group 1 and the second lens group 10 are controlled back and forth on the lens optical axis for zoom control. Although FIG. 1 shows a single lens in a simplified manner, the first lens group 1 and the second lens group 10 are usually composed of a plurality of lenses. The first lens group 1 and the second lens group 10 may have functions of focus control and image stabilization control in addition to zoom control. The first lens group 1 and the second lens group 10 may be composed of a single lens. In the present embodiment, the first lens group 1 and the second lens group 10 are referred to as a lens group including a case where the first lens group 1 and the second lens group 10 are composed of one lens.

  The lens drive motor 2 drives the first lens group 1 back and forth on the optical axis in accordance with the drive power supplied from the zoom drive unit 13. Similarly, the lens drive motor 11 also drives the second lens group 10 back and forth on the optical axis in accordance with the drive power supplied from the zoom drive unit 13. The lens state detection circuit 3 detects the driving state of the first lens group 1 and outputs the detection result to the microcomputer 20. Similarly, the lens state detection circuit 12 also detects the driving state of the second lens group 10 and outputs the detection result to the microcomputer 20.

  The diaphragm 4 is a diaphragm blade that adjusts the amount of light incident from the outside of the camera 100. The aperture drive motor 5 drives the aperture 4 according to the drive power supplied from the aperture drive unit 14. The diaphragm state detection circuit 6 detects the driving state of the diaphragm 4 and outputs the detection result to the microcomputer 20. When the focal length changes due to the movement of the first lens group 1 and the second lens group 10, the aperture 4 is controlled by the aperture drive motor 5 and the aperture state detection circuit 6 so that the aperture diameter calculated from the microcomputer 20 is obtained. Control.

  An ND (Neutral Density) filter 7 is configured to be able to be inserted into and retracted from an incident optical path to the image sensor 16, and reduces light incident from the first lens group 1 by being inserted. The ND filter drive motor 8 drives the ND filter 7 according to the drive power supplied by the ND filter drive unit 15. The ND filter drive detection circuit 9 detects the drive state of the ND filter 7 and outputs the detection result to the microcomputer 20. As will be described later, when the diaphragm 4 is corrected and the aperture diameter changes, the microcomputer 20 calculates the position of the ND filter 7 from the output detection result, and the ND filter drive motor 8 and the ND filter drive detection circuit. 9 controls the ND filter 7. In the present embodiment, an ND filter whose transmittance or density changes in a gradation shape will be described as an example of the ND filter 7. However, a single density filter may be used. Further, in FIG. 1, a single ND filter is illustrated in a simplified manner, but the ND filter 7 may be composed of a plurality of sheets.

  Reference numeral 16 denotes an XY address type solid-state imaging device such as a CMOS image sensor, which photoelectrically converts an optical image of an object incident through the first lens group 1 to the second lens group 10 and accumulates the obtained charges. To do. The electric charge (image signal) stored in each pixel of the image sensor 16 is output to the CDS / AGC circuit 17 in accordance with a drive pulse of the image sensor drive unit 27 described later. The CDS / AGC circuit 17 samples and amplifies the output image signal. Note that correlated double sampling (CDS) is performed for sampling, and automatic gain control (AGC: Auto Gain Control) is performed for amplification. The image signal (analog signal) processed by the CDS / AGC circuit 17 is converted into a digital image signal by the A / D converter 18. The digital signal processing circuit 19 performs various signal processing on the digital image signal output from the A / D converter 18.

  The recording unit 25 writes the image signal subjected to the signal processing by the digital signal processing circuit 19 to an external memory such as a memory card.

  The microcomputer 20 is a circuit called a microcontroller or simply a controller, and comprehensively controls the operation of the camera 100. For example, the microcomputer 20 receives various information such as luminance and color from the digital signal processing circuit 19 and performs various arithmetic processes.

  The aperture drive unit 14 supplies drive power to the aperture drive motor 5 based on control by the microcomputer 20. For example, the diaphragm drive unit 14 is a drive for narrowing or opening the diaphragm 4 by control based on the exposure value obtained by the microcomputer 20 in accordance with the photometric value (luminance value) of the image captured by the image sensor 16. Supply power. Thereby, in the camera 100, it is possible to perform aperture adjustment so that an appropriate amount of light enters the image sensor 16. A photometric sensor may be provided separately, and the diaphragm 4 may be controlled using a photometric value obtained from the photometric sensor.

  The ND filter drive unit 15 supplies drive power to the ND filter drive motor 8 based on control by the microcomputer 20. For example, the ND filter driving unit 15 changes the insertion position of the ND filter 7 by control based on the exposure value obtained by the microcomputer 20 according to the photometric value (luminance value) of the image captured by the image sensor 16. Supply drive power.

  As a result, the imaging apparatus can adjust the amount of light incident on the imaging device 16 in accordance with the photometric value of the captured image.

  The image sensor driving unit 27 supplies drive pulses and the like for driving the image sensor 16 to the image sensor 16 based on the control by the microcomputer 20, and reads the image captured by the image sensor 16 and adjusts the exposure time. . For example, the image sensor drive unit 27 supplies drive pulses for performing exposure of the image sensor 16 with a predetermined exposure time under the control of the microcomputer 20 in accordance with the photometric value of the image captured by the image sensor 16. As a result, the camera 100 can adjust the exposure time of the image sensor 16 according to the photometric value of the captured image and read the image.

  The memory 22 is a RAM (Random Access Memory) or the like, and temporarily stores data. For example, the memory 22 temporarily stores image data after being captured by the image sensor 16 and processed by the digital signal processing circuit 19. A program for driving the camera 100 is also stored in the memory 22 and is sequentially called and executed by the microcomputer 20. In addition, when using the correction positions of the diaphragm 4 and the ND filter 7 by simulation in advance, correction position data is stored.

  The external key 23 is used to change the zoom magnification of the camera 100 and make various settings. At this time, an icon or a message is displayed on the display device 24 to inform the user of the operation of the external key.

  Next, details of the digital signal processing circuit 19 will be described with reference to FIG.

  An image signal to be processed by the digital signal processing circuit 19 (digital image signal output from the A / D converter 18) is input in parallel to the evaluation value acquisition unit 100 and the flicker correction unit 103. Here, the image signal input to the digital signal processing circuit 19 may be a luminance / color difference signal (YCC) that can be input by a general television, or may be an RGB image signal obtained by outputting RGB images in chronological order.

  The evaluation value acquisition unit 100 combines the input image signals, acquires RGB color signal values as integration values for each row of the image, and outputs them to the correction value prediction unit 101. The correction value prediction unit 101 extracts a periodic signal level change component (hereinafter referred to as a flicker component) that occurs in the column direction in the image from the input color signal integration value for each row, and cancels the flicker component. Correction parameters are calculated and output to the correction value correction unit 102.

<Calculation of flicker component correction value>
When taking an image using an XY addressing type image pickup device such as a CMOS image sensor, an image with a bright and dark stripe pattern in the image column direction (vertical direction) may be obtained as shown in FIG. This occurs because the acquisition cycle (frame rate) of an image shot under a light source in which a periodic light amount change occurs does not match the light amount change cycle of the light source in the shooting environment. The correction value prediction unit 101 extracts a stationary signal component in the time direction according to the following equation in order to extract a flicker component from a bright and dark stripe pattern generated in the image.
mem = ave × k + mout × (1-k)

  mem is a value stored in the memory as an output of the above equation, and ave is a signal value of the color component of each row in the image input from the evaluation value acquisition unit 100. k is a filter coefficient of the cyclic low-pass filter, and mout is a calculation result of the above expression calculated when the signal value of the image of the previous frame is input (that is, the above expression is calculated for the previously input image). Mem). By performing the above calculation for each row with each color component, it is possible to extract a stationary signal component in the time direction. The correction value prediction unit 101 calculates a flicker component by dividing the extracted stationary signal component by the signal level of each color component by the signal level input from the evaluation value acquisition unit 100 in the current frame. . Further, the correction value prediction unit 101 generates a flicker model that is a fluctuation characteristic of the level of the signal in the vertical direction from the calculated flicker component. That is, the flicker model represents the fluctuation characteristics of the flicker component when the flicker components calculated for each row of the image are arranged in the column direction. For example, the flicker model is a sine wave having amplitude w, frequency f, and phase θ in the vertical direction. Can be approximated. The frequency f is calculated based on the frame rate and the light amount change period of the light source, and the phase θ of each row is the phase of the row taking the intermediate value between the adjacent minimum value and maximum value among the flicker components calculated for each row. = 0. The amplitude W is calculated from the difference between the flicker component of the row whose phase is π / 2 or 3π / 2 and the flicker component of the row whose phase is 0. The generated flicker model is generated for each color component, and the correction value predicting unit 101 corrects the phase, amplitude, and frequency of a sine wave whose phase is shifted by π from the calculated flicker model as a correction value for canceling the color component. To 102.

  The correction value correction process in the correction value correction unit 102 will be described with reference to FIG.

  In step S1, the correction value correcting unit 102 calculates the reliability of the correction value of each color component. As the shutter speed increases, the difference between the ideal sine wave and the actual flicker model increases. Further, depending on the light source, a difference in the amplitude of the flicker component occurs between the color components, and the correction component of the color component having a large amplitude also increases. Therefore, it can be considered that the higher the shutter speed, the smaller the reliability of the correction value. Therefore, in this embodiment, the shutter speed is acquired from the meta information added to the input image, and the reliability of each color component is calculated from the amplitude value and the shutter speed included in the correction value. In the present embodiment, the shutter speed and the amplitude value are used to calculate the reliability. However, other methods may be used for the reliability calculation. Furthermore, although the meta information added to the image is given as an example of a method for acquiring the shutter speed, other methods such as acquiring the shutter speed from the microcomputer 20 may be used.

  FIG. 5 shows the signal level of the flicker component at the time of high-speed shutter under a white fluorescent lamp and the signal level after correction. FIG. 5A shows the G component flicker component with the column direction on the horizontal axis and the signal level on the vertical axis. FIG. 5B shows the column direction on the horizontal axis and the signal level on the vertical axis. B component flicker component is shown.

  In a white fluorescent lamp or the like, the amplitude of the B component is larger than that of the other color components, and the deviation from the ideal characteristic of the B component is greater than that of the other color components. For this reason, as shown in FIG. 5B, only the B component remains uncorrected, the RGB balance is lost, and coloring occurs.

  By the way, according to the analysis of human visual characteristics from the viewpoint of bioinformatics, it is known that the perceptual performance is limited when the color of the image is switched at high speed, and the perceptual performance is also different depending on the color components. The response speeds of the L cone and M cone that mainly perceive the R component and the G component have perceptual performance in the time direction of 60 Hz or more, while the perception performance of the S cone that perceives the B component is less than 50 Hz. is there. A color signal that switches at a frequency higher than the perceptual performance is perceived as a signal smoothed in the time direction.

  Considering the correction of the fringe pattern generated in the image with respect to the image signal obtained at the frame rate of 60 fps (hereinafter referred to as flicker correction), the remaining correction occurs in the B component as shown in FIG. When the flicker component generated when an image is obtained at 60 fps under a light source in which a light amount change of 100 Hz occurs is corrected without considering the characteristics of the B component, the remaining correction state circulates in about 3 frames. Therefore, when a certain area in the image is watched, two of the three frames are corrected to some good extent, and the remaining correction occurs in one frame. Since the B component of 2 frames corrected well continues as a stable signal level, when the image is displayed at 60 fps, the same color continues for a period of 1/60 (second) × 2 = 1/30 (second). Become. This period of 1/30 (second) is an update cycle within the human perceptual performance range. On the other hand, one frame that remains corrected continues for 1/60 (second). This period of 1/60 (second) is an update cycle outside the human perceptual performance range. When an image is displayed with an update period outside the perceptual performance range, it is perceived as a signal smoothed in the time direction by humans, so as a signal value between the image with the remaining correction and the image that has been well corrected Perceived. For this reason, if a B component image that has been successfully corrected and an image that remains to be corrected are mixed in the above-described cycle, the color signal is perceived as being biased in a certain direction when it is perceived after being smoothed in the time direction. . Therefore, when the reliability of flicker correction is low and the occurrence of a remaining correction is expected, increasing the remaining B component correction increases the color deviation of each image, but smoothes it in the time direction. The bias in the case can be reduced.

  FIG. 6 shows a distribution in the color space of representative color signals of a predetermined row of a plurality of images obtained by continuously photographing a subject of a predetermined color under a white fluorescent lamp. The representative color signal here is obtained by a predetermined known calculation method from the R component, G component, and B component of a predetermined row. FIG. 6A shows the distribution when the B component correction is normally performed on a plurality of images, and FIG. 6B shows the distribution on the vector space when the B component correction is weakened. The distribution of is shown. When the correction of the B component is performed as usual, as shown in FIG. 6A, the distribution is concentrated in a part of the area surrounded by a circle, so that there is a large bias when smoothing in the time direction. Become. On the other hand, when the correction of the B component is weakened, as shown in FIG. 6B, since the distribution is spread over a wide range surrounded by a circle, the bias when smoothing in the time direction is reduced. That is, when the correction of the B component is weakened rather than the correction of the B component as usual, coloring in the case of smoothing in the time direction can be suppressed.

  Note that the perceptual characteristics only affect the image update cycle, and some frame rates are not limited to the B component and are not perceived. If the viewing frame rate is 50 fps or more and 100 fps or less, the correction value is corrected only for the B component depending on the reliability of the color signal. If the viewing frame rate is 100 fps or more, the correction is made according to the reliability of each RGB component. It is possible to improve coloring by correcting the value.

  Therefore, in step S2 of FIG. 4, the correction value correction unit 102 determines whether or not the frame rate as the image update cycle is 100 Hz or more (whether or not the update cycle is longer than a predetermined cycle). Here, the information related to the frame rate of the image signal may use information added to the meta information of the image signal, may use a method obtained from the microcomputer 20, or may be other methods.

  If the frame rate is 100 Hz or higher, in step S3, the correction value correcting unit 102 determines whether or not the reliability of each color component is equal to or lower than a predetermined threshold. If there is a color component whose reliability is below the threshold, the correction value correction unit 102 corrects the correction value that suppresses the correction degree for the color component whose reliability is below the predetermined threshold in step S4. Here, the correction degree for the color component whose reliability is not lower than the predetermined threshold is not corrected. That is, the correction degree for the color component whose reliability is less than the predetermined threshold is made weaker than the correction degree for the color component whose reliability is not less than the predetermined threshold.

  If the frame rate is 100 fps or less, in step S5, the correction value correcting unit 102 determines whether or not the frame rate is 50 fps or more.

  If it is 50 fps or more, in step S6, the correction value correcting unit 102 determines whether or not the reliability of the B component is equal to or less than a predetermined threshold. If the reliability of the B component is below a predetermined threshold value, the correction value that suppresses the correction degree for the B component is corrected. Here, the correction degree for the R component and the G component is not corrected.

  After performing the correction value correction process for each color component as described above, the correction value correction unit 102 outputs the correction value for each color component to the flicker correction unit 103.

  The flicker correction unit 103 performs a correction process by controlling the gain for the signal for each row and each color component of the image to be corrected according to the input correction value, and outputs a flicker-corrected image.

  As described above, in the present embodiment, the periodic light quantity change is caused by making the correction degree weaker than the color component whose reliability is not lower than the predetermined threshold for the color component whose reliability is lower than the threshold. It is possible to correct a striped pattern generated in an image taken under a generated light source while suppressing coloring. Further, by making the correction degree for the B component weaker than for the other color components, it is possible to correct a striped pattern generated in an image taken under a light source in which a periodic light quantity change is suppressed while suppressing coloring.

  In the above-described embodiment, reducing the correction degree may include not performing correction.

  In the above embodiment, the reliability is determined. However, the correction degree for the B component may be made weaker than the correction degrees for the R component and the G component without determining the reliability. .

  In the above embodiment, an example of an image processing apparatus having an imaging function has been described. However, an image processing apparatus that does not have an imaging function may be used. In the case of an image processing apparatus that does not have an imaging function, the above correction process may be performed on an image input from an external device. Note that in the image processing apparatus having an imaging function, the above correction processing may be performed on an image input from an external device.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Reference Signs List 16 Image Sensor 19 Digital Signal Processing Circuit 20 Microcomputer 100 Evaluation Value Acquisition Unit 101 Correction Value Prediction Unit 102 Correction Value Correction Unit 103 Flicker Correction Unit

Claims (7)

Extraction means for extracting flicker components for each color component of the input image;
Correction value calculation means for calculating a correction value for flicker correction for each color component of the input image based on the flicker component for each color component extracted by the extraction means; and color of the input image Correction means for performing flicker correction for each component;
The reliability of the correction value for each color component calculated by the correction value calculation unit is calculated based on the flicker component for each extracted color component and the shutter speed when the input image is acquired. A reliability calculation means,
For the color component for which the reliability of the correction value calculated by the reliability calculation unit is less than a predetermined threshold, the correction unit is more than the color component for which the reliability of the correction value is not less than the predetermined threshold. An image processing apparatus characterized by weakening the degree of flicker correction.   The correction unit is configured to apply the input image and a predetermined frame rate to a color component for which the reliability of the correction value calculated by the reliability calculation unit with respect to the input image is lower than the predetermined threshold. The flicker correction correction degree of an image obtained continuously from the input image is weaker than a color component whose reliability of the correction value does not fall below the predetermined threshold. Image processing apparatus.   When the predetermined frame rate is equal to or higher than a predetermined value, the correction unit determines the correction value of the correction value for a color component whose reliability calculated by the reliability calculation unit is lower than the predetermined threshold. The image processing apparatus according to claim 2, wherein the degree of flicker correction is weaker than that of a color component whose reliability does not fall below the predetermined threshold.   The reliability calculation means calculates the reliability of the correction value for each color component calculated by the correction value calculation means based on the flicker amplitude and shutter speed for each of the extracted color components. The image processing apparatus according to any one of claims 1 to 3. Extraction means for extracting a flicker component for each of the R component, G component, and B component of an image obtained at a predetermined frame rate;
Calculation means for calculating a correction value for flicker correction for each color component of the image based on the flicker component for each color component extracted by the extraction means;
Correction means for performing flicker correction for each of the R component, G component, and B component of the image, and the correction means uses the B component of the image when the predetermined frame rate is a predetermined value or more. On the other hand, an image processing apparatus characterized in that the degree of flicker correction is weaker than the G component and R component of the image. An extraction step of extracting a flicker component for each color component of the input image;
A correction value calculating step for calculating a correction value for flicker correction for each color component of the input image based on the flicker component for each color component extracted in the extraction step;
A correction step for performing flicker correction for each color component of the input image;
The reliability of the correction value for each color component calculated in the correction value calculation step is calculated based on the flicker component for each extracted color component and the shutter speed at the time of acquiring the input image. A reliability calculation step, and
In the correction step, for the color component in which the reliability of the correction value calculated in the reliability calculation step is less than a predetermined threshold, the reliability of the correction value is less than the color component in which the reliability is not lower than the predetermined threshold. An image processing method characterized by weakening the degree of flicker correction. An extraction step of extracting a flicker component for each of the R component, G component, and B component of an image obtained at a predetermined frame rate;
A calculation step for calculating a correction value for flicker correction for each color component of the image based on the flicker component for each color component extracted in the extraction step;
A correction step of performing flicker correction for each of the R component, G component, and B component of the image,
In the correction step, when the predetermined frame rate is equal to or higher than a predetermined value, the correction degree of flicker correction is weakened with respect to the B component of the image as compared with the G component and the R component. Processing method.