JP2018085637A - Imaging apparatus and control method therefor, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus and control method therefor, and program, capable of appropriately performing flash band detection when electronic vibration isolation is performed.SOLUTION: A flash band over two frames generated by external flash light is detected from a plurality of frames of a video signal obtained from an image signal read out from an imaging element by a rolling shutter system and when an image of the detected frame is corrected into a whole area flash image, a detection method of the flash band is changed according to each blur amount of a plurality of frames corrected by electronic vibration isolation.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an imaging apparatus, a control method thereof, and a program, and more particularly to an imaging apparatus that performs electronic image stabilization and flash band detection, a control method thereof, and a program thereof.

  Conventionally, in an image pickup apparatus that reads an image signal from an image pickup element by a rolling shutter method, there is a problem of a flash band in which a luminance step is generated in an image in one frame due to an external flash that is instantaneously emitted. As a method for correcting this, in a frame in which a flash band is detected, there is no luminance step by using an image of the previous frame instead of that frame, or by averaging a plurality of previous and subsequent frames. A method of creating a painting is known.

  On the other hand, as a flash band detection method, first, an integral value of luminance of each line is calculated. Next, a conventional example in which it is determined that a flash band has occurred when the total from the line transitioning from low luminance to high luminance to the line transitioning from high luminance to low luminance in the next frame is approximately one frame. Is known (Patent Document 1).

JP 2009-253367 A

  However, when the flash band is detected by the above-described conventional method after electronic image stabilization, the electronic image stabilization amount may differ between two consecutive frames. In this case, the total of the lines where the flash band is generated does not become one frame, and the flash band cannot be detected. Also, the flash band detection frame is not set in the intended area after electronic image stabilization, and there is a problem that the detection accuracy is lowered.

  Therefore, the present invention is to provide an imaging apparatus capable of appropriately performing flash band detection even when electronic image stabilization is performed, a control method thereof, and a program.

  The image pickup apparatus of the present invention acquires an image signal read from the image pickup element by a rolling shutter method, converts the image signal into a video signal composed of a plurality of frames, and among the plurality of frames generated by the external flash A flash band detecting means for detecting a flash band extending over the two, a flash band correcting means for correcting an image of a frame in which the flash band is detected to a full flash image, and a plurality of frames by electronic image stabilization. The image processing apparatus includes: an anti-vibration unit that corrects each shake; and a change unit that changes a detection method of the flash band according to the shake amount corrected by the anti-vibration unit.

  According to the present invention, it is possible to appropriately detect a flash band even when electronic image stabilization is performed.

It is a block diagram of the video camera as an imaging device concerning the present invention. It is a flowchart which shows the procedure of the image correction process after imaging | photography in the 1st Embodiment of this invention. It is a figure explaining the method of the electronic image stabilization in the image stabilization part of FIG. 1 in step S101 of FIG. It is a figure for demonstrating the flash band detection method when the electronic vibration isolation by a vibration isolator is not performed. It is a figure for demonstrating the flash band detection method in case the electronic image stabilization by an image stabilization part is performed. It is a flowchart which shows the procedure of the image correction process after imaging | photography in the 2nd Embodiment of this invention. It is a figure for demonstrating the change method of the flash band detection frame of step S202 of FIG.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

  First, the configuration of a video camera as an imaging apparatus according to the present invention will be described with reference to FIG. Thereafter, first and second embodiments will be described with reference to FIGS.

  FIG. 1 is a block diagram of a video camera 1 as an imaging apparatus according to the present invention.

  In FIG. 1, reference numeral 101 denotes a photographing optical system composed of a plurality of optical lenses that form an optical image of a subject, and a variable magnification lens (zoom lens) that moves in the optical axis direction upon zooming, and at the time of focus adjustment (focusing). A focus lens that moves in the optical axis direction. In addition, the photographing optical system 101 includes an aperture and ND that control the amount of incident light.

  Reference numeral 102 denotes a CMOS sensor having an image sensor that photoelectrically converts an optical image formed by the photographing optical system 101 into an image signal in a two-dimensional array. The CMOS sensor 102 performs exposure by a rolling shutter method and reads an image signal. Reference numeral 103 denotes a video signal processing unit that acquires an image signal read from the CMOS sensor 102 and converts it into a video signal composed of a plurality of frames in a predetermined format.

  Reference numeral 104 denotes an overall control calculation unit for controlling the entire video camera 1. The overall control calculation unit 104 calculates an appropriate exposure amount from the output signal of the video signal processing unit 103, the aperture and ND in the photographing optical system 101, the shutter speed in the CMOS sensor 102, and the video signal processing unit. The value of gain applied at 103 is controlled. The shutter speed here is the exposure time of the electronic shutter, and is controlled by the overall control calculation unit 104 sending the charge accumulation start timing and the charge read timing to the CMOS sensor 102. In addition, the overall control calculation unit 104 controls a flash band detection unit 109, a flash band correction unit 110, and an image stabilization unit 111, which will be described later, and executes image correction processing after shooting. This process will be described later with reference to FIG.

  A memory unit 105 temporarily stores calculation results of the overall control calculation unit 104 and output signals of the video signal processing unit 103, such as the changed number of reference lines. Reference numeral 106 denotes a recording medium such as a DVD, a hard disk, or a nonvolatile memory. A recording medium control I / F unit 107 controls data input / output with the recording medium 106. Reference numeral 108 denotes an external I / F unit, which can output a captured video to an external monitor or recorder, and can input a video from another video camera or player. Alternatively, the video camera 1 can be connected to a computer via the external I / F unit 108 and can acquire necessary information via the computer and the Internet. Reference numeral 109 denotes a flash band detection unit, which detects an area where a flash band is generated by external flash using image information of two consecutive frames stored in the memory unit 105. A flash band correction unit 110 performs flash band correction for correcting a full flash image by combining the frame in which the flash band stored in the memory unit 105 is detected and the image of the immediately preceding frame. Reference numeral 111 denotes a vibration isolating unit that calculates electronic camera shake based on gyro information and shifts the image in units of pixels in the vertical and horizontal directions for each frame so as to reduce the shake. The video signal from the CMOS sensor 102 is processed by the video signal processing unit 103 and then input to the flash band detection unit 109 through the image stabilization unit 111.

<First Embodiment>
In the first embodiment, an example is shown in which the number of lines used for determination at the time of flash band detection (hereinafter referred to as “reference line number”) is changed in accordance with the image stabilization amount.

  FIG. 2 is a flowchart showing a procedure of image correction processing after photographing in the first embodiment of the present invention. This process is executed by the overall control calculation unit 104.

  In step S101, the image stabilization unit 111 performs electronic image stabilization on the input image by shifting the image in the horizontal and vertical directions by the amount of shake detected by the gyro. Details will be described later with reference to FIG. When the process of step S101 is completed, the process proceeds to step S102.

  In step S102, the reference line number for determination in the flash band detection is changed according to the image stabilization correction amount in the image stabilization unit 111. Details will be described later with reference to FIGS. When the process of step S102 is completed, the process proceeds to step S103.

  In step S103, the flash band detection unit 109 performs flash band detection on the video signal from the image stabilization unit 111. At this time, flash band detection is performed using the reference line number determined in step S102. When the process is completed, the process proceeds to step S104.

  In step S104, the flash band detection unit 109 determines whether a flash band has been detected. If a flash band is detected, the process proceeds to step S105, and if not, the process of FIG.

  In step S105, the flash band correction unit 110 performs flash band correction, and then ends the process of FIG. Since the flash band correction in step S105 is performed by a known method, the description thereof is omitted.

  FIG. 3 is a diagram for explaining a method of electronic image stabilization in the image stabilization unit 111 in FIG. 1 in step S101 in FIG.

  As shown in the upper part of FIG. 3, there is a stationary subject at the center of the image before the image stabilization correction, and due to the camera shake of the photographer, the light from the video camera 1 moves in the lower right direction in the second frame and in the upper left direction in the third frame. It shows how the axis is blurred. Further, a flash band is generated in the lower part of the image in the second frame and in the upper part of the image in the third frame.

  On the other hand, these images are as shown in the lower part of FIG.

  Since no blur is detected for the first frame image, the input image is output as it is without correction.

  In the image of the second frame, blurring of H2 pixels in the horizontal direction and V2 pixels in the vertical direction occurs. Therefore, the address (X, Y) of the input pixel data is output as data of the address (X + H2, Y + V2) after image stabilization correction. Since the peripheral portion of the image is corrected using the surplus pixels outside the effective area, the peripheral portion of the image is not lost even after the image stabilization synthesis. By doing so, the subject is moved to the center of the image and the start line of the flash band is moved by V2 pixels as the image after the image stabilization.

  In the image of the third frame, blurring of H3 pixels in the horizontal direction and V3 pixels in the vertical direction occurs. At this time, the address (X, Y) of the input pixel data is output as data of the address (X + H3, Y + V3) after image stabilization. As in the second frame, the peripheral portion of the image is corrected by using the surplus pixels outside the effective area, so that the peripheral portion of the image is not lost even after the image stabilization synthesis. By doing so, the subject is moved to the center of the image and the end line of the flash band is moved by V3 pixels as the image after the image stabilization.

  Since no blur is detected for the fourth frame image, the input image is output as it is without correction.

  FIG. 4 is a diagram for explaining a flash band detection method when electronic image stabilization by the image stabilization unit 111 is not performed. This method is a conventional flash band detection method.

  FIG. 4 shows an example in which the subject is at the center and flash bands are generated in the lower part of the second frame and the upper part of the third frame. FIG. 4 shows an example in which the optical axis of the video camera 1 is not shaken with respect to the subject.

  First, as shown in FIGS. 4A, 4 </ b> B, and 4 </ b> C, the average value (average level) of the luminance level is calculated for each line for the images of the first frame, the second frame, and the third frame. To do.

  Next, as shown in FIG. 4D, the average level difference of each line is calculated for the second frame and the first frame (hereinafter referred to as a first level difference). Here, of the first level difference, N is the number of lines higher than the threshold.

  Further, as shown in FIG. 4E, the average level difference of each line is calculated for the third frame and the first frame (hereinafter referred to as a second level difference). Here, of the second level difference, M is the number of lines higher than the threshold.

  Here, if the reference line number for determination in flash band detection is the total number of lines for one frame, and N + M corresponds to the total number of lines for one frame, it is determined that a flash band has been detected. This reference line number is held in the memory unit 105 as a default value.

  FIG. 5 is a diagram for explaining a flash band detection method when electronic image stabilization is performed by the image stabilization unit 111.

  FIG. 5 shows an example in which the subject is at the center and flash bands are generated in the lower part of the second frame and the upper part of the third frame. Further, FIG. 5 shows an example of an image after the optical axis of the video camera 1 is shaken with respect to the subject and the shake is corrected by the image stabilization unit 111. It is assumed that the image stabilization correction amount in the image stabilization unit 111 is horizontal H2 pixels, vertical V2 pixels in the second frame, horizontal H3 pixels, and vertical V3 pixels in the third frame. The anti-shake correction amount is a right or downward correction if the value is positive, and a left or upward correction if the value is negative.

  First, as shown in FIGS. 5A, 5B, and 5C, the average level of each line is calculated for the images of the first frame, the second frame, and the third frame.

  Next, as shown in FIG. 5D, the average level difference of each line is calculated for the second frame and the first frame (hereinafter referred to as a first level difference). Here, of the first level difference, the number of lines higher than the threshold is N ′.

  Further, as shown in FIG. 5E, the average level difference of each line is calculated for the third frame and the first frame (hereinafter referred to as a second level difference). Here, of the second level difference, the number of lines higher than the threshold value is M ′.

  Here, the reference line number for determination in the flash band detection is changed from the default value held in the memory unit 105 to (total number of lines for one frame−V2 + V3). This process corresponds to the process in step S102 of FIG.

  At this time, if N ′ + M ′ corresponds to (the total number of lines for one frame−V2 + V3), it is determined that a flash band has been detected. This processing corresponds to the processing in step S105 in FIG.

  As described above, in the first embodiment, even when the optical axis is shaken with respect to the subject by changing the reference line number for determination according to the image stabilization amount, the electronic It is possible to appropriately detect the flash band after performing vibration isolation.

<Second Embodiment>
The second embodiment shows an example in which the flash band detection frame is changed according to the image stabilization correction amount.

  FIG. 6 is a flowchart showing the procedure of image correction processing after shooting in the second embodiment of the present invention.

  The processing in steps S101, S104, and S105 in FIG. 6 is the same as that in FIG.

  In step S202, the number of reference lines for determination in flash band detection is changed in the same manner as in step S102 in FIG. 2 in accordance with the image stabilization correction amount in the image stabilization unit 111, and the flash in the method in FIG. Change the band detection frame. After the processing is completed, the process proceeds to step S203.

  In step S203, flash band detection is performed using the reference line number and flash band detection frame changed in step S202. The detection method is the same as the method described in FIG. After the processing is completed, the process proceeds to step S104.

  FIG. 7 is a diagram for explaining the method of changing the flash band detection frame in step S202 of FIG.

  As shown in the upper part of FIG. 7, there is a stationary subject at the center of the image before the image stabilization correction. It shows how the axis is blurred. Further, a flash band is generated in the lower part of the image in the second frame and in the upper part of the image in the third frame. Regarding the flash band detection frame, the memory unit 105 holds HS as the default value of the horizontal start position and HE as the default value of the horizontal end position. For example, the flash band detection frame is set in this way when it is desired to increase the detection accuracy while avoiding the region where the optical characteristics are deteriorated in the photographing optical system 101 (for example, the sensitivity is decreased due to a decrease in the amount of peripheral light).

  On the other hand, these images are as shown in the lower part of FIG. Here, the image stabilization correction amount is horizontal H2 pixels, vertical V2 pixels in the second frame, horizontal H3 pixels in the third frame, and vertical V3 pixels in the first frame, and blurring occurs in the first and fourth frames. Not electronic vibration isolation.

  In the first frame, since electronic image stabilization is not performed, the flash band detection frame does not change, and the horizontal start position HS and horizontal end position HE remain unchanged.

  In the second frame, electronic image stabilization for horizontal H2 pixels and vertical V2 pixels is performed. At this time, the flash band detection frame after the image stabilization correction is changed by the amount of the image stabilization correction and becomes the horizontal start position HS + H2 and the horizontal end position HE + H2. Although not shown in FIG. 7, the reference line number for determination in flash band detection is changed from the default value held in the memory unit 105 to (total number of lines for one frame−V2 + V3). This process corresponds to the process in step S202 of FIG.

  In the third frame, electronic image stabilization for horizontal H3 pixels and vertical V3 pixels is performed. At this time, the flash band detection frame after the image stabilization correction is changed by the amount of the image stabilization correction, and becomes the horizontal start position HS + H3 and the horizontal end position HE + H3. As in the case of 2 frames, although not shown in FIG. 7, the reference line number for determination in flash band detection is changed from the default value held in the memory unit 105 to (total number of lines for 1 frame−V2 + V3). change. This process corresponds to the process in step S202 of FIG.

  In the fourth frame, since electronic image stabilization is not performed, the flash band detection frame does not change, and the horizontal start position HS and horizontal end position HE remain unchanged.

  In step S203, flash band detection is performed using the flash band detection frame determined as described above.

  As described above, in the second embodiment, suitable flash band detection is achieved by changing not only the number of reference lines for determination in flash band detection but also the flash band detection frame in accordance with the image stabilization amount. It becomes possible. In particular, it is useful for a frame set for the purpose of improving detection accuracy by avoiding areas with poor characteristics such as the photographing optical system 101 and the CMOS sensor 102, since the effect can be maintained even after electronic image stabilization. .

  Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. A part of the above-described embodiments may be appropriately combined.

  Also, when a software program that realizes the functions of the above-described embodiments is supplied from a recording medium directly to a system or apparatus having a computer that can execute the program using wired / wireless communication, and the program is executed Are also included in the present invention.

  Accordingly, the program code itself supplied and installed in the computer in order to implement the functional processing of the present invention by the computer also realizes the present invention. That is, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.

  In this case, the program may be in any form as long as it has a program function, such as an object code, a program executed by an interpreter, or script data supplied to the OS.

  As a recording medium for supplying the program, for example, a magnetic recording medium such as a hard disk or a magnetic tape, an optical / magneto-optical storage medium, or a nonvolatile semiconductor memory may be used.

  Further, as a program supply method, a computer program that forms the present invention is stored in a server on a computer network, and a connected client computer downloads the computer program.

DESCRIPTION OF SYMBOLS 101 Image | photographing optical system 102 CMOS sensor 103 Image | video signal processing part 104 Whole control calculating part 105 Memory part 106 Recording medium 107 Recording medium control I / F part 108 External I / F part 109 Flash band detection part 110 Flash band correction | amendment part 111 Vibration isolation Part

Claims (7)

Conversion means for acquiring an image signal read from the image sensor by a rolling shutter method and converting the image signal into a video signal composed of a plurality of frames;
A flash band detecting means for detecting a flash band generated by an external flash and extending over two of the plurality of frames;
A flash band correcting means for correcting the image of the frame in which the flash band is detected into a full flash image;
Anti-vibration means for correcting each shake of the plurality of frames by electronic anti-vibration;
An imaging apparatus comprising: a changing unit that changes a method of detecting the flash band in accordance with a shake amount corrected by the image stabilizing unit. The method of detecting the flash band is a method of determining that the flash band has occurred when the number of lines having a luminance level higher than a threshold value in two consecutive frames is a predetermined number of lines,
The said flash band detection means changes the said predetermined number of lines according to the amount of blurring of the perpendicular direction correct | amended by the said anti-vibration means about the said 2 continuous flame | frames. Imaging device. First holding means for holding the amount of shake in the vertical direction corrected by the vibration isolation means for the two consecutive frames;
The flash band detecting unit changes the predetermined number of lines based on the amount of shake in the vertical direction held in the first holding unit when detecting the flash band for the two consecutive frames. The imaging apparatus according to claim 2, wherein:   The flash band detection means changes the flash band detection frame in the horizontal direction by one of the plurality of frames by the amount of horizontal blur corrected by the image stabilization means. The imaging device according to any one of claims 1 to 3. Second holding means for holding a horizontal blur amount corrected by the vibration isolation means for the one frame;
When the flash band detecting unit detects the flash band for the one frame, the flash band detecting unit detects a horizontal start position of the flash band detecting frame based on the horizontal blur amount held by the second holding unit. The imaging apparatus according to claim 4, wherein the horizontal end position is changed. A conversion step of acquiring an image signal read from the image sensor by a rolling shutter method and converting the image signal into a video signal composed of a plurality of frames;
A flash band detecting step of detecting a flash band generated by an external flash and extending over two of the plurality of frames;
A flash band correcting step of correcting the image of the frame in which the flash band is detected into a full flash image;
An anti-vibration step of correcting each shake of the plurality of frames by electronic image stabilization;
A control method comprising a changing step of changing the flash band detection method in accordance with the shake amount corrected in the image stabilization step. A program for executing the control method according to claim 6.