JP2018142983A - Image processing device and method of controlling the same, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of selecting an optimum image as a still picture from a photographed dynamic image with ease.SOLUTION: An image processing device comprises: a first calculation unit that calculates a moving picture shake correction amount by using a shake signal outputted from a shake detector; a second calculation unit that calculates a still picture shake correction amount having a higher shake correction performance than the moving picture shake correction amount by using the shake signal outputted from the shake detector; a generation unit that generates an evaluation value indicating a magnitude of a shake amount on an imaging surface of an imaging element on the basis of a difference between the moving picture shake correction amount and the still picture shake correction amount; and a recording unit that records an image of each frame of a moving picture and an evaluation value while associating them with each other. The generation unit changes a display form for displaying information on the evaluation value on a display unit while being associated with the image of each frame of the moving picture, depending on the magnitude of the shake amount.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for generating a still image from a captured moving image.

  In recent years, an increase in the number of pixels of an imaging device capable of capturing a moving image has been rapidly progressing. Imaging devices that shoot full HD moving images have already become widespread, and imaging devices capable of 4K and 2K moving image shooting are gradually beginning to enter the market.

  Due to such high definition of moving images, each frame image of moving images has a sufficient number of pixels to be used as a still image. Accordingly, it is considered that usage methods for generating a still image from each frame of a moving image will further expand in the future.

JP 2010-252078 A

  A problem in generating a still image from a moving image is that it is difficult for the user to determine which frame image is the optimum image as a still image. For example, when viewed as a moving image, there are many cases in which image blurring or defocusing that is not noticed because the image constantly changes is at an unacceptable level when viewed as a still image. It is very troublesome for the user to check it frame by frame.

  In response to such a problem, for example, Patent Document 1 discloses the following method. That is, the camera image status information recorded during moving image recording is read during moving image reproduction. Then, an image suitable as a still image is picked up from the frames before and after the pressing operation of the still image recording switch.

  However, in the above conventional example, although the concept itself of recording status information such as AF, AE, AWB, shake, etc. in synchronization with a moving image and using it when selecting a still image during moving image reproduction is disclosed, The method for generating status information is not specifically described.

  The present invention has been made in view of the above-described problems, and an object thereof is to provide an image processing apparatus that can easily select an optimal image as a still image from a captured moving image.

  An image processing apparatus according to the present invention uses a first calculation unit that calculates a shake correction amount for a moving image using a shake signal output from the shake detection unit, and a shake signal output from the shake detection unit. Second calculation means for calculating a still image shake correction amount having a higher shake correction performance than the moving image shake correction amount, and the moving image shake correction amount and the still image shake correction amount. A generating unit configured to generate an evaluation value indicating a magnitude of a shake amount on an imaging surface of the image sensor based on the difference; and a recording unit configured to record the image of each frame of the moving image and the evaluation value in association with each other. The generating unit is characterized in that a display format in which information relating to the evaluation value is displayed on the display unit in association with an image of each frame of the moving image is changed according to the magnitude of the shake amount.

  According to the present invention, it is possible to provide an image processing apparatus that can easily select an optimal image as a still image from a captured moving image.

1 is a block diagram illustrating a configuration of a video camera that is an embodiment of an imaging apparatus of the present invention. The figure for demonstrating the example of the calculation method of the metadata for a focus. The figure for demonstrating the example of the calculation method of the metadata for exposure. The figure for demonstrating the example of the calculation method of the metadata for white balance. The figure which shows the block structure for calculating the metadata for shake. The figure for demonstrating the output of a shake correction amount calculating part and a shake amount calculating part. The figure for demonstrating the example of the calculation method of the metadata for shake. The figure for demonstrating another example of the calculation method of the metadata for shake. The figure for demonstrating the method of calculating panning speed. The figure for demonstrating the calculation of the shake amount calculating part which considered panning speed. The flowchart when producing | generating a still image from a moving image using metadata. The figure which shows the example of a display which alert | reports the appropriateness | suitability as a still image to a user.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a block diagram showing a configuration of a video camera capable of capturing a moving image, which is an embodiment of an imaging apparatus of the present invention. FIG. 1 shows each block of the imaging apparatus used at the time of moving image shooting. The configuration of the imaging apparatus 100 and the operation during moving image shooting will be specifically described with reference to FIG.

  In FIG. 1, an imaging apparatus 100 includes a variable power lens 101, a correction optical system 102, a diaphragm 103, and a focus lens 104 sequentially arranged in the optical axis direction, and a photographing optical system is configured by these and other optical systems (not shown). Is done.

  The zoom lens 101 is a lens that moves in the optical axis direction to perform zooming. The focus lens 104 is a lens that has both a function of correcting the movement of the focal plane accompanying zooming and a function of focusing. The diaphragm 103 is a diaphragm blade that adjusts the amount of incident light.

  An imaging element 105 is disposed behind the focus lens 105. The image sensor 105 images a subject by photoelectric conversion. The image sensor 105 is constituted by, for example, an XY address type CMOS (Complementary Metal Oxide Semiconductor) image sensor. The signal photoelectrically converted by the image sensor 105 is converted into a digital signal inside the image sensor 105 and then supplied to the signal processing unit 111. The signal processing unit 111 performs various signal processes such as gamma correction and color correction on the image information output from the image sensor 105.

  The system control unit 121 controls the entire imaging apparatus 100. For example, the system control unit 121 receives information such as luminance values and colors from the signal processing unit 111 and performs various arithmetic processes. Details of the control performed by the system control unit 121 will be described later.

  The zoom drive unit 106 is a drive source for moving the variable magnification lens 101, and performs a zooming operation in accordance with instructions from the system control unit 121. The system control unit 121 sends an instruction to the zoom drive unit according to an operation of a zoom operation unit (not shown) by the user.

  The focus drive unit 109 is a drive source for moving the focus lens 104, and is driven according to an instruction from the system control unit 121. The system control unit 121 determines the drive position of the focus lens 104 according to the signal supplied from the distance information generation unit 117. The distance information generation unit 117 generates distance information indicating the distance between the imaging device 100 and the subject from the result of signal processing performed on the image signal acquired by the imaging element 105 in the signal processing unit 111. The distance information may be a known method such as a method of performing phase difference AF using a plurality of pixels on the image sensor 105. Further, the distance information may be generated using a distance information acquisition device such as a sensor dedicated to phase difference AF or an infrared sensor.

  The aperture drive unit 108 is a drive source for adjusting the amount of light incident on the image sensor 105 by driving the aperture 103, and is driven according to an instruction from the system control unit 121. The image sensor driving unit 110 supplies drive pulses and the like for driving the image sensor 105 to the image sensor 105 in accordance with instructions from the system control unit 121, and reads out the charge accumulated in the image sensor 105 and the exposure time, that is, the shutter speed. Make adjustments. In general, the image sensor 105 performs an electronic shutter operation in which a signal charge accumulated in a pixel is swept away by applying a shutter pulse, and an optical image is photoelectrically converted and accumulated in a period until the next readout. Is done. This accumulation period is the shutter speed. If the appropriate shutter speed cannot be set when the subject brightness is low, the level adjustment of the image signal output from the image sensor 105, that is, the gain adjustment is performed to correct inappropriate exposure due to insufficient exposure.

  The AE signal generation unit 118 calculates a photometric value according to the brightness of the subject and supplies the calculated value to the system control unit 121 by performing arithmetic processing mainly including cumulative addition of digital signals for each pixel. The system control unit 121 drives the aperture drive unit 108 and the image sensor drive unit 110 based on the photometric value generated by the AE signal generation unit 118, and sets the aperture 103, the shutter speed, and the gain of the image sensor 105. , Control the exposure.

  The AWB signal generation unit 119 converts each R / G / B pixel signal supplied from the image sensor 105 to the signal processing unit 111 into luminance Y and color difference signals (RY, BY). Then, it is determined whether or not the color is an achromatic color based on the black body radiation locus, and all the color difference signals of the pixels determined to be an achromatic color are integrated to obtain an average value, thereby generating an AWB signal. The signal processing unit 111 has a circuit block that can multiply R / G / B individual colors, and the system control unit 121 outputs an image output from the image sensor 105 based on the AWB signal. White balance control is performed to control so that the white color in the data can be displayed correctly.

  The correction optical system 102 is a correction system capable of optical shake correction that deflects the direction of the optical axis by being moved in a direction perpendicular to the optical axis. The correction optical system drive unit 107 is a drive source for moving the correction optical system 102 and is driven according to an instruction from the system control unit 121. The angular velocity sensor 120 detects a shake applied to the imaging device 100 as an angular velocity signal, and supplies the angular velocity signal to the system control unit 121. The system control unit 121 generates a control signal to the correction optical system driving unit 107 in order to move the correction optical system 102 so as to correct the movement of the subject image on the imaging surface from the angular velocity signal. As a result, the movement of the subject image on the imaging surface caused by the shake of the apparatus is corrected and formed on the imaging element 105. The correction optical system 102 may be replaced with a configuration in which the image sensor 105 moves relative to the imaging optical system in a direction perpendicular to the optical axis.

  The metadata generation unit 113 generates predetermined metadata indicating the quality of the captured state of the captured image based on the data supplied from the system control unit 121, and associates it with the moving image data output from the signal processing unit 111. To the recording medium 109. Details of the metadata handled by the metadata generation unit 113 will be described later. The recording medium 114 is an information recording medium such as a magnetic recording medium such as a hard disk or a semiconductor memory. The display device 112 displays an image output from the signal processing unit 111 by a liquid crystal display element (LCD) or the like.

  Next, details of a method of calculating metadata supplied to the metadata generation unit 113 will be described. As for the metadata, four types of data of focus, exposure, WB (white balance), and shake are calculated in the system control unit 121 and supplied to the metadata generation unit 113. In still image shooting, these control parameters are set to follow the control target value calculated at the time of exposure with as little error as possible. On the other hand, in moving image shooting, if these control parameters are changed abruptly, the image will change suddenly as a moving image, so that it looks unnatural. Therefore, generally, control is performed in which the parameters are gradually changed toward the control target value. The metadata generation unit 113 calculates the metadata about the difference between the optimum control target value as a still image generated at this time and the actual setting value for natural appearance as a moving image by the method described below, It is recorded on the recording medium 114 in association with moving image data.

  Details of the focus metadata generation method will be described with reference to FIG. To make the explanation easier to understand, the symbols are defined as follows.

Dt: Subject distance detected by the distance information generation unit 117 Dl: Shooting subject distance determined by the position of the focus lens 104 Df: Depth of field (infinity side)
Dn: Depth of field (closest side)
FIG. 2A shows an example of a graph in which the horizontal axis is D1-Dt, that is, the difference between the target subject distance and the shooting subject distance determined by the position of the focus lens 104, and the vertical axis is metadata Mdata_focus generated. Is shown. When D1−Dt is 0, it indicates a state where the focus is completely achieved. At this time, Mdata_focus = 0. In FIG. 2A, Mdata_focus is calculated by the following calculation formula.

Mdata_focus = | (Dl−Dt) / (Df−Dt) |
However, D1-Dt ≧ 0
Mdata_focus = | (Dl−Dt) / (Dn−Dt) |
However, D1-Dt <0
That is, the amount of deviation from the target value of the shooting subject distance is normalized by the depth of field. As a result, Mdata_focus is data indicating that the closer to 0, the more focused, and the greater the value, the more out-of-focus.

  FIG. 2B is a graph in which the horizontal axis and the vertical axis are the same as those in FIG. 2A and show another example of the method for calculating Mdata_focus. In FIG. 2B, the calculation method of Mdata_focus is the same as that of FIG. 2A in the range of Dx <(D1-Dt) <Dy, and in other ranges, Mdata_focus when D1-Dt changes. The amount of change (gain) is increased. This can be easily corrected by performing edge enhancement or the like for some out-of-focus images by image processing inside the imaging apparatus 100 or image processing software of a PC, but when the edge enhancement is performed for an image with a large amount of blur. This is because the pseudo contour or the like is conspicuous, and the degree of deterioration of the image quality as a still image increases.

  Details of the exposure metadata generation method will be described with reference to FIG. The horizontal axes of the graphs of FIGS. 3A to 3C are expressed in a unit system using an APEX (Additive System of Photographic Exposure) system. The definition of each symbol is as follows.

Ev_now: exposure value determined by the current aperture and shutter speed Ev_target: appropriate exposure value determined by the output of the AE signal generator 118 FIG. 3A shows Ev_target-Ev_now, that is, the appropriate exposure value and the current exposure value. And an example of a graph in which the vertical axis represents generated metadata Mdata_exposure. The abscissa indicates overexposure when it increases in the plus direction, underexposure when it decreases in the minus direction, and indicates a state where the exposure is accurately matched when 0. At this time, Mdata_exposure = 0. In FIG. 3A, Mdata_exposure is calculated by the following calculation formula.

Mdata_exposure = | Ev_target−Ev_now | / (1/3)
That is, the amount of deviation of the current exposure from the appropriate exposure is normalized by a predetermined Ev value (here, 1/3 Ev). Here, normalization by 1/3 Ev is merely an example. This numerical value may be arbitrarily set by the user, or may be variable depending on the luminance distribution of the subject. For example, when a deviation of 1/3 Ev occurs, overexposure or blackout occurs, a method such as normalization by 1/5 Ev may be employed. By this calculation, Mdata_exposure is data indicating that the exposure is more appropriate as it is closer to 0, and that the exposure is underexposed or overexposed as it is greater than 1.

  FIG. 3B is a graph showing another example of the method for calculating Mdata_exposure, in which the horizontal axis and the vertical axis are the same as those in FIG. In FIG. 3B, the calculation method of Mdata_exposure is the same as that in FIG. 3A in the range of −1 <(Ev_target−Ev_now) <1, and in other ranges, Ev_target−Ev_now is changed. The change amount (gain) of Mdata_exposure is increased. This can be easily corrected by adjusting the brightness level of the image if there is a slight exposure shift by image processing inside the imaging apparatus 100 or image processing software of the PC, but in an image with a large exposure shift black This is because the degree of deterioration of the image quality as a still image becomes large, such as being unable to correct crushing and whiteout, and making noise stand out even after correction.

  FIG. 3C is a graph in which the horizontal axis and the vertical axis are the same as those in FIGS. 3A and 3B, and show still another example of the Mdata_exposure calculation method. In FIG.3 (c), Mdata_exposure is calculated with the following formulas.

Mdata_exposure = (2 ^{(| Ev_target-Ev_now |) -1)} / (2 ^(1/3) -1)
The Ev value is a unit system in which the amount of light incident on the image sensor is expressed as a logarithm with a base of 2. That is, when the Ev value changes by 1, the light amount is doubled or halved. The above expression is normalized after the unit system of the APEX system is converted to a unit system of actual light quantity, and the amount of exposure deviation can be expressed more accurately by metadata Mdata_exposure.

  Details of the white balance metadata generation method will be described with reference to FIG. The graph of FIG. 4A uses the color difference of RY and BY as coordinates. When the coordinates of the AWB signal described above are located in the vicinity of the origin O in FIG. 4A, it indicates that the RGB balance is achieved, that is, the white balance is matched. Conversely, the farther from the origin O, the more white balance is shifted. A vector of the AWB signal on the coordinates in FIG. 4A is defined as WB_Vector.

  The horizontal axes of the graphs of FIGS. 4B and 4C are examples of graphs in which the size of WB_Vector is taken and the vertical axis is metadata Mdata_wb to be generated. As described above, the horizontal axis is an image in which the white balance is shifted as the value increases. In FIG. 4B, Mdata_wb is calculated by the following calculation formula.

Mdata_wb = | WB_Vector | / WB_TH
That is, the amount of deviation from the optimum white balance value is normalized by the predetermined threshold WB_TH. Here, WB_TH is set as an allowable value of the deviation amount of white balance. Since the allowable value of the color misregistration amount varies greatly among individuals and is difficult to determine uniquely, the user may arbitrarily set it. Further, depending on the light source, it may not be able to converge to the origin. In this case, the threshold value of WB_TH may be widened, or the origin of FIG. 4A may be shifted according to the light source. By this calculation, Mdata_wb becomes data indicating that the white balance is more appropriate as it is closer to 0, and that the white balance is shifted as it is larger than 1.

  FIG. 4C is a graph showing another example of the method for calculating Mdata_wb, in which the horizontal axis and the vertical axis are the same as those in FIG. 4B. In FIG. 4C, in the range of | WB_Vector | <WB_TH2, the calculation method of Mdata_wb is the same as that in FIG. 4B, and in other ranges, the amount of change in Mdata_wb when | WB_Vector | changes ( (Gain) is increased. This can be easily corrected by adjusting the color level of the image if there is some color misregistration by image processing inside the imaging apparatus 100 or image processing software of the PC, but noise is large in an image with large color misregistration. This is because the degree of deterioration of the image quality as a still image increases, such as becoming conspicuous.

  The details of the image blur metadata generation method will be described with reference to FIGS. FIG. 5A is an example of a block diagram for generating image blur metadata. FIG. 5A is obtained by adding processing inside the system control unit 121 to FIG. 1, and the blocks outside the system control unit 121 are as described in FIG.

  The shake correction amount calculation unit 201 calculates the drive position of the correction optical system 102 based on the angular velocity detection result of the angular velocity sensor 120 and sends a drive instruction to the correction optical system drive unit 107. The shake amount calculation unit 202 calculates the shake amount applied to the imaging device 100 during the charge accumulation time of the image sensor 105 based on the output of the angular velocity sensor 120. The metadata generation unit 203 calculates image shake metadata to be passed to the metadata generation unit 113 based on outputs from the shake correction amount calculation unit 201 and the shake amount calculation unit 202.

  FIGS. 6A to 6C are graphs for explaining the calculation timing of metadata. FIG. 6A is a graph showing the operation timing of an image for two frames, with the horizontal axis representing time and the vertical axis representing charge accumulation and readout timing for each line of the image sensor 105. In order to make the explanation easy to understand, it is assumed that the image of the previous frame in time is the frame 1 and the image of the subsequent frame is the frame 2.

  In FIG. 6A, time T10 indicates the timing at which charge accumulation on one end line of the frame 1 of the image sensor 105 starts, and the diagonal parallelograms are sequentially time-sequential toward the other end. Charge accumulation starts at the timing of the left side. Time T11 indicates the timing at which charge reading is started for the line where charge accumulation has started at time T10, and the thick line portion starting from time T11 indicates the timing at which the charge of each line is read. The time between time T10 and T11 is the shutter speed. Time T12 is the timing when the charge accumulation and readout of all the lines in frame 1 are completed. Times T20, T21, and T22 indicate the timings of the start of charge accumulation, the start of reading of charges (end of charge accumulation), and the end of reading of all charges in the frame 2, respectively.

  FIG. 6B is a graph showing changes with time, with the horizontal axis representing time and the vertical axis representing the result of converting the output of the shake correction amount calculation unit 201 into the number of moving pixels on the imaging surface. The solid line graph in FIG. 6C is a graph showing changes with time, with the horizontal axis representing time and the vertical axis representing the output of the shake amount calculation unit 202 converted to the number of moving pixels on the imaging surface. .

  FIG. 6B shows how much image blur correction has been performed by the correction optical system 102. The solid line graph in FIG. 6C shows how much shake has occurred in the imaging apparatus 100 during the time from the start of charge accumulation in frame 1 and frame 2 to the end of charge readout. Therefore, the shake amount of frame 1 starts from time T10 and ends at time T12, and the shake amount of frame 2 starts from time T20 and ends at time T22. When evaluating the shake amount of a moving image, it is necessary to calculate the shake amount generated between frames 1 and 2, that is, the shake amount generated between T20 and T22, from the intermediate time between T10 and T12. . On the other hand, since the present embodiment is an invention related to the use of generating a still image from each frame of a moving image, the amount of shake generated within the time for generating an image of each frame is calculated.

  Comparing the graphs of FIG. 6B and FIG. 6C, it can be determined how accurately the shake applied to the imaging apparatus 100 is corrected for the frame 1 and the frame 2. For frame 1, the locus of shake correction amount from point A to point C in FIG. 6B and the shake locus of frame 1 in FIG. Is getting smaller. On the other hand, for frame 2, the locus of shake correction amount from point B to point D in FIG. 6 (b) and the shake locus of frame 2 in FIG. 6 (c) are inconsistent as will be described below. Yes.

  First, unlike the point A, the coordinate of the vertical axis of the point B is a numerical value B0 that is not zero. Therefore, in order to compare with the deflection trajectory from T20 in FIG. 6C, it is necessary to subtract B0 from the graph in FIG. 6B. The result of this subtraction is represented by a dotted line graph in FIG. When the dotted and solid graphs in FIG. 6C are compared, there is a difference between the two. This is due to the following reason. In the case of moving image shake correction, if 100% shake correction is continued until the correction limit, shake correction cannot be performed, and the state in which the shake is corrected and the state in which the shake is not corrected are repeated, resulting in poor quality video. In order to avoid this, when the correction optical system 102 approaches the correction limit, control such as changing the cutoff frequency of the low frequency band cutoff filter inside the shake correction amount calculation unit 201 is performed to weaken the shake correction effect, In general, control for continuously operating the correction optical system 102 is performed. On the other hand, still image shake correction is required to perform shake correction as close to 100% as possible to the correction limit during still image exposure. The difference in the concept of shake correction between the moving image and the still image causes the difference in FIG. That is, in order to perform shake correction as a still image in frame 2, it is desirable to control the correction optical system 102 according to the locus of the solid line in FIG. 6C, but in moving image shooting, the correction limit is approached. Control is performed according to the dotted locus. The difference between the two becomes the amount of shake of the frame 2.

  In the graph of FIG. 7A, the horizontal axis indicates the horizontal shake amount of the image sensor 105, and the vertical axis indicates the vertical shake amount (unit is pixel). If the graphs of FIGS. 6B and 6C indicate the shake correction amount and shake amount of either the horizontal or vertical direction of the image sensor 105, the final axis of one axis of the frame 2 is final. The shake amount is Shake_Amount in FIG. If this is calculated with both vertical and horizontal axes, the shake amount of each frame of the moving image can be expressed on two-dimensional coordinates as shown in FIG. A vector of shake on the two-dimensional coordinates is defined as Shake_Vector. As a matter of course, the shake becomes smaller as the coordinates of the Shake_Vector are closer to the origin.

  FIG. 7B and FIG. 7C show examples of graphs in which the horizontal axis of the graph is the size of Shake_Vector and the vertical axis is metadata Mdata_shake that is generated. As described above, the horizontal axis is an image with a greater shake as the value increases. In FIG. 7B, Mdata_shake is calculated by the following calculation formula.

Mdata_shake = | Shake_Vector | / Shake_TH
That is, the shake amount on the image sensor 105 is normalized by the predetermined threshold value Shake_TH. Here, Shake_TH is set as an allowable shake amount on the imaging surface. The allowable value of the shake amount depends on the number of pixels of the image sensor 105, the resolution of the photographing optical system, and the like. Further, the user may be able to set it arbitrarily. As a result of this calculation, Mdata_shake becomes data indicating that the shake amount is at a level that does not cause a problem as a still image as it is closer to 0.

  FIG. 7C is a graph in which the horizontal axis and the vertical axis are the same as those in FIG. 7B and show another example of the method for calculating Mdata_shake. In FIG. 7C, in the range of | Shake_Vector | <Shake_TH2, the calculation method of Mdata_shake is the same as that in FIG. 7B. However, in other ranges, the amount of change (gain) of Mdata_shake when | Shake_Vector | changes. This can be easily corrected by a known image restoration technique or the like if there is a slight shake by image processing inside the imaging apparatus 100 or image processing software of a PC, but a pseudo contour in an image with a large shake. This is because noise and noise become conspicuous, and the degree of deterioration of image quality as a still image increases.

  Another example of the method for calculating Mdata_shake will be described using the graph of FIG. In FIG. 7, the calculation of Mdata_shake is performed using the difference between the shake correction amount corrected by the shake correction amount calculation unit 201 and the shake amount detected by the shake amount calculation unit 202 at the end of reading of all charges. In many cases, the shake amount of each frame of the moving image can be correctly expressed by this calculation method, but there are cases where it cannot be correctly expressed as shown in FIG. In FIG. 8A, the horizontal axis and the vertical axis are the same as those in FIG. 7B, and an example of a shake trajectory on the imaging surface from the start of charge accumulation of the image sensor 105 to the end of readout of all charges is shown. It is a graph. In the example of FIG. 8A, the end of reading out all charges is a point A0, which is a point closer to the origin O, but the trajectory in the middle passes coordinates farther from the origin O than A0. When the shutter speed is high, the frequency is very low, but the frequency of drawing such a trajectory increases as the shutter speed decreases. In this case, the shake amount of each frame of the moving image cannot be expressed correctly only with A0 at the end of reading out all charges.

  Therefore, a method for calculating the shake amount instead of the Shake_Amount will be described with reference to FIG. The graph in FIG. 8B is a graph in which the movement positions for each predetermined time are plotted by A1 to A7 with respect to the shake locus in FIG. Then, the calculation formula for the shake amount is as follows.

| A1-O | + | A2-A1 | + | A3-A2 | + | A4-A3 | + | A5-A4 | + | A6-A5 | + | A7-A6 | (Formula 1)
(Expression 1) is an expression (integrated value) in which the magnitude of the shake on the imaging surface at every predetermined time is sequentially integrated from the start of charge accumulation to the end of reading of all charges. According to (Expression 1), the total movement distance of the shake trajectory can be calculated, and the above-described problem that the shake amount of each frame of the moving image cannot be correctly expressed can be avoided. In FIG. 7B or FIG. 7C, Mdata_shake can be calculated by replacing the calculation result of (Equation 1) with | Shake_Vector |.

  Another example of a shake amount calculation method instead of the above Shake_Amount will be described with reference to FIG. FIG. 8C is a graph in which the deflection locus from the origin O to A2 in FIG. As shown in the figure, if the angle formed by the horizontal axis and the vector A1-O is θ0, the angle formed by the horizontal axis and the vector A2-A1 is θ1, and the relative angle between the vector A1-O and the vector A2-A1 is θ2. , Θ2 = θ0−θ1. When the locus of the vector on the image plane of the shake amount is close to a straight line, it is relatively easy to restore the image without shake by image processing inside the imaging apparatus 100, image processing software of the PC, etc. The more you draw, the more difficult it is to restore. Therefore, when integrating the magnitude of shake on the imaging surface every predetermined time (per unit time), a process of multiplying a gain that increases as the angle change θ2 of the difference vector increases is performed. Thus, it is possible to obtain data that takes into account the difficulty of restoration. For example, when | A2-A1 | is added to | A1-O |, a method of performing the following calculation is conceivable.

| A1-O | + | A2-A1 | (1 + sin θ2)
According to this equation, when θ2 is 0 deg, sin θ2 = 0, and when θ2 is 90 deg, sin θ2 = 1, and a gain corresponding to the magnitude of θ2 can be set. This calculation is performed from the start of charge accumulation to the end of reading out all charges, and the result is replaced with | Shake_Vector | in FIG. 7B or FIG. 7C to calculate Mdata_shake taking into account the ease of shake recovery. Can do.

  Still another example of the calculation method of Mdata_shake will be described. FIG. 5B is an example of a block diagram for generating shake metadata. Since FIG. 5B is obtained by adding blocks 301 to 303 to FIG. 5A, description of the other blocks is omitted.

  The motion vector detection unit 303 includes a luminance signal included in the current video signal generated by the signal processing unit 111 and a luminance signal included in the video signal of the previous frame stored in the image memory inside the motion vector detection unit 303. The motion vector of the image is detected based on the above. The motion vector output detected by the motion vector detection unit 303 is supplied to the panning determination unit 301 and the panning speed calculation unit 302. Note that the motion vector detection unit 303 is not an essential component.

  The panning determination unit 301 determines whether the imaging apparatus 100 is in a panning state based on the output from the angular velocity sensor 120 or the motion vector detection unit 303. When the panning determination unit 301 determines that the imaging apparatus 100 is in the panning state, the panning speed calculation unit 302 calculates the current panning speed and supplies it to the shake amount calculation unit 202. If it is determined that the panning state is not set, the panning speed is set to zero. The panning speed is calculated from the output of the angular velocity sensor 120, the output of the motion vector detection unit 303, or both. The shake correction amount calculation unit 202 calculates a shake amount in consideration of the panning speed calculated by the panning speed calculation unit 302.

  In moving image shooting, panning operation is frequently performed unlike still image shooting. When the main subject moves, shooting is often performed so that the main subject is held near the center of the screen. At this time, if only the method of normalizing the shake amount on the imaging surface described above and storing it as metadata is used, all data indicating that the shake is large during panning. When panning is performed in accordance with the movement of the main subject, it is in the same state as so-called panning shot of a still image, so all the frames of the moving image during the panning are shaken. It is not appropriate to be determined to be large.

  Therefore, the panning determination unit 301 first determines whether or not the imaging apparatus 100 is in a panned state. A known method may be used as the panning determination method. For example, there is a method of determining panning when the output of the angular velocity sensor 120, the output of the motion vector detection unit 303, or the integrated output (integrated value) thereof exceeds a predetermined value.

  The panning speed calculation unit 302 calculates the panning speed. The panning speed can be calculated by calculating the average value of the outputs of the angular velocity sensor 120. Further, the output of the motion vector detection unit 303 may be used. A panning speed calculation method using the output of the motion vector detection unit 303 will be described with reference to FIG. FIG. 9A shows an image in which a moving vehicle at the center of the screen is photographed so as to be held at the center of the screen. For motion vector detection, a block matching method is used in which an image is divided into a plurality of blocks and a motion vector is calculated for each block. A dotted line indicates each block for which motion vector detection is performed.

  FIG. 9B shows the result of motion vector detection in each block in the direction and size of the arrow. The motion vector within the bold line frame indicates the motion vector of the vehicle, and the motion vector outside the thick line frame indicates the motion vector of the other background area. If the average value of the motion vectors in the background area is Vector_Back and the average value of the motion vectors in the vehicle area is Vector_Car, the panning speed can be calculated by Vector_Back-Vector_Car. The above equation is the panning speed for zeroing the motion vector in the vehicle area. When the motion vector of the vehicle area is zero, it indicates that the shooting is 100% following the movement of the vehicle, and can be said to be close to an ideal panning shot.

  The shake amount calculation in consideration of the panning speed in the shake amount calculation unit 202 in FIG. 5B will be described with reference to FIG. FIG. 10A is a graph showing the output of the angular velocity sensor 120 (solid line) and the output of the panning speed (dotted line) calculated by the panning speed calculator 302 when the horizontal axis is time and the vertical axis is angular velocity. It is. FIG. 10A shows a graph when the imaging apparatus 100 is panned, and the output of the angular velocity sensor has a waveform in which a shake angular velocity signal having a high frequency is superimposed on the panning velocity.

  Here, the time T30 is set as the charge accumulation start timing of the image sensor 105, and the time T31 is set as the timing for ending all charge reading. At this time, if the shake amount is calculated by the calculation method that does not consider the panning speed described with reference to FIG. 6C, the result is the graph shown in FIG. FIG. 10A is a graph in a state where the panning speed is always generated. Therefore, when the shake amount is calculated by integrating the panning speed, the movement of the imaging device 100 due to panning is integrated, and the shake amount is shown in FIG. As shown in FIG. Since the correction optical system 102 performs control so as not to correct the normal panning movement, the metadata Mdata_shake calculated based on the difference between the shake amount illustrated in FIG. 10B and the shake correction amount calculation unit 201 is also a large value. Become.

  On the other hand, FIG. 10C is a graph in which the shake amount is calculated by integrating the result of subtracting the panning speed from the output of the angular velocity sensor 120. FIG. 10C shows the shake amount excluding the panning motion component. Since the correction optical system 102 performs control so as not to correct the normal panning movement, the shake amount shown in FIG. 10C and the output of the shake correction amount calculation unit 201 are close outputs, and are calculated based on the difference between the two. The metadata Mdata_shake has a small value.

  Thus, when the user performs panning, it is possible to avoid the phenomenon that Mdata_shake always has a large value and it is determined that the shake is large. Note that when there is no main subject and panning is performed only on a landscape, the above-described panning is not performed, and the image of each frame is simply an image with a landscape shake. In such a case, by determining that the panning determination unit 301 is not panning, it is possible to prevent the occurrence of a phenomenon that the image is determined to be small despite the large image shake. This determination can be made by using the motion vector described in FIG. For example, when the motion vector of the entire screen is directed in the same direction, it can be determined that the shooting is not chasing the main subject.

  The description so far has been made on the assumption that there is a mechanism for driving the correction optical system 102 or the image sensor 105. In the case of an imaging apparatus that does not have a means for optically correcting such shakes, the shake correction amount calculation unit 201 is not provided in FIGS. 5A and 5B. For this reason, the calculation of Mdata_shake may be performed using only the output of the shake amount calculation unit 202, and other calculation methods are the same as those in the case of having a means for optically correcting shake.

  Next, an example of how to use the above four types of metadata Mdata_focus, Mdata_exposure, Mdata_wb, and Mdata_shake so that the user can select an optimal still image from moving images will be described.

  First, FIG. 11 shows a flowchart until the user generates a still image from moving image data. S100 is a user operation, where the user operates an operation member (not shown) of the imaging apparatus 100, sets a mode for generating a still image from the captured moving image, and a moving image for generating a still image. Is selected, the process of this flowchart is started. In S101, the system control unit 121 reads metadata of all frames of the moving image selected in S100. In S102, an assist function is provided for the user to select an optimal still image using the metadata read in S101. S103 is a user operation, and a frame for generating a still image is determined by the user using the function of S102. In S104, decoding for creating one still image from the frame image determined by the user in S103 and encoding for further JPEG compression of the decoded image are performed. In S105, the still image generated in S104 is recorded on the recording medium 114. After the process of S105, the process of this flowchart ends.

  An example of the assist function in S102 is shown below. As the simplest example, as shown in FIG. 12, the numerical values of Mdata_focus, Mdata_exposure, Mdata_wb, and Mdata_shake are displayed on a diamond-shaped graph and displayed on the display device 112 simultaneously with the reproduction of the moving image. The user can select an optimal frame as a still image by selecting an image of a frame in which all numerical values are arranged close to the innermost diamond when reproducing a moving image.

  Further, a method of automatically selecting an image with high evaluation indicated by metadata on the imaging apparatus 100 side is also conceivable. For example, a method of displaying images on the display device 112 in order from a frame with a high evaluation so that the user can preferentially select the image, or a frame with a color that makes a frame with a high evaluation stand out. You can also take. In addition, if a frame with a high evaluation continues, only a similar image may be extracted. Therefore, one frame may be selected for each scene using known scene switching detection. Good. Furthermore, an image having a high metadata evaluation may be automatically selected from the faces of a specific person using a known face recognition function.

  Here, an example of a method for determining the final evaluation from the four metadata will be described. As the simplest example, a result obtained by adding four metadata Mdata_focus, Mdata_exposure, Mdata_wb, and Mdata_shake is used. The closer to 0, the higher the evaluation, and the higher the numerical value, the lower the evaluation. Moreover, you may multiply four metadata. The smaller the value is, the higher the evaluation is, and the larger the value is, the lower the evaluation is. Also, the user may be able to determine the weighting. For example, since the exposure may be intentionally shifted or the color may be changed intentionally, the exposure deviation may not be evaluated, and WB may be used by multiplying Mdata_wb by a coefficient smaller than 1.

  As described above, by using the four metadata in various forms, it is possible to provide various systems that allow the user to easily select an optimal image as a still image from a moving image.

  As described above, in the present embodiment, each of four types of camera parameters, focus shift, exposure shift, WB shift, and shake, is used as metadata for determining whether the image is appropriate as a still image. The optimal calculation method was shown considering the characteristics. Further, for these four types of camera parameters, for each frame at the time of moving image shooting, the amount of deviation from the allowable value is normalized by the allowable value and recorded in association with each frame of the moving image. Thus, the above four types of parameters can be determined on the same scale, and it is possible to appropriately notify the user which frame image is optimal as a still image. The user can easily generate an optimal still image from the moving image.

  Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. For example, it is not necessary to generate all of the above four types of parameters, and a system using any one or more types of parameters may be used.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

DESCRIPTION OF SYMBOLS 100: Imaging device, 102: Correction | amendment optical system, 103: Diaphragm, 104: Focus lens, 105: Image sensor, 110: Image sensor drive part, 111: Signal processing part, 113: Metadata production | generation part, 117: Distance information production | generation 118: AE signal generator, 119: AWB signal generator, 120: angular velocity sensor, 121: system controller

Claims (7)

First calculation means for calculating a shake correction amount for moving images using a shake signal output from the shake detection means;
Second calculation means for calculating a shake correction amount for a still image having a shake correction performance higher than the shake correction amount for the moving image, using a shake signal output from the shake detection means;
Generating means for generating an evaluation value indicating the magnitude of the shake amount on the imaging surface of the image sensor based on the difference between the shake correction amount for the moving image and the shake correction amount for the still image;
Recording means for recording the image of each frame of the moving image and the evaluation value in association with each other;
The image processing apparatus, wherein the generation unit changes a display format in which information on the evaluation value is displayed on the display unit in association with an image of each frame of the moving image according to the magnitude of the shake amount.   The image processing apparatus according to claim 1, further comprising a control unit that controls an optical shake correction unit using the moving image shake correction amount.   The generating means uses an integrated value of a difference per unit time between the moving image shake correction amount and the still image shake correction amount as the shake amount of the image of each frame, and the moving image shake correction amount and the 2. The shake amount of the image of each frame is calculated so that the integral value increases as the angle change of the vector of the difference per unit time of the shake correction amount for still images increases. The image processing apparatus described.   A determination means for determining whether or not the camera is in a panning state; and when the determination means determines that the camera is in a panning state, the generation means calculates a panning speed based on the output of the shake detection means; 2. The image according to claim 1, wherein data obtained by subtracting a motion amount due to the panning from the shake correction amount for the moving image and the shake correction amount for the still image is used as the shake amount of the image of each frame. Processing equipment. A first calculation step of calculating a shake correction amount for a moving image using a shake signal output from the shake detection means;
A second calculation step of calculating a shake correction amount for a still image having a shake correction performance higher than the shake correction amount for the moving image, using a shake signal output from the shake detection unit;
A generation step of generating an evaluation value indicating the magnitude of the shake amount on the imaging surface of the image sensor based on the difference between the shake correction amount for the moving image and the shake correction amount for the still image;
A recording step of recording the image of each frame of the moving image and the evaluation value in association with each other,
In the generation step, a display format in which information relating to the evaluation value is displayed on a display unit in association with an image of each frame of the moving image is changed according to a magnitude of the shake amount. Method.   The program for making a computer perform each process of the control method of Claim 5.   A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 5.

Images