JP2020065157A - Imaging apparatus, control method therefor, and program - Google Patents

Abstract

To provide an imaging apparatus capable of electronically applying a motion blur to a picked-up image, thereby achieving a moving image with high quality.SOLUTION: The imaging apparatus includes a sensor executing imaging at a preset shutter speed and calculates a difficulty degree involved in application of a motion blur to a picked-up image on the basis of the information concerning the movement between images acquired from continuously picked-up images and when the calculated difficulty degree is low against a preset determination reference, makes the sensor execute imaging at a high shutter speed and when the calculated difficulty degree is high, makes the sensor execute imaging at a low shutter speed.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an imaging device, a method for controlling the imaging device, and a program.

  When shooting a fast-moving subject such as an athlete or car as a still image, a relatively high shutter speed of (1/8000) seconds to (1/500) seconds is used to suppress blurring called motion blur. Often imaged. On the other hand, when a moving image captured at 60 frames per second is captured and viewed at a high shutter speed of (1/8000) seconds, for example, the image of the subject becomes too sharp and motion discontinuity called jerkiness occurs. I can perceive it.

  Therefore, in order to capture a moving image captured at 60 frames per second as a smooth moving image with motion blur left in the image, a comparison of (1/60) seconds to (1/125) seconds is made. Images are often taken at extremely low shutter speeds. That is, an image obtained by cutting out one frame of a moving image captured at a low shutter speed which is a normal moving image capturing method has motion blur and is not suitable for viewing as a still image.

  As one method to solve this problem, we propose a technology that electronically gives motion blur as post-processing to the captured image by capturing an image with a high shutter speed even for a moving image and using an optical flow for each pixel. (See Patent Document 1). In other words, a pseudo moving image with a low shutter speed is reproduced to generate a smooth moving image with less jerkiness. According to this, the user can view both the image corresponding to the image capturing at the high shutter speed for the still image and the image corresponding to the image capturing at the low shutter speed for the moving image.

  The optical flow is high-density motion vector information representing the motion of the subject, and is represented as a motion vector map for each pixel, for example. Optical flow can be calculated from successive images. As a method for calculating the optical flow, there is, for example, the Lucas Kanade method (hereinafter, also referred to as “LK method”) described in Non-Patent Document 1.

JP, 2010-28524, A

B. D. Lucas and T.L. Kanade, (1981), "An iterative image registration technique with an application to stereo vision," Processings of Imaging Understanding, 130-121.

  In a system that uses an optical flow to electronically add motion blur in order to achieve both moving image and still image quality, the optical flow detected when the movement of the subject in the captured scene is complicated May be incorrect. When motion blur is added to a captured image using an incorrect optical flow, an artifact that does not occur when the same scene is captured at a low shutter speed may occur in the image after the motion blur is added. The present invention has been made in view of such circumstances, and an object of the present invention is to provide a moving image of good quality in an imaging device capable of electronically adding motion blur to a captured image. To do.

  An image pickup apparatus according to the present invention includes a first image pickup unit that takes an image at a set shutter speed, an acquisition unit that obtains information regarding movement between images from continuously picked-up images, and the obtained movement. Based on the information related to, the difficulty calculation means for calculating the difficulty of motion blur addition to the captured image, and a preset judgment criterion, if the calculated difficulty is low, A control unit that causes the first image capturing unit to capture an image at a shutter speed of 1 and causes the first image capturing unit to capture an image at a second shutter speed that is slower than the first shutter speed when the difficulty level is high. And having.

  According to the present invention, it is possible to obtain a moving image of good quality in which the occurrence of artifacts is suppressed in an imaging device that can electronically add motion blur to a captured image.

It is a figure which shows the structural example of the imaging device in 1st Embodiment. 6 is a flowchart showing an example of control of the image pickup apparatus in the first embodiment. It is a figure which shows the structural example of the imaging device in 2nd Embodiment. It is a flow chart which shows the example of map generation processing in a 2nd embodiment. 9 is a flowchart showing an example of control of the image pickup apparatus according to the third embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The image capturing apparatus in each of the embodiments described below has a function of electronically adding motion blur to captured images based on information about movement between images acquired from continuously captured images. In the following, an optical flow will be described as an example of information related to movement between images, but the present invention is not limited to this, and any information indicating movement between images that can be used to add motion blur can be applied. is there.

(First embodiment)
A first embodiment of the present invention will be described. FIG. 1 is a block diagram showing a configuration example of an image pickup apparatus according to the first embodiment. The image pickup apparatus according to this embodiment includes a CPU 102, a memory 103, a non-volatile memory 104, a GUI unit 105, and an image processing engine 106. Further, the imaging device according to the present embodiment includes a sensor unit 107 having a lens unit 108 and a sensor 109, an optical flow generation unit 110, a difficulty level calculation unit 111, a motion blur adding unit 112, and an information storage unit 113. The modules 102 to 113 included in the image pickup apparatus are connected via a bus 101, and input / output of data between the modules is performed via the bus 101.

  A CPU (Central Processing Unit) 102 controls each module included in the imaging device. The CPU 102 is an example of a control unit. The memory 103 is a memory used as a main storage device of the CPU 102, for example. The nonvolatile memory 104 stores, for example, programs such as an OS (operating system) and applications, and various data such as image data. In the present embodiment, for example, a program stored in the non-volatile memory 104 is read into the memory 103, and the CPU 102 executes the read program to execute various processes.

  A GUI (Graphical User Interface) unit 105 is an interface unit for receiving an operation of the imaging device from the user and providing information of the imaging device to the user. For example, the user uses the GUI unit 105 to set a mode described later. The GUI unit 105 is, for example, a display with a touch panel. However, the GUI unit 105 is not limited to this, and may be, for example, a GUI for making settings using a cross key, a dial, a button, or the like.

  The image processing engine 106 converts the digital image signal output from the sensor unit 107 into image data and stores the image data in the memory 103. The image processing engine 106 also converts the image data into encoded data and records the encoded data in the non-volatile memory 104. In the present embodiment, the image data is assumed to be 8-bit RGB data of 1920 horizontal pixels and 1080 vertical pixels, but is not limited to this. Further, the encoded data is assumed to be image data converted by the HEVC (High Efficiency Video Coding) system, but the encoding system is not limited, and the operation in the present embodiment is not limited to non-compressed data. It is possible to carry out.

  The sensor unit 107 converts an optical image of a subject into an electric signal and outputs a digital image signal. The sensor unit 107 has a lens unit 108 including a lens and a diaphragm, and a sensor (image pickup element) 109 such as a CMOS image sensor or a CCD image sensor. The light captured through the lens unit 108 is converted into a digital image signal by the sensor 109. Note that FIG. 1 shows an example in which the image pickup apparatus has the lens unit 108 and the sensor 109, but the lens unit 108 may be detachably (replaceable) separately from the image pickup apparatus.

  In image pickup by an image pickup apparatus, a Tv (Time Value) value indicating a shutter speed, an Av (Aperture Value) value indicating a diaphragm, and a sensitivity are controlled so as to obtain an optimum exposure. In this embodiment, the Tv value is first determined, and the Av value and the sensitivity are determined accordingly. The method of determining the Tv value will be described later. Further, the method of determining the Av value and the sensitivity is a well-known technique, and thus the description thereof will be omitted. In imaging, the aperture of the lens unit 108 is controlled according to the determined Av value, the electronic shutter of the sensor 109 is controlled according to the Tv value, and the analog gain of the sensor 109 is controlled according to the sensitivity. In the present embodiment, image pickup, image data storage, coded data recording, Tv value control, Av value control, and sensitivity control are performed continuously at a rate of 60 frames per second.

  The optical flow generation unit 110 is continuously input with the image data stored in the memory 103, and generates an optical flow representing a movement between continuous images for each pixel based on the input image data. The optical flow generation unit 110 is an example of an acquisition unit. In the present embodiment, the optical flow generation unit 110 is assumed to generate an optical flow by the LK method, but the optical flow generation method is not limited to the LK method. The difficulty level calculation unit 111 uses the optical flow generated by the optical flow generation unit 110 to calculate the difficulty level of motion blur addition when electronically adding motion blur to a captured image.

  The motion blur adding unit 112 electronically adds motion blur to the captured image. The motion blur imparting unit 112 electronically imparts motion blur to a large Tv value, that is, an image captured at a high shutter speed, using the optical flow generated by the optical flow generating unit 110. The information storage unit 113 stores user setting information and the like set by the user of the imaging device via the GUI unit 105. The information storage unit 113 stores, for example, information indicating a mode of the imaging device set by the user.

  Hereinafter, a method of determining the Tv value in this embodiment will be described with reference to FIG. FIG. 2 is a flowchart showing a control example relating to determination of the Tv value of the sensor 109 in the image pickup apparatus according to the first embodiment. It is assumed that the operation of each step in the flowchart shown in FIG. 2 is executed by the CPU 102 of the image pickup apparatus.

  First, in step S201, the CPU 102 has a high shutter speed Tv value (hereinafter, also referred to as “high speed Tv value”) and a low shutter speed Tv value (hereinafter, also referred to as “low speed Tv value”). From the information storage unit 113. In the present embodiment, as an example, the high speed Tv value is a value indicating a shutter speed of (1/1000) seconds, and the low speed Tv value is a value indicating a shutter speed of (1/120) seconds. The high speed Tv value and the low speed Tv value are not limited to this. As the high-speed Tv value, a Tv value for shooting a sports scene or the like as a still image may be set, and in general, may be a value indicating a shutter speed of (1/500) seconds or less. Further, as the low-speed Tv value, a value at which the shutter speed is the reciprocal of the frame rate or a half thereof is generally used, and therefore, a value of that level may be set.

  Next, in step S202, the CPU 102 acquires the difficulty level information regarding motion blur addition calculated by the difficulty level calculation unit 111 as described below. The CPU 102 acquires the difficulty level of motion blur addition when electronically adding motion blur to a captured image from the difficulty level calculation unit 111.

  Next, in step S203, the CPU 102 compares the difficulty level of motion blur addition acquired in step S202 with a predetermined criterion (threshold value) to determine whether the difficulty level of motion blur addition is low. To judge. When the CPU 102 determines that the degree of difficulty in adding motion blur is low, the CPU 102 sets the shutter speed of the sensor 109 to the high-speed Tv value in step S204. When the CPU 102 determines that the degree of difficulty in adding motion blur is high, the CPU 102 sets the shutter speed of the sensor 109 to the low speed Tv value in step S205. The sensor unit 107 captures an image according to the Tv value set by the CPU 102.

  Next, a method of calculating the degree of difficulty of adding motion blur to a captured image will be described. The difficulty level of adding motion blur is calculated by the difficulty level calculation unit 111. The difficulty level calculation unit 111 may be a circuit or a configuration in which a CPU (programmable circuit) performs software processing. When it is realized by software processing, the CPU 102 can also perform calculation.

Since the optical flow represents a motion between two images, two consecutive input images are used as a first frame and a second frame, and a two-dimensional method is used to determine how the pixels of the first frame move in the second frame. It will be described as being expressed by a vector. The brightness of the pixel at the coordinates (x, y) in the first frame and the second frame is expressed as I ₁ (x, y) and I ₂ (x, y), respectively. The brightness may be calculated in advance from RGB, or a brightness image may be created in advance.

  At the coordinates (x, y) of the first frame, the motion vector expressing the motion from the first frame to the second frame is (u (x, y), v (x, y)). At this time, the mean square error (MSE: Mean Square Error) of the pixel value between the image of the first frame and the image generated by warping the pixels of the first frame by the optical flow (warping image) is expressed by the following equation (1) ).

  However, the expression {element | condition} shown in Expression (2) represents a set of elements that satisfy the condition. Further, in Expression (2), w and h are the width and height of the image, and the function round (a) is a function that returns a value obtained by rounding the real number a.

  The mean squared error (MSE) of the pixel values calculated by the equation (1) indicates the correlation between the two images, and the difficulty level calculation unit 111 calculates the mean squared error (MSE) of the pixel values as motion blur. It is calculated as the degree of difficulty of grant. The smaller the mean square error (MSE) of the pixel values, the higher the correlation of the images, and the more accurate the optical flow can be calculated, that is, the lower the difficulty of adding motion blur. Therefore, in the present embodiment, in step S203 shown in FIG. 2, the CPU 102 has high optical flow accuracy and difficulty of motion blur addition when the mean square error (MSE) of pixel values is less than 8, for example. Judge as low.

  In the example described above, the mean squared error (MSE) of pixel values is calculated as the degree of difficulty of motion blur addition, but another index indicating the correlation of images may be used. For example, as shown in the following formula (3), a peak signal-to-noise ratio (PSNR: Peak Signal Noise Ratio) may be calculated as the degree of difficulty of adding motion blur.

  When the peak signal-to-noise ratio (PSNR) calculated by the equation (3) is used, in step S203 shown in FIG. 2, the CPU 102 performs motion blur when the peak signal-to-noise ratio (PSNR) is 30 or more, for example. It is determined that the degree of difficulty of grant is low.

  In addition, structural similarity (SSIM) may be used as the degree of difficulty in providing motion blur. When structural similarity (SSIM) is used, in step S203 shown in FIG. 2, the CPU 102 determines that the degree of difficulty of motion blur addition is low when the structural similarity (SSIM) is 0.95 or more, for example. To do.

  In addition, in this embodiment, the round function is used in the above-mentioned formula (1). This represents the reference of the pixel value at the nearest neighbor (the pixel closest to the position) at the position moved according to the motion vector. However, the present invention is not limited to this, and if an increase in the amount of calculation can be allowed, for example, a virtual pixel value at the coordinate position moved by the bilinear interpolation or the bicubic interpolation method may be calculated. Further, in the present embodiment, the correlation value is calculated from the integrated value of the comparison result of all pixels, but it is not always necessary to process all pixels. For example, the vertical and horizontal 10 pixels may be sampled, and the correlation value may be calculated using 1% of the pixels.

  In the present embodiment, the shutter speed of the sensor 109 is described as the Tv value being reflected immediately, but the shutter speed actually captured by the sensor 109 may be gradually changed. Further, in the present embodiment, the device for capturing and recording by controlling the Tv value of the sensor 109 has been described, but it may be configured to electronically add motion blur to an image at the time of capturing. In this case, an optical flow is generated in real time at the time of image capturing, and the motion blur adding unit 112 adds motion blur to image data according to the optical flow. The image to which the motion blur has been added is encoded and recorded in the non-volatile memory 104. From the viewpoint of compatibility between still images and moving images, the non-volatile memory 104 reproduces coded data of images captured at high shutter speeds (high-speed Tv values) for still images and low-speed shutter speeds for moving images. Will be recorded.

  The image capturing apparatus according to the first embodiment has a function of calculating the difficulty level of motion blur addition for each frame while capturing an image at a high shutter speed, and reducing the shutter speed when the difficulty level is high. There is. As a result, when it is determined that it is difficult to add electronic motion blur and reproduce the shooting with a low shutter speed while recording a video with little motion blur, the image is shot with a slow shutter speed for the video. You can shoot and record. Therefore, the user can watch the smooth moving image with less jerkiness while suppressing the occurrence of artifacts.

(Second embodiment)
Next, a second embodiment of the present invention will be described. In the second embodiment, the shutter speed of the sensor included in the sensor unit is controlled using the signal output from the sub sensor unit different from the sensor unit. Below, in the imaging device according to the second embodiment, the points different from the above-described first embodiment will be described.

  FIG. 3A is a block diagram showing a configuration example of the image pickup apparatus according to the second embodiment. In FIG. 3A, components having the same functions as those of the components shown in FIG. 1 are designated by the same reference numerals, and duplicated description will be omitted. The image pickup apparatus according to this embodiment includes a CPU 102, a memory 103, a non-volatile memory 104, a GUI unit 105, and an image processing engine 106. Further, the image pickup apparatus according to the present embodiment further includes a sub sensor unit 301 in addition to the sensor unit 107, the optical flow generation unit 110, the difficulty level calculation unit 111, the motion blur adding unit 112, and the information storage unit 113. The modules 102 to 113 and 301 included in the image pickup apparatus are connected via a bus 101, and input / output of data between the modules is performed via the bus 101.

  The sub sensor unit 301 has a lens and a sensor that captures an image, and captures an image under a shooting condition that is independent of the sensor unit 107. That is, the sub sensor unit 301 performs imaging under imaging conditions that do not depend on the sensor unit 107. The sensor included in the sub sensor unit 301 captures an image at a high shutter speed of (1/1000) seconds, for example. Also in this embodiment, the shutter speed of the sensor 109 of the sensor unit 107, which is the main sensor, is determined by performing the process according to the flowchart shown in FIG. However, the optical flow generation unit 110 generates the optical flow from the image captured by the sub sensor unit 301, not from the image captured by the sensor 109.

  In the second embodiment, an occlusion map is used to determine the degree of difficulty of adding motion blur. The occlusion map is a map having the same resolution as the input image size and indicating whether or not it is an occlusion area. Here, the occlusion area is an area in which the motion vector cannot be calculated because the object (pixel) that should be at the end position in the motion vector calculation is hidden by the foreground object. In the following description, in the occlusion map, “0” indicates occlusion and “1” or more indicates non-occlusion.

  The occlusion map is generated by the difficulty level calculation unit 111. In the present embodiment, the difficulty level calculation unit 111 is described as a CPU (programmable circuit), but it may be a circuit that performs similar processing. FIG. 4 is a flowchart showing an example of map generation processing for generating an occlusion map. As mentioned above, the occlusion map has the same resolution as the input image size. At the coordinates (x, y) of the first frame, the motion vector expressing the motion from the second frame to the first frame is (u '(x, y), v' (x, y)).

  First, in step S401, the difficulty level calculation unit 111 initializes all values of the occlusion map Oc to “0”. Next, in step S402, the difficulty level calculation unit 111 acquires the coordinates (x, y) of the pixel of interest while scanning the image. For example, the difficulty level calculation unit 111 shifts from the upper left of the image to the right by one pixel each time the process of step S402 is executed, and when the right end of one line is reached, scans again from the left end of the line immediately below. The target pixel is set by starting.

  In step S403, the difficulty level calculation unit 111 acquires the motion vector (u ′ (x, y), v ′ (x, y)) related to the coordinates (x, y) of the pixel of interest acquired in step S402. . Next, in step S404, the difficulty level calculation unit 111 increments the value of the coordinates (round (x + u '(x, y)), round (y + v' (x, y))) of the occlusion map by one. That is, the difficulty level calculation unit 111 increments the coordinate value of the occlusion map corresponding to the pixel for which the motion vector (u '(x, y), v' (x, y)) has been calculated by one.

  In step S405, the difficulty level calculation unit 111 determines whether or not scanning of all the pixels of the image has been completed. For example, when scanning is performed from the upper left of the image to the right, and when the right end of one line is reached and scanning is performed again from the left end of the line below to the right, scanning is performed by the difficulty calculation unit 111. It is determined whether or not the process of setting the lower right pixel as the pixel of interest has been completed. The difficulty level calculation unit 111 ends the map generation process when it is determined that the scanning of all the pixels of the image is completed, and transitions to step S402 when it is determined that the scanning of all the pixels of the image is not completed. Then, the processing from step S402 onward is executed.

  As described above, the occlusion area is an area in which the motion vector cannot be calculated because the object (pixel) that should be at the end point in the motion vector calculation is hidden by the foreground object. In the occlusion map generated as shown in FIG. 4, the larger the value “0”, the more occlusion, that is, the correct motion vector cannot be calculated. Therefore, it can be determined that the greater the value “0” in the occlusion map, the higher the degree of difficulty in adding motion blur. In the present embodiment, in step S203 shown in FIG. 2, the CPU 102 determines that the degree of difficulty in providing motion blur is high when the number of values “0” in the occlusion map exceeds 5% of the whole.

  In the above description, the image is captured by the sub sensor, but the present invention is not limited to this. For example, as shown in FIG. 3B, an optical flow sensor 302 that can directly generate an optical flow from an optical signal may be used. In this case, the optical flow generation unit is unnecessary, and the difficulty level calculation unit 111 calculates the difficulty level of motion blur addition using the optical flow generated and output by the optical flow sensor 302.

  As described above, even when the ratio of the occlusion area is regarded as the degree of difficulty of adding motion blur, the same effect as that of the first embodiment can be obtained, and the user can suppress the occurrence of artifacts and smooth the jerkiness. It becomes possible to watch the video.

  Note that the output result of the sub-sensor unit 301 may be used to calculate the difficulty level of adding motion blur from the correlation of images as in the first embodiment. Also in this case, similarly to the first embodiment, the user can obtain the effect of suppressing the occurrence of artifacts and being able to view smooth moving images with less jerkiness. At this time, since the Tv value in the sub sensor unit 301 is fixed, the optical flow can be generated from the image of the same image quality without depending on the Tv value of the sensor 109. Therefore, whether the Tv value of the sensor 109 is switched from the high-speed Tv value to the low-speed Tv value or when the low-speed Tv value is switched to the high-speed Tv value, determination using the difficulty level of motion blur addition calculated under the same imaging condition is performed. It is possible to improve the stability of the judgment.

(Third Embodiment)
Next, a third embodiment of the present invention will be described. The configuration of the image pickup apparatus according to the third embodiment is similar to that of the image pickup apparatus according to the first embodiment described above, and thus the description thereof will be omitted. In the following, in the control of the image pickup apparatus according to the third embodiment, points different from the above-described first embodiment will be described. FIG. 5 is a flowchart showing a control example relating to determination of the Tv value of the sensor 109 in the image pickup apparatus according to the third embodiment. The operation of each step in the flowchart shown in FIG. 5 is assumed to be executed by the CPU 102 of the image pickup apparatus.

  First, in step S501, the CPU 102, similar to step S201 shown in FIG. 2, obtains a Tv value with a high shutter speed (high speed Tv value) and a Tv value with a low shutter speed (low speed Tv value). It is acquired from the storage unit 113. Next, in step S502, the CPU 102 acquires information on the mode set in the imaging device from the information storage unit 113. In the present embodiment, the information indicating whether the set mode is the low jerkiness priority mode or the sharpness priority mode is acquired. Note that the processes of steps S501 and S502 are in no particular order, and the process of step S501 may be performed after the process of step S502, or the processes of steps S501 and S502 may be performed in parallel.

  Here, the low jerkiness priority mode is a mode of a low speed shutter that suppresses jerkiness by capturing an image at a low shutter speed for moving images. The sharpness priority mode is a high-speed shutter mode in which image sharpness is prioritized by capturing an image at a high shutter speed for still images. The mode is previously specified by the user using the GUI unit 105, and information indicating the specified mode is stored in the information storage unit 113.

  Next, in step S <b> 503, the CPU 102 acquires the difficulty level of motion blur addition, which is calculated by the difficulty level calculation unit 111 when electronically adding motion blur to a captured image. Next, in step S504, the CPU 102 compares the degree of difficulty of motion blur addition acquired in step S503 with a predetermined criterion (threshold value) to determine whether or not the degree of difficulty of motion blur addition is low. To judge. If the CPU 102 determines that the degree of difficulty of adding motion blur is low, the process proceeds to step S506, and if the degree of difficulty of applying motion blur is high, the process proceeds to step S505.

  When the CPU 102 determines that the degree of difficulty of adding motion blur is high, the CPU 102 makes a mode determination in step S505, based on the information acquired in step S502. When the CPU 102 determines that the set mode is the sharpness priority mode, the process proceeds to step S506, and when the CPU 102 determines that the set mode is the low jerkiness mode, the process proceeds to step S507.

  When the CPU 102 determines that the degree of difficulty of adding motion blur is low or the set mode is the sharpness priority mode, the CPU 102 sets the shutter speed of the sensor 109 to the high-speed Tv value in step S506. . In addition, in step S507 transitioning when the CPU 102 determines that the degree of difficulty in providing motion blur is high and the set mode is the low jerkiness mode, the CPU 102 sets the shutter speed of the sensor 109 to the low speed Tv value. Set.

  In the above description, on the assumption that the motion blur is electronically added in the post-processing, the low jerkiness priority mode or the sharpness priority mode is configured to be selected by the user, but the mode selected by the user is It is not limited. For example, the configuration may be selected from four modes including a Tv value fixed mode and an Av value fixed mode. The Tv value fixing mode is a mode in which the Tv value is fixed and the Av value and the sensitivity are changed, and the Av value fixing mode is a mode in which the Av value is fixed and the Tv value and the sensitivity are changed. When the Tv value fixing mode and the Av value fixing mode are selected, the process of the flowchart shown in FIG. 5 is not executed and each value calculated by the automatic exposure calculation formula is set.

  In the third embodiment, the difficulty level calculation unit 111 calculates the degree of difficulty of adding motion blur using the formula shown below. The difficulty level calculation unit 111 will be described as a circuit having a function of calculating the following formula, but the CPU (programmable circuit) may perform any software processing. When it is realized by software processing, the CPU 102 can also execute it.

  The function shown in Expression (4) is a function that compares the difference between two variables and returns the square of the difference when the absolute value of the difference is larger than the threshold value th, and returns 0 otherwise. In this embodiment, the threshold value th is set to 3, but is not limited to this. However, it is preferably about 0.1% to 0.5% of the width of the input image.

  Expression (5) is an expression in which the magnitude of the difference between motion vectors corresponding to adjacent pixels is calculated using the function shown in Expression (4), and the results are integrated and normalized. The variables and symbols in Expression (5) are the same as those described in the first embodiment. A large integrated value P of the magnitudes of the motion vector differences calculated by the equation (5) indicates that the difference between the adjacent motion vectors is large. Generally, in the optical flow, the motion vectors corresponding to adjacent pixels tend to be similar (small difference). Therefore, when the integrated value P is large, it is highly likely that motion detection in the image has failed, and It is difficult to detect the motion because the motion vector of V is complicated and varies. Therefore, in the present embodiment, in step S504, the CPU 102 determines that the degree of difficulty in adding motion blur is low, for example, when the integrated value P is 2 or less.

  By calculating the magnitude of the difference between the motion vectors included in the optical flow as in the present embodiment, it is possible to calculate the variation in motion, that is, the difficulty level of motion blur addition. The method of calculating the magnitude of the difference between the motion vectors and calculating the difficulty level of motion blur addition is not limited to the example described above. For example, the integrated value P ′ of the magnitude of the difference between the motion vectors may be calculated as the degree of difficulty of adding motion blur, using the following formula.

An important point in this embodiment is that the magnitude of the difference between the motion vector corresponding to the pixel of interest and the neighboring motion vector is integrated. However, the magnitude of the difference here indicates a norm or a power of the norm.
Further, the difficulty level of adding motion blur may be calculated using the following formula.

  In Expression (8), the variance of the motion vector of the entire image, that is, the degree of variation of the motion vector is directly calculated. Even if the variation of the motion vector is large, it is possible to determine that the motion is difficult to detect and the motion blur is difficult to apply. For example, when the variance var calculated by the equation (8) exceeds 400, it can be determined that the degree of difficulty in adding motion blur is high.

  In the above-described first and second embodiments, when the degree of difficulty in adding motion blur is high, the low speed Tv value is set as the shutter speed of the sensor for moving images. However, there are times when the user wants to prioritize recording of still images over moving images. Therefore, in the third embodiment, the user sets the mode in advance, and the mode is determined in the process of determining the Tv value. This makes it possible to select which of the high speed Tv value and the low speed Tv value to set when the degree of difficulty in adding motion blur is high, according to the user's judgment, thereby enhancing convenience. In this case, similar to the first and second embodiments, the same effect can be obtained even if the processing is performed by regarding the image correlation and the occlusion area ratio as the difficulty level of the motion blur addition. .

  Further, in the third embodiment, the variation in motion is calculated, but the difficulty level of motion blur can also be calculated by directly analyzing the motion information in this way. Therefore, if the information set by the user is not used, the same effect as that of the first embodiment can be obtained, and by using the information indicating the mode set by the user, the convenience for the user is further enhanced.

(Other Embodiments of the Present Invention)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  It should be noted that each of the above-described embodiments is merely an example of the embodiment in carrying out the present invention, and the technical scope of the present invention should not be limitedly interpreted by these. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

101: bus 102: CPU 103: memory 104: non-volatile memory 105: GUI unit 107: sensor unit 109: sensor 110: optical flow generation unit 111: difficulty calculation unit 112: motion blur addition unit 113: information storage unit 301: sub Sensor unit 302: Optical flow sensor

Claims (14)

A first imaging means for imaging at a set shutter speed;
An acquisition unit that acquires information regarding movement between images from continuously captured images;
A difficulty level calculating means for calculating a level of difficulty related to adding motion blur to a captured image based on the acquired information related to the movement;
When the calculated difficulty level is low with respect to a preset judgment criterion, the first image pickup means takes an image at the first shutter speed, and when the calculated difficulty level is high, the first shutter speed is reached. An image pickup apparatus comprising: a control unit that causes the first image pickup unit to take an image at a second shutter speed that is slower than the above.   The image pickup apparatus according to claim 1, wherein the information related to the movement is an optical flow acquired from the continuously picked-up images.   The control means causes the first image capturing means to capture an image at a shutter speed according to a mode set in the image capturing device when the degree of difficulty is high with respect to the criterion. The imaging device according to 1 or 2.   When the set mode is a low-speed shutter mode, the control unit causes the first imaging unit to capture an image at the second shutter speed, and the set mode is a high-speed shutter mode. 4. The image pickup apparatus according to claim 3, wherein the first image pickup unit takes an image at a third shutter speed that is faster than the second shutter speed. The difficulty level calculation means calculates, as the difficulty level, a correlation between an image generated by using the information related to the movement and a captured image and a corresponding image,
The image pickup apparatus according to claim 1, wherein the control unit determines that the difficulty level is low when the calculated correlation is higher than the determination criterion. The difficulty level calculation means calculates a ratio of an occlusion area in a captured image as the difficulty level,
The imaging device according to claim 1, wherein the control unit determines that the difficulty level is low when the calculated proportion of the occlusion area is lower than the determination criterion. The information related to the movement is an optical flow acquired from the continuously captured images,
The difficulty level calculation means calculates, as the difficulty level, an integrated value of the magnitude of the difference between motion vectors forming the optical flow,
The image pickup apparatus according to claim 1, wherein the control unit determines that the degree of difficulty is low when the calculated integrated value is smaller than the determination criterion. The information related to the movement is an optical flow acquired from the continuously captured images,
The difficulty level calculation means calculates a variation in motion vectors forming the optical flow as the difficulty level,
The image pickup apparatus according to claim 1, wherein the control unit determines that the degree of difficulty is low when the calculated variation is smaller than the determination criterion.   9. The image forming apparatus according to claim 1, further comprising an adding unit that adds motion blur to images captured continuously at the first shutter speed based on information about the movement between the images. The imaging device according to the item. And a second image pickup unit that picks up an image under a shooting condition independent of the first image pickup unit,
The image capturing apparatus according to claim 1, wherein the acquisition unit acquires information regarding movement between images from the image captured by the second image capturing unit. A sensor for generating and outputting the optical flow from an optical signal,
The imaging apparatus according to claim 2, wherein the difficulty level calculation means calculates the difficulty level by using the optical flow output by the sensor.   The image pickup apparatus according to claim 4, wherein the high-speed shutter mode is a mode in which the sharpness of an image is prioritized, and the low-speed shutter mode is a mode in which jerkiness is suppressed. A control method for an image pickup apparatus having an image pickup means for picking up an image at a set shutter speed,
An acquisition step of acquiring information relating to movement between images from continuously captured images;
A difficulty level calculating step of calculating a difficulty level related to motion blur addition to a captured image based on the acquired information related to the movement;
When the calculated difficulty level is lower than a preset criterion, the image pickup means takes an image at the first shutter speed, and when the calculated difficulty level is higher, the speed is lower than the first shutter speed. And a control step of causing the image capturing unit to capture an image at a second shutter speed. In a computer of an image pickup device having an image pickup means for picking up an image at a set shutter speed,
An acquisition step of acquiring information related to movement between images from continuously captured images;
A difficulty level calculating step of calculating a difficulty level of motion blur addition to the captured image based on the acquired information on the movement;
When the calculated difficulty level is lower than a preset criterion, the image pickup means takes an image at the first shutter speed, and when the calculated difficulty level is higher, the speed is lower than the first shutter speed. And a control step of causing the image capturing unit to capture an image at a second shutter speed.

Images