JP2020150448A - Image pickup apparatus, control method therefor, program, and storage medium - Google Patents

Abstract

To provide an image pickup apparatus which can easily confirm a state of blur in each region of a main subject, and an appropriate panning speed of a camera.SOLUTION: The image pickup apparatus includes: an acquisition unit that acquires a first captured image obtained by the first imaging according to a first imaging parameter, and motion information on a subject in the first captured image; a setting unit that sets a second imaging parameter independently of the first imaging parameter; an estimation unit that estimates a motion blur of the subject in a second captured image which is obtained when the second imaging is performed according to the second imaging parameter, by using the motion information and the second imaging parameter; and a notification unit that notifies the estimated motion blur. The notification unit changes a display form in each partial region on the basis of at least one of a magnitude and an orientation of the motion blur in each of the partial regions of the image, and notifies the motion blur.SELECTED DRAWING: Figure 9

Description

The present invention relates to a technique for notifying subject blurring that occurs in an image to be captured.

Conventionally, a shooting method called panning that expresses the sense of speed of the main subject by causing blurring in the background area other than the main subject by shooting while panning the camera following the moving main subject is known. Has been done. In order to express the sense of speed of the main subject in panning, shooting is performed with the moving speed of the main subject and the panning speed of the camera matched, and the main subject is not blurred and is placed in the background area. It is important to cause blurring only.

In this regard, Patent Document 1 discloses a technique for displaying a UI so that the photographer can grasp an appropriate panning speed of the camera as an auxiliary display for a photographer who performs panning photography. Specifically, the photographer can grasp whether the panning speed of the camera is too fast or too slow according to the display positional relationship of the icon display indicating the panning state of the camera.

Japanese Unexamined Patent Publication No. 2016-220024

However, in the case of the display method of Patent Document 1, it is possible to grasp whether or not the panning speed of the camera is appropriate for the entire main subject area by displaying the icon. However, when the speed and direction of movement are different in each part of the main subject area, there is a problem that it cannot be determined whether or not the panning speed of the camera is appropriate for each part area.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an imaging device capable of easily confirming a blur state in each region of a main subject and an appropriate panning speed of a camera. That is.

The imaging apparatus according to the present invention includes an acquisition means for acquiring a first captured image obtained by the first imaging with the first imaging parameter and motion information of a subject in the first captured image, and the first imaging device. Obtained when the second imaging is performed with the second imaging parameter by using the setting means for setting the second imaging parameter independently of the imaging parameter and the motion information and the second imaging parameter. A means for estimating the motion blur of the subject in the second captured image and a notification means for notifying the estimated motion blur are provided, and the notification means has a magnitude of the motion blur for each partial region of the image. It is characterized in that the display form is changed for each of the partial regions to notify the motion blur based on at least one of the directions.

According to the present invention, it is possible to provide an imaging device capable of easily confirming a blur state in each region of a main subject and an appropriate panning speed of a camera.

The figure which shows the structure of the imaging apparatus in 1st Embodiment. The figure which shows the imaging process flow of the imaging apparatus in 1st Embodiment. The figure which shows the structure of the motion blur notification image generation part in 1st Embodiment. The figure which shows the processing flow which generates the motion blur notification image in 1st Embodiment. The figure which shows the preparatory photograph image and the motion vector in 1st Embodiment. The figure which shows the processing flow which calculates the motion vector in 1st Embodiment. The figure which shows the calculation method of the motion vector in 1st Embodiment. The figure which shows the motion vector and the conversion motion blur in the 1st Embodiment. The figure which shows the motion blur notification method in 1st Embodiment. The figure which shows the structure of the motion blur notification image generation part in 2nd Embodiment. The figure which shows the processing flow which generates the motion blur notification image in 2nd Embodiment. The figure which shows the motion blur notification method in 2nd Embodiment. The figure which shows the structure of the motion blur notification image generation part in 3rd Embodiment. The figure which shows the processing flow which generates the motion blur notification image in 3rd Embodiment. The figure which shows the preparatory photograph image and the motion vector in 3rd Embodiment. The figure which shows the absolute value of the motion vector in an image and the frequency thereof in 3rd Embodiment. The figure which shows the motion blur notification method in 3rd Embodiment. The figure which shows the structure of the image pickup apparatus in 4th Embodiment. The figure which shows the structure of the motion blur notification image generation part in 4th Embodiment. The figure which shows the processing flow which generates the motion blur notification image in 4th Embodiment. The figure which shows the structure of the image pickup apparatus in 5th Embodiment. The figure which shows the structure of the motion blur notification image generation part in 5th Embodiment. The figure which shows the processing flow which generates the motion blur notification image in 5th Embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate description is omitted.

(First Embodiment)
In the first embodiment of the present invention, the motion information of the subject calculated from the image and the motion information of the image pickup apparatus main body detected by using the angular velocity sensor provided in the image pickup apparatus during the preparatory shooting such as live view. Based on the comparison result, the motion blur is notified.

FIG. 1 is a block diagram showing a configuration of an image pickup apparatus 100 according to a first embodiment of the present invention.

In FIG. 1, the control unit 101 is, for example, a CPU, and reads a control program for each block included in the image pickup apparatus 100 from a ROM 102 described later, expands the control program into a RAM 103 described later, and executes the control program. As a result, the control unit 101 controls the operation of each block included in the image pickup apparatus 100.

The ROM 102 is a non-volatile memory that can be electrically erased and recorded, and stores, in addition to the operation program of each block included in the image pickup apparatus 100, parameters and the like necessary for the operation of each block.

The RAM 103 is a rewritable volatile memory, and is used for developing a program executed by the control unit 101 or the like, temporarily storing data generated by the operation of each block included in the image pickup apparatus 100, or the like.

The optical system (photographing lens) 104 is composed of a lens group including a zoom lens and a focus lens, and forms a subject image on an imaging surface of an imaging unit 105 described later.

The image pickup unit 105 is provided with an image pickup element such as a CCD or a CMOS sensor, and the optical image formed on the image pickup surface of the image pickup unit 105 by the optical system 104 is photoelectrically converted, and the obtained analog image signal is A / D. Output to the conversion unit 106.

The A / D conversion unit 106 converts the input analog image signal into digital image data. The digital image data output from the A / D conversion unit 106 is temporarily stored in the RAM 103.

The image processing unit 107 applies various image processing such as white balance adjustment, color interpolation, and gamma processing to the image data stored in the RAM 103. Further, the image processing unit 107 includes a motion blur notification image generation unit 300, which will be described later, and superimposes an image plane on which the motion blur can be easily confirmed on the image stored in the RAM 103 to generate a motion blur notification image. To do.

The recording unit 108 is a removable memory card or the like. The recording unit 108 records the image data processed by the image processing unit 107 as a recorded image via the RAM 103.

The display unit 109 is a display device such as an LCD, and displays an image stored in the RAM 103, an image recorded in the recording unit 108, an operation user interface for receiving an instruction from the user, and the like. In addition, the display unit 109 displays an image captured by the imaging unit 105 for composition adjustment and the like during the preparatory shooting. The instruction input unit 110 is a touch panel, a mouse, or the like, and inputs an instruction by the user.

The angular velocity sensor 111 is, for example, a gyro sensor or the like, and periodically detects the angular velocity indicating the movement amount and the movement direction of the image pickup apparatus 100, converts it into an electric signal, and outputs it to the image processing unit 107. The configuration and basic operation of the image pickup apparatus 100 have been described above.

Next, the processing of the image pickup apparatus 100 will be described with reference to the flowchart of FIG. Each step of this flowchart is executed by each part of the image pickup apparatus 100 according to the instruction of the control unit 101.

When the user turns on the power of the image pickup apparatus 100, in step S201, the control unit 101 controls the optical system 104 and the image pickup unit 105 in response to the power of the digital camera 100 being turned on to start the preparatory shooting. During the preparatory imaging period, the imaging device 100 images sequentially images and sequentially acquires them, and the acquired captured images are displayed on the display device of the display unit 109. The user can adjust the composition while checking the preparatory shot images that are displayed sequentially. The processes of steps S202, S203, S204, S205 and S206 described later are performed during the preparatory imaging period.

In step S202, the user sets the exposure time at the time of main shooting by using the instruction input unit 110. The exposure time at the time of main shooting may be automatically set by the control unit 101.

In step S203, the control unit 101 determines whether the motion blur notification is set to ON or OFF. The setting of ON or OFF of the motion blur notification is, for example, set by the user using the instruction input unit 110, and once set, the set value is retained.

If the motion blur notification is set to ON in step S203, the process proceeds to step S204, and if the motion blur notification is set to OFF, the process proceeds to step S206. Then, in step S206, the preparatory shooting image is displayed on the display unit 109.

In step S204, the control unit 101 determines whether or not the angular velocity sensor 111 has detected the movement of the image pickup device 100 in a certain direction, that is, the panning operation of the camera by the photographer. When the panning operation is detected, it is determined that the preparatory shooting for panning has been started, and the process proceeds to step S205. Then, in step S206, the motion blur notification image is displayed on the display unit 109. If no panning operation is detected, the process proceeds to step S206. Then, in step S206, the preparatory shooting image is displayed on the display unit 109.

In step S205, the motion blur notification image generation unit 300 generates a motion blur notification image in which the motion blur notification plane is superimposed on the preparatory shooting image.

In step S207, the user presses the shutter button at the time of a shutter chance while viewing the image displayed on the display unit 109. When the motion blur notification image is displayed on the display unit 109, the user can easily check the blur state (blurring amount) of the main subject and the guideline for the appropriate panning speed of the camera during the preparatory shooting for panning. can do. If the panning speed of the camera does not match the moving speed of the main subject, the user can properly perform panning by adjusting the panning speed of the camera and then pressing the shutter button.

When the shutter button is pressed in step S207, the image pickup apparatus 100 performs the main shooting and records the main shooting image in the recording unit 108.

Next, FIG. 3 is a diagram showing a configuration of a motion blur notification image generation unit 300 included in the image processing unit 107. In FIG. 3, the motion blur notification image generation unit 300 includes a motion vector calculation unit 301, a conversion motion blur calculation unit 302, a camera motion vector calculation unit 303, a motion vector comparison unit 304, a motion blur notification plane creation unit 305, and an image superimposition unit. 306.

The operation of the motion blur notification image generation unit 300 configured as described above will be described. FIG. 4 is a flowchart showing a process in which the motion blur notification image generation unit 300 generates a motion blur notification image.

In step S401, the preparatory shooting image captured by the imaging device 100 during the preparatory shooting is input to the motion blur notification image generation unit 300. An example of the preparatory photographed image is shown in FIG. 5 (a). In the present embodiment, as shown in FIG. 5A, an example in which the dog 501 running from right to left is photographed by panning will be described. In this case, the panning operation of the camera during panning is also a movement from right to left.

In step S402, the motion vector calculation unit 301 of the motion blur notification image generation unit 300 calculates the motion vector between the images of the preparatory captured image as motion information. The motion vector represents the amount of movement of the subject in the horizontal direction and the amount of movement in the vertical direction between the images of the preparatory photographed image as a vector. The method of calculating the motion vector will be described with reference to FIGS. 6 and 7.

FIG. 6 is a flowchart showing a motion vector calculation process by the motion vector calculation unit 301. In the present embodiment, the block matching method will be described as an example of the motion vector calculation method, but the motion vector calculation method is not limited to this example, and may be, for example, the gradient method.

In step S601, two temporally adjacent preparatory shot images are input to the motion vector calculation unit 301. Then, the motion vector calculation unit 301 sets the preparatory shooting image of the Mth frame as the reference frame, and sets the preparatory shooting image of the M + 1th frame as the reference frame.

In step S602, the motion vector calculation unit 301 arranges the reference block 702 of N × N pixels in the reference frame 701 as shown in FIG. 7.

In step S603, as shown in FIG. 7, the motion vector calculation unit 301 uses (N + n) × (N + n) pixels around the point 704 at the same coordinates as the center coordinates of the reference block 702 of the reference frame 701 with respect to the reference frame 703. Is set as the search range 705.

In step S604, the motion vector calculation unit 301 correlates with the reference block 702 of the reference frame 701 while moving the reference block 706 of N × N pixels existing in the search range 705 of the reference frame 703 within the search range 705. Perform the calculation and calculate the correlation value. The correlation value is calculated based on the sum of the absolute values of the differences between frames for the pixels of the reference block 702 and the reference block 706. That is, the coordinate having the smallest value of the sum of the absolute values of the differences between frames is the coordinate having the highest correlation value. The method of calculating the correlation value is not limited to the method of obtaining the sum of the absolute values of the differences between frames, and may be, for example, a method of calculating the correlation value based on the sum of squares of the differences between frames or the normal cross-correlation value. In the example of FIG. 7, it is assumed that the position of the reference block 706 shown in FIG. 7 shows the highest correlation.

In step S605, the motion vector calculation unit 301 calculates the motion vector based on the coordinates of the reference block showing the highest correlation value obtained in step S604. In the case of the example of FIG. 7, the motion vector is obtained based on the coordinates 704 corresponding to the center coordinates of the reference block 702 of the reference frame 701 and the center coordinates of the reference block 706 in the search range 705 of the reference frame 703. That is, the inter-coordinate distance and direction from the coordinates 704 to the center coordinates of the reference block 706 are obtained as motion vectors.

In step S606, the motion vector calculation unit 301 determines whether or not the motion vector has been calculated for all the pixels of the reference frame 701. When the motion vector calculation unit 301 determines in step S606 that the motion vectors of all the pixels have not been calculated, the process returns to step S602. Then, in step S602, the reference block 702 of N × N pixels is arranged in the reference frame 701 described above centering on the pixels for which the motion vector has not been calculated. Then, in the same manner as described above, the processes of steps S603 to S605 are performed. That is, the motion vector calculation unit 301 calculates the motion vector of all the pixels of the reference frame 701 by repeating the processes from step S602 to step S605 while moving the reference block 702 in FIG. 7.

Examples of motion vectors are shown in FIGS. 5 (b) and 5 (c). In FIGS. 5 (b) and 5 (c), the case where the motion vector directions are (b) rightward and (c) leftward, respectively, is shown as an example. In the figure, the direction of the motion vector is constant, but it goes without saying that the direction of the motion vector may be different for each region.

Further, in the panning auxiliary display in the present embodiment, since the state of the motion blur in the main subject area is focused on, the auxiliary display of the motion blur in the background region is unnecessary. Therefore, when the absolute value of the calculated motion vector is controlled to be notified only in a region smaller than the absolute value of the camera motion vector representing the motion of the camera body calculated by the camera motion vector calculation unit 303. Is illustrated. The target area for motion blur notification may be determined by discriminating the main subject using a known method, and controlling the motion vector calculation process and motion blur notification process so as to be performed only on the main subject area. Good.

Here, when the ideal panning shot is performed, that is, when the moving speed of the main subject and the panning speed of the camera are completely matched, the motion vector for the region of the dog 501, which is the main subject, The size is zero. However, in reality, the panning speed of the camera is too fast or too slow with respect to the moving speed of the subject, so that movement blurring may occur with respect to the main subject.

The motion vector calculation unit 301 may calculate the motion vector for each predetermined pixel instead of calculating the motion vector of all the pixels.

Through the above processing, the motion vector calculation unit 301 calculates the motion vector between the preparatory captured images that are adjacent in time. The process of calculating the motion vector by the motion vector calculation unit 301 has been described above.

Returning to the description of FIG. 4, in step S403, the conversion motion blur calculation unit 302 sets the shooting conditions (independent shooting parameters) between the exposure time of the main shooting and the images in the preparatory shooting set in step S202 of FIG. Get the time interval.

In step S404, the conversion motion blur calculation unit 302 calculates the motion vector for each pixel calculated in step S402 based on the exposure time of the main shooting and the time interval between the images in the preparatory shooting acquired in step S403. Convert to.

A method of converting the motion blur of the main shooting will be described with reference to FIG. FIG. 8 is a diagram showing a motion vector in the preparatory shooting and a converted motion blur converted into a motion blur in the main shooting. In FIG. 8, as shooting conditions, the time interval between the images of the preparatory shooting is 1/60 second, and the exposure time of the main shooting is 1/120 second and 1/30 second as an example.

The conversion motion blur calculation unit 302 converts the motion vector for each pixel into the motion blur of the main shooting based on the conversion formulas shown in the equations (1) and (2).

CONV_GAIN = EXP_TIME / INT_TIME ... Equation (1)
CONV_BLUR = VEC_LEN x CONV_GAIN ... Equation (2)
Here, in the equation (1), CONV_GAIN indicates the conversion gain for converting the motion vector of the preparatory shooting into the motion vector of the main shooting, EXP_TIME indicates the exposure time of the main shooting, and INT_TIME indicates the interval between the images of the preparatory shooting. Indicates the time interval. Further, in the equation (2), CONV_BLUR indicates the converted motion blur of the main imaging, and VEC_LEN indicates the length of the motion vector in the preparatory imaging.

In equation (1), the conversion gain is calculated by dividing the exposure time of the main shooting by the time interval between the images of the preparatory shooting. Then, in the equation (2), the converted motion blur of the main shooting is calculated by multiplying the length of the motion vector by the converted gain.

Specifically, when the length of the motion vector in the preparatory shooting is 10 pixels as shown in FIG. 8, the converted motion blur with the exposure time of 1/120 second in the main shooting is 1/2 times the converted gain. It becomes 5 pixels. Further, the conversion motion blur with an exposure time of 1/30 second in the main shooting is 20 pixels because the conversion gain is doubled.

In step S405 of FIG. 4, the camera motion vector calculation unit 303 acquires the angular velocity information of the image pickup device 100 from the angular velocity sensor 111, converts the result into motion information on the image, and moves the image pickup device 100. Calculate the camera motion vector that represents the direction. Equations (3) and (4) show approximate conversion equations for converting the angular velocity into motion information on the image.

MOV_yaw indicates the amount of movement in the yaw direction, and MOV_pitch indicates the amount of movement in the pitch direction. Further, f indicates the focal length, ω_yaw indicates the angular velocity in the yaw direction, ω_pitch indicates the angular velocity in the pitch direction, fps indicates the frame rate of preparatory shooting, and pp indicates the pixel pitch of the imaging unit 105. In the conversion formulas shown in the formulas (3) and (4), the amount of movement on the imaging surface is calculated based on the angle and focal length moved by the image pickup device 100 at the time interval between images in the preparatory shooting, and is divided by the pixel pitch. By doing so, the amount of movement (number of moving pixels) on the image is calculated. The movement amount on the image calculated here is not a movement amount different for each pixel, but a uniform movement amount for all pixels.

The camera motion vector calculation unit 303 regards the amount of movement in the yaw direction as the amount of movement in the horizontal direction and the amount of movement in the pitch direction as the amount of movement in the vertical direction, and outputs the camera motion vector to the motion vector comparison unit 304 as a uniform camera motion vector for all pixels. To do.

The camera motion vector calculation unit 303 may calculate the camera motion vector based on the motion vector of the subject in the background region. That is, assuming that the subject in the background area is stationary, the motion vector of the subject calculated in the background area in which the start point and the end point are reversed corresponds to the camera motion vector.

In step S406 of FIG. 4, the motion vector comparison unit 304 compares the motion vector of the subject calculated by the conversion motion vector calculation unit 302 with the camera motion vector calculated by the camera motion vector calculation unit 303. The processing content here will be specifically described with reference to FIG.

Figures 5 (d) and 5 (e) show the methods of comparing motion vectors corresponding to FIGS. 5 (b) and 5 (c), respectively. In FIGS. 5D and 5E, 502 exemplifies a camera motion vector, and 503 and 504 exemplify a motion vector of a subject. The angle formed by the vector when the start points of the motion vectors are aligned and shown is represented by θ. It is assumed that the motion vector 502 of the imaging device is illustrated by normalizing its magnitude to a predetermined unit. In this embodiment, since the camera is panned following the dog running from right to left, the camera motion vector is directed to the left.

Next, a method of determining the panning speed of the camera by comparing the motion vectors will be described.

In FIG. 5 (b), the panning speed of the camera is appropriate to match the moving speed of the main subject, as shown by the fact that the dog 501, which is the main subject, is shifted to the right side of FIG. 5 (a). It corresponds to the case where it is faster than the simple panning speed. In this case, as shown in FIG. 5D, the camera motion vector 502 and the subject motion vector 503 are in opposite directions (the absolute value of the angle θ is larger than 90 degrees).

In FIG. 5 (c), the panning speed of the camera is appropriate to match the moving speed of the main subject, as shown by the fact that the dog 501, which is the main subject, is shifted to the left side of FIG. 5 (a). It corresponds to the case where the panning speed is slower than the normal panning speed. In this case, as shown in FIG. 5E, the camera motion vector 502 and the subject motion vector 503 are in the same direction (the absolute value of the angle θ is smaller than 90 degrees).

The motion vector comparison unit 304 sequentially performs the above comparison process on the motion vector of the subject corresponding to the main subject area, and pans the comparison result (whether it is faster or slower than the appropriate panning speed). Output to the creation unit 305.

In step S407 of FIG. 4, the motion blur notification plane creation unit 305 creates an image plane for notifying the motion blur based on the comparison result of the motion vectors determined in step S406.

In step S408 of FIG. 4, the image superimposing unit 306 superimposes the motion blur notification plane created in step S407 on the preparatory captured image to generate a motion blur notification image.

Here, a method of generating a motion blur notification image will be described with reference to FIG. FIG. 9 shows two examples of motion blur notification images.

9 (a) and 9 (b) show an example of notifying the motion blur by displaying the motion blur frame, and FIGS. 9 (a) and 9 (b) are shown in FIGS. 5 (b) and 5 (c), respectively. It corresponds.

Here, a method of generating a motion blur notification image by displaying a motion blur frame will be described. Based on the comparison result in step S406, the motion blur notification plane creation unit 305 determines the number of converted motion blur pixels indicating that the pixels included in each divided region are faster than the appropriate panning speed, and the appropriate panning speed. Compare with the number of pixels of conversion motion blur, which indicates that it is slower than. The motion blur notification plane creation unit 305 determines the state of the motion blur with the larger number of pixels as the state of the subject motion blur within the divided region, and creates the motion blur frame 901 based on the result. That is, the motion blur frame changes the display style depending on whether it is faster or slower than the appropriate panning speed. For example, in FIG. 9A, since the panning speed is faster than the appropriate panning speed, the motion blur frame is displayed as a solid line. Further, in FIG. 9B, since the panning speed is slower than the appropriate panning speed, the movement blur frame is displayed by a dotted line. These displays allow the user to know if the current panning speed is too fast or too slow. The motion blur notification plane creation unit 305 creates a motion blur notification plane 901 as a motion blur notification plane and superimposes it on the preparatory shooting image to generate motion blur notification images as shown in FIGS. 9A and 9B.

9 (c) and 9 (d) show an example of notifying the motion blur by highlighting (displaying form) the edge (contour) in which the motion blur has occurred, and FIGS. 9 (c) and 9 (d) show the motion blur. Corresponds to FIGS. 5 (b) and 5 (c), respectively. Here, a method of generating a motion blur notification image by highlighting the motion blur edge will be described.

The motion blur notification plane creation unit 305 detects the edge intensity of the preparatory captured image. The edge strength is calculated by using an existing method such as a sobel filter, and the description thereof will be omitted. Then, the motion blur notification plane creating unit 305 extracts pixels having an edge strength of a predetermined value or more and a converted motion blur of a predetermined value or more. As shown in 902 of FIGS. 9 (c) and 9 (d), a motion blur notification plane for highlighting the motion blur edge is created for the extracted pixels, and the motion blur notification plane is superimposed on the preparatory photographed image to obtain FIG. 9 (c). ) To generate a motion blur notification image. In the example of 902 of FIG. 9C, an example of thickening the motion blur edge is shown.

In addition, the motion blur notification plane creation unit 305 indicates that the motion blur state of the target pixel is faster than the appropriate panning speed or slower than the appropriate panning speed based on the comparison result in step S406. Is determined, and the degree of emphasis of edge enhancement is changed according to the result. For example, in FIG. 9C, since the panning speed is faster than the appropriate panning speed, the motion blur edge is displayed thick with a solid line. Further, in FIG. 9D, since the panning speed is slower than the appropriate panning speed, the motion blur edge is displayed thick with a dotted line.

As another example of the highlighting method, there is a highlighting method in which pixels having an edge intensity of a predetermined value or more and a conversion motion blur of a predetermined value or more are extracted and the extracted pixels are colored. In this case, based on the comparison result of the motion vectors, the color may be changed depending on whether the panning speed is faster or slower than the appropriate value.

The process of generating the motion blur notification image by the motion blur notification image generation unit 300 has been described above.

In the present embodiment, two examples of motion blur frame display and motion blur edge highlighting have been described as motion blur notification methods, but the motion blur notification method is not limited to this. For example, an area where motion blur occurs may be highlighted, including a flat area. Specifically, the motion blur notification plane creation unit 305 highlights pixels whose converted motion blur for each pixel is equal to or greater than a predetermined value, in which the color is changed according to the direction of the motion blur. By highlighting not only the edge region but also the region other than the edge region in this way, the entire subject is highlighted, so that it becomes easier to confirm the movement blur.

Further, as a method of motion blur notification, it may be controlled so that motion blur notification is not performed in a region where the direction of the motion vector of the camera and the direction of the motion vector of the subject are different by a predetermined angle or more. By doing so, the photographer pays attention to the camera when taking a panning shot of a subject whose movement direction is different between regions, such as the torso and limbs of a running person. The motion blur notification is displayed only in the area, which makes it easier for the photographer to check the motion blur in the area of interest.

In the present embodiment, as a method of notifying the motion blur, an example of displaying the motion blur to be notified to the display unit 109 has been described, but the method of notifying the motion blur is not limited to this. For example, motion blur may be notified by sound. Specifically, the motion blur notification sound is generated when the ratio of the number of pixels showing the converted motion blur of a predetermined value or more to the entire screen is equal to or more than the predetermined ratio among the converted motion blurs for each pixel. In addition, the notification sound to be generated is changed based on the comparison result of the direction of motion blur.

(Second Embodiment)
Hereinafter, the imaging device according to the second embodiment of the present invention will be described with reference to FIG. In the first embodiment, a case where the notification of motion blur is controlled by comparing the directions of the motion vector of the subject and the motion vector of the camera body has been described. On the other hand, in the second embodiment, a case where the notification of the motion blur is controlled based on the direction of the motion vector of the subject will be described. Elements having the same functions as those of the first embodiment are designated by the same reference numerals as those of the first embodiment, and the description thereof will be omitted.

FIG. 10 is a diagram showing the configuration of the motion blur notification image generation unit 300 in the second embodiment. The motion blur notification image generation unit 300 includes a motion vector calculation unit 301, a conversion motion blur calculation unit 302, a motion blur notification plane creation unit 305, and an image superimposition unit 306. Compared with the configuration shown in FIG. 3 in the first embodiment, the configuration for acquiring camera motion information is excluded.

Since the processing flow of the image pickup apparatus 100 is the same as that of the first embodiment shown in FIG. 2, the description thereof will be omitted. However, since the processing content of the motion blur notification image generation unit 300 to generate the motion blur notification image is different from that of the first embodiment, the content will be described.

FIG. 11 is a flowchart of a process in which the motion blur notification image generation unit 300 generates a motion blur notification image. Since the processing in each step of steps S1101 to S1104 of FIG. 11 is the same as the processing content in each step of steps S401 to S404 of FIG. 4 in the first embodiment, the description thereof will be omitted.

In step S1105 of FIG. 11, the motion blur notification plane creation unit 305 creates an image plane for notifying the motion blur based on the direction of the motion vector calculated in step S1104.

In step S1106, the image superimposing unit 306 superimposes the motion blur notification plane created in step S1105 on the preparatory shooting image to generate a motion blur notification image.

Here, a method of generating a motion blur notification image will be described with reference to FIG. 12 (a) and 12 (b) exemplify the preparatory photographed image and the calculated motion vector in the present embodiment. As shown in the figure, the case where the direction of the motion vector is different for each partial region of the main subject is taken as an example.

FIG. 12C shows an example of notifying the motion blur by displaying the motion blur frame. Here, a method of generating a motion blur notification image by displaying a motion blur frame will be described. In step S1105, the motion blur notification plane creation unit 305 calculates the ratio of the number of pixels showing the converted motion blur of a predetermined value or more to the entire divided region among the converted motion blurs for each pixel in the divided region. A motion blur frame 1201 as shown in FIG. 12C is created as a motion blur notification plane for a divided region having a ratio of a predetermined ratio or more. However, the motion blur notification plane creation unit 305 changes the display style of the motion blur frame according to the direction of the motion blur. Specifically, the motion blur notification plane creation unit 305 averages the motion blur directions (directions of the motion vectors) with respect to the converted motion blurs of a predetermined value or more among the converted motion blurs for each pixel in the divided area. The value is calculated, and the display format of the motion blur notification is determined based on the average value. For example, in FIG. 9C, the movement blur frame is displayed by a solid line when the movement blur direction is the horizontal direction, and the movement blur frame is displayed by a dotted line when the movement blur direction is in any other direction.

FIG. 12D shows an example of notifying the motion blur by highlighting the edge where the motion blur has occurred. Here, a method of generating a motion blur notification image by highlighting the motion blur edge will be described. In step S1105, the motion blur notification plane creation unit 305 detects the edge intensity of the preparatory photographed image. The edge strength is calculated by using an existing method such as a sobel filter, and the description thereof will be omitted. Then, the motion blur notification plane creating unit 305 extracts pixels having an edge strength of a predetermined value or more and a converted motion blur of a predetermined value or more. For the extracted pixels, as shown in 1202 of FIG. 12 (d), a motion blur notification plane that highlights the motion blur edge is created, and by superimposing it on the preparatory photographed image, as shown in FIG. 12 (d). Generates a motion blur notification image.

However, the motion blur notification plane creation unit 305 changes the display style of edge highlighting according to the direction of motion blur of the target pixel. For example, in the example of FIG. 12D, the edge highlighting is displayed with a solid line when the movement blur direction is the horizontal direction, and the edge highlighting is displayed with a dotted line when the movement blur direction is in the other direction.

As another example of the highlighting method, pixels having an edge strength of a predetermined value or more and a conversion motion blur of a predetermined value or more are extracted, and the extracted pixels are painted with a highlighting color such as red or blue. Can be mentioned. Also in this case, the highlighting color is changed according to the direction of motion blur. For example, control is performed so that the movement blur in the horizontal direction is colored in red, and the movement blur in the vertical direction is colored in blue. Further, the angle of the motion vector is divided into predetermined sections (for example, every 30 degrees), the color to be colored is set in advance for each divided section, and the color is colored with the preset color according to the direction of the motion vector. It may be.

The method of generating the motion blur notification image in the present embodiment has been described above. In this way, by controlling the notification display of the motion blur based on the direction of the motion blur, the user can easily confirm whether the motion blur of the subject is in a desired state. For example, in the case of panning, when motion blur in a direction different from the panning direction of the camera is detected, it can be confirmed that there is an influence such as camera shake. Further, when the direction of motion blur is different for each partial region of the subject, the direction of motion blur in the region of interest of the user can be confirmed.

(Third Embodiment)
Hereinafter, a third embodiment of the present invention will be described. In the present embodiment, during the preparatory shooting, the motion information calculated from the image is converted into the motion blur equivalent to the main shooting, and the motion blur is notified.

Since the configuration of the image pickup apparatus and the overall operation thereof in the present embodiment are the same as the configuration and overall operation of the image pickup apparatus 100 of the first embodiment shown in FIGS. 1 and 2, the description thereof will be omitted.

FIG. 13 is a diagram showing a configuration of a motion blur notification image generation unit 1300 included in the image processing unit 107. In FIG. 13, the motion blur notification image generation unit 1300 includes a motion vector calculation unit 1301, an image motion information conversion unit 1302, a region division unit 1303, a first region conversion motion blur calculation unit 1304, and a second region conversion motion blur calculation unit 1305. The first region motion blur notification plane creation unit 1306, the second region motion blur notification plane creation unit 1307, and the image superimposition unit 1308 are provided.

Next, the operation of the motion blur notification image generation unit 1300 configured as described above will be described. FIG. 14 is a flowchart showing a process in which the motion blur notification image generation unit 1300 generates a motion blur notification image.

In step S1401 of FIG. 14, the preparatory shooting image captured by the imaging device 100 during the preparatory shooting is input to the motion blur notification image generation unit 1300. In this embodiment, an example of panning a runner 1501 running to the left as shown in FIG. 15 will be described. FIG. 15A is a diagram showing a successful example of panning. No blurring can be confirmed in the runner 1501, and blurring due to camera work can be confirmed in the background area. FIG. 15B is a diagram showing a failure example of panning shot in which blur remains in the runner 1501. Further, FIG. 15C shows a shooting preparation image taken with a shorter exposure time than the main shooting.

In step S1402, the motion vector calculation unit 1301 of the motion blur notification image generation unit 1300 calculates the motion vector between the images of the preparatory captured image as motion information. The motion vector represents the amount of movement of the subject in the horizontal direction and the amount of movement in the vertical direction between the images of the preparatory shot image as a vector. Since the motion vector calculation method in the motion vector calculation unit 1301 is the same as the calculation method in the first embodiment described above with reference to FIGS. 6 and 7, the description thereof will be omitted.

Here, a representative example of the motion vector when the subject is moving as shown in FIG. 15 (c) is shown in FIG. 15 (d). In the example of FIG. 15D, the running runner 1501 is detected as a motion vector in the left direction, and the other background is detected as a motion vector in the right direction opposite to the movement direction of the camera by the camera work.

In step S1403 of FIG. 14, the image motion information conversion unit 1302 converts the angular velocity information detected by the angular velocity sensor 111 into motion information on the image. The approximate conversion formula for converting the angular velocity into motion information on the image is the same as the formulas (3) and (4) described above in the first embodiment.

The image motion information conversion unit 1302 regards the amount of movement in the yaw direction calculated by the equations (3) and (4) as the amount of movement in the horizontal direction and the amount of movement in the pitch direction as the amount of movement in the vertical direction, and all pixels are one. It is output to the area division unit 1303 as a motion vector like this.

In step S1404 of FIG. 14, the image region is divided by using the motion vector for each pixel calculated by the motion vector calculation unit 1301 and the motion vector uniform for all pixels calculated by the image motion information conversion unit 1302.

The method of image division will be described with reference to FIG. In FIG. 16, the horizontal axis represents the absolute value of the motion vector and the vertical axis represents the frequency, and is a histogram showing the frequency of each absolute value of the motion vector for each pixel calculated by the motion vector calculation unit 1301. The area division threshold value is obtained by subtracting a predetermined value WIDTH from the absolute value (absolute value of the image motion vector) of the motion vector uniform for all pixels calculated by the image motion information conversion unit 1302. The pixel having the absolute value of the motion vector equal to or less than the region division threshold is set as the first region, the pixel having the absolute value of the motion vector equal to or higher than the region division threshold is set as the second region, and each motion vector is referred to as the first region conversion motion blur calculation unit 1304. It is output to the second area conversion motion blur calculation unit 1305.

Here, the predetermined value WIDTH may be a fixed value determined by the photographer, or may be set based on the distribution of the histogram around the absolute value of the image motion vector. For example, WIDTH may be set so that the average value of the absolute value of the motion vector in the second region becomes the absolute value of the image motion vector. Further, although the horizontal axis of FIG. 16 is the absolute value of the motion vector, if it is desired to pay attention only to a certain direction, the horizontal axis may be the inner product value of the unit vector and the motion vector in a certain direction. For example, when it is desired to pay attention only to the horizontal left direction (-1,0), the inner product value of the motion vector (x, y) and (-1,0) of each pixel obtained by the motion vector calculation unit 1301 and the image motion information. The inner product value of the motion vector (x', y') and (-1,0), which is uniform for all pixels, calculated by the conversion unit 1302 may be used.

In step S1405 of FIG. 14, the first area conversion motion blur calculation unit 1304 and the second area conversion motion blur calculation unit 1305 set the exposure time of the main shooting and the image in the preparatory shooting set in step S202 of FIG. 2 as shooting conditions. Get the time interval between.

In step S1406, the first area conversion motion blur calculation unit 1304 and the second area conversion motion blur calculation unit 1305 are performed in step S1402 based on the exposure time of the main shooting and the time interval between the images in the preparatory shooting acquired in step S1403. The calculated motion vector for each pixel is converted into the motion blur of the main shooting. The method of converting to the motion blur of the main shooting is the same as the method calculated using FIG. 8 in the first embodiment. That is, CONV_BLUR is calculated using the above-mentioned equations (1) and (2).

In step S1407, the first region motion blur notification plane creation unit 1306 and the second region motion blur notification plane creation unit 1307 are image planes for notifying the motion blur based on the converted motion blur for each pixel calculated in step S1404. To create.

In step S1408, the image superimposing unit 1308 superimposes the motion blur notification plane created in step S1407 on the preparatory captured image to generate the motion blur notification image.

Here, a method of generating a motion blur notification image will be described with reference to FIG. FIG. 17 shows three examples as motion blur notification images. By displaying the motion blur notification image on the display unit 109 during the preparatory shooting, the user can easily confirm the motion blur. Further, the user can select whether to generate the broadcast image in each of the regions divided by the region division unit 1303, or to generate the broadcast image in only one of the regions. When generating a broadcast image for each, the user can easily check the degree of blurring for each region that moves differently by distinguishing the broadcast images by changing the color, size, and orientation of the broadcast images.

FIG. 17A shows an example in which motion blur is notified by displaying an icon. Here, a method of generating a motion blur notification image by displaying an icon will be described. In step S1408, the first region motion blur notification plane creation unit 1306 and the second region motion blur notification plane creation unit 1307 have a number of pixels indicating conversion motion blur within a predetermined range, which is different from the conversion motion blur for each pixel. Calculate the percentage of the entire screen. When the ratios are equal to or more than a predetermined ratio, the motion blur icon 1001 as shown in FIG. 17A is created as the first region motion blur notification plane, and the motion blur icon 1002 is created as the second region motion blur notification plane. To do. Then, by superimposing it on the preparatory photographed image, a motion blur notification image as shown in FIG. 17A is generated.

FIG. 17B shows an example of notifying the motion blur by displaying the motion blur frame. Here, a method of generating a motion blur notification image by displaying a motion blur frame will be described. In step S1408, the first region motion blur notification plane creation unit 1306 and the second region motion blur notification plane creation unit 1307 have a number of pixels indicating conversion motion blur within a predetermined range, which is different from the conversion motion blur for each pixel. Calculate the percentage of the entire screen. A motion blur frame 1003 as shown in FIG. 17B is created as a first region motion blur notification plane, and a motion blur frame 1004 is used as a motion second region blur notification for each of the divided regions having a ratio of a predetermined ratio or more. Create as a plane. Then, by superimposing it on the preparatory photographed image, a motion blur notification image as shown in FIG. 17B is generated.

FIG. 17C shows an example of notifying the motion blur by highlighting the edge where the motion blur has occurred. Here, a method of generating a motion blur notification image by highlighting the motion blur edge will be described. In step S1408, the first region motion blur notification plane creation unit 1306 and the second region motion blur notification plane creation unit 1307 detect edge intensities in each region of the preparatory photographed image. The edge strength is calculated by using an existing method such as a sobel filter, and the description thereof will be omitted. Then, the first region motion blur notification plane creation unit 1306 and the second region motion blur notification plane creation unit 1307 extract pixels whose edge intensities are equal to or greater than a predetermined value and whose converted motion blur is equal to or greater than a predetermined value. For the extracted pixels, the first region motion blur edge emphasis line as shown in 1005 of FIG. 17 (c) and the second region motion blur edge emphasis line as shown in 1006 are provided on the first region blur notification plane and the second region, respectively. Created as a region blur notification plane. By superimposing the first region blur notification plane and the second region blur notification plane on the preparatory captured image, a motion blur notification image as shown in FIG. 17C is generated. In the example of FIG. 17C, an example of changing the color depth of the motion blur edge is shown. As another example of the highlighting method, pixels having an edge strength of a predetermined value or more and a conversion motion blur of a predetermined value or more are extracted in each of the first region and the second region, and the extracted pixels are extracted. Highlights such as painting in red and blue and highlighting that changes the thickness can be mentioned.

The process of generating the motion blur notification image by the motion blur notification image generation unit 300 has been described above.

In the present embodiment, an example of notifying the motion blur when the converted motion blur is equal to or more than a predetermined value has been described, but it may be notified when the converted motion blur is equal to or less than the predetermined value. This makes it easier to confirm the lack of motion blur during the preparatory shooting period in the case of long-exposure shooting in which motion blur is desired to be expressed as a sense of motion.

In the present embodiment, three examples of motion blur icon display, motion blur frame display, and motion blur edge highlighting have been described as motion blur notification methods, but the motion blur notification method is not limited to this. For example, an area where motion blur occurs may be highlighted, including a flat area. Specifically, the first region motion blur notification plane creation unit 1306 and the second region motion blur notification plane creation unit 1307 emphasize that pixels whose converted motion blur for each pixel is equal to or greater than a predetermined value are painted in red and blue, respectively. Display. By highlighting not only the edge region but also the region other than the edge region in this way, the entire subject is highlighted, so that it becomes easier to confirm the movement blur.

In the present embodiment, an example of displaying the motion blur on the display unit 109 as a motion blur notification method has been described, but the motion blur notification method is not limited to this. For example, motion blur may be notified by sound. Specifically, the motion blur notification sound is generated when the ratio of the number of pixels showing the converted motion blur of a predetermined value or more to the entire screen is equal to or more than the predetermined ratio among the converted motion blurs for each pixel.

(Fourth Embodiment)
In the fourth embodiment of the present invention, during the preparatory shooting, the change in the focal length (focal length detection) due to the zoom operation during the shooting is converted into the motion blur equivalent to the main shooting, instead of the motion information of the imaging device, and the movement is performed. The area for notifying blur is divided.

It should be noted that the elements having the same reference numerals as those of the first to third embodiments perform the same operations and processes as those of the first to third embodiments, and the description thereof will be omitted.

FIG. 18 is a block diagram showing a configuration of an image pickup apparatus according to a fourth embodiment. In FIG. 18, the image pickup apparatus 1100 differs from the image pickup apparatus 100 of the first embodiment in the configuration of the image processing unit 1101. The configuration and operation other than the image processing unit 1101 are the same as those of the image pickup apparatus 100, and the description thereof will be omitted.

FIG. 19 is a diagram showing a configuration of a motion blur notification image generation unit 1200 included in the image processing unit 1101 in the present embodiment. The motion blur notification image generation unit 1200 includes a motion vector calculation unit 1301, an image motion information estimation unit 1201, a region division unit 1303, a first region conversion motion blur calculation unit 1304, a second region conversion motion blur calculation unit 1305, and a first region. It includes a motion blur notification plane creation unit 1306, a second region motion blur notification plane creation unit 1307, and an image superimposition unit 1308. Since the operations and processes other than the image motion information conversion unit 1201 are the same as those in the third embodiment, the description thereof will be omitted.

Next, the process of generating the motion blur notification image by the motion blur notification image generation unit 1200 will be described with reference to the flowchart of FIG. The flowchart of FIG. 20 differs from the flowchart of FIG. 14 of the third embodiment only in step S2001.

In step S2001 of FIG. 20, the image motion information estimation unit 1201 estimates motion information on the image due to the zoom operation from the change in the focal length of each pixel. Equation (5) shows an approximate conversion formula for estimating motion information on the image from changes in the focal length of each pixel.

Here, MOV_f (x, y) indicates a change in the image height direction at the coordinates (x, y) in the image. Further, f_pre indicates the focal length at the start of exposure of the preparatory photographed image, and f_post indicates the focal length at the end of exposure of the preparatory photographed image. High (x, y) indicates the image height value in pixel units at the coordinates (x, y) in the image. In the conversion formula shown in the formula (5), the amount of movement in the image height direction on the imaging surface is calculated based on the rate of change of the focal length moved at the time interval between the images in the preparatory shooting. The image motion information estimation unit 1201 outputs this as an estimated motion vector of each pixel to the area dividing unit 1202.

The area division unit 1202 compares the motion vector in the image height direction for each pixel estimated by the image motion information estimation unit 1201 with the motion vector for each pixel calculated by the motion vector calculation unit 1301, and has a pixel with a high degree of similarity and a degree of similarity. Divide the area by low pixels. For example, when the motion vector estimated by the image motion information estimation unit 1201 is V1 and the motion vector calculated by the motion vector calculation unit 1301 is V2, the region in which the inner product of the two vectors is less than the threshold is the first region and the threshold. The above region is divided into regions as a second region, and the respective motion vectors V1 and V2 are output to the first region conversion motion blur calculation unit 1304 and the second region conversion motion blur calculation unit 1305. Subsequent operations of the first region conversion motion blur calculation unit 1304, the second region conversion motion blur calculation unit 1305, the first region motion blur notification plane creation unit 1306, the second region motion blur notification plane creation unit 1307, and the image superimposition unit 1308. Is the same as that of the third embodiment, and thus the description thereof will be omitted.

The process of generating the motion blur notification image by the motion blur notification image generation unit 1200 has been described above.

In this way, the photographer can easily distinguish between the area of blurring due to the zoom operation during shooting and the area of the subject moving differently from the area.

(Fifth Embodiment)
In the fifth embodiment of the present invention, the region for notifying the motion blur is divided according to the distance between the image pickup device and the subject during the preparatory shooting.

FIG. 21 is a block diagram showing the configuration of the image pickup apparatus 1400 according to the fifth embodiment. In FIG. 21, the image pickup apparatus 1400 is different in that it has the image pickup apparatus 100 of the first embodiment, the distance measuring unit 1401, and the main subject detection unit 1402, and the configuration of the image processing unit 1403 is different. The configuration and operation other than the distance measuring unit 1401, the main subject detecting unit 1402, and the image processing unit 1403 are the same as those of the image pickup apparatus 100, and the description thereof will be omitted.

The distance measuring unit 1401 acquires distance information to the subject at the time of shooting and generates a distance map. The distance map is two-dimensional information showing the distance information to the subject in pixel units of the captured image. The existing method shall be used as the distance measuring method, and the description thereof will be omitted.

The main subject detection 1402 detects the coordinates of a specific subject, for example, the face of a person, from the captured image. It is assumed that the existing method is used for subject detection, and the description thereof will be omitted.

FIG. 22 is a diagram showing a configuration of a motion blur notification image generation unit 1500 included in the image processing unit 1403 in the fifth embodiment. The motion blur notification image generation unit 1500 includes a motion vector calculation unit 1301, a region division unit 1501, a first region conversion motion blur calculation unit 1304, a second region conversion motion blur calculation unit 1305, and a first region motion blur notification plane creation unit 1306. A second region motion blur notification plane creating unit 1307 and an image superimposing unit 1308 are provided. Since the operations and processes other than the region dividing portion 1501 are the same as those in the third embodiment, the description thereof will be omitted.

Next, the process of generating the motion blur notification image by the motion blur notification image generation unit 1500 will be described with reference to the flowchart of FIG. 23. The flowchart of FIG. 23 differs from the flowchart of FIG. 14 of the third embodiment only in steps S2301 and S2302.

In step S2301 of FIG. 23, the distance map generated by the distance measuring unit 1401 of the image pickup apparatus 1400 and the main subject coordinates detected by the main subject detecting unit 1402 are input to the area dividing unit 1501.

Next, in step S2302, the area dividing unit 1501 refers to the distance map and acquires the distance value from the image pickup device 1400 to the subject and the distance value of the main subject coordinates in each pixel. The difference between the distance value of each pixel and the distance value of the main subject coordinates is calculated, the pixel whose distance value difference is smaller than the predetermined value is set as the first region, and the pixel whose distance value difference is greater than or equal to the predetermined value is defined as the first region. The area is divided as the second area. Then, the motion vector of each region is output to the first region conversion motion blur calculation unit 1304 and the second region conversion motion blur calculation unit 1305. Subsequent operations of the first region conversion motion blur calculation unit 1304, the second region conversion motion blur calculation unit 1305, the first region motion blur notification plane creation unit 1306, the second region motion blur notification plane creation unit 1307, and the image superimposition unit 1308. Is the same as that of the third embodiment, and thus the description thereof will be omitted.

The process of generating the motion blur notification image by the motion blur notification image generation unit 1500 has been described above.

In this way, the photographer can easily distinguish between the area of the main subject and the area of the subject that is not the main subject, and can confirm the state of motion blur.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

(Other embodiments)
The present invention also supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads the program. It can also be realized by the processing to be executed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100: Imaging device, 101: Control unit, 102: ROM, 103: RAM, 104: Optical system, 105: Imaging unit, 106: A / D conversion unit, 107: Image processing unit, 108: Recording unit, 109: Display Unit, 110: Instruction input unit, 300: Motion blur notification image generation unit, 301: Motion vector calculation unit, 302: Conversion motion blur calculation unit, 303: Camera motion vector calculation unit, 304: Motion vector comparison unit, 305: Motion Blur notification plane creation unit, 306: Image superimposition unit

Claims (22)

An acquisition means for acquiring the first captured image obtained by the first imaging with the first imaging parameter and the movement information of the subject in the first captured image, and
A setting means for setting the second shooting parameter independently of the first shooting parameter, and
Using the motion information and the second imaging parameter, an estimation means for estimating the motion blur of the subject in the second captured image obtained when the second imaging is performed with the second imaging parameter, and
A notification means for notifying the estimated motion blur is provided.
The notifying means is an imaging device that notifies the motion blur by changing the display form for each of the partial regions based on at least one of the magnitude and the direction of the motion blur for each partial region of the image. The imaging device according to claim 1, wherein the estimation means estimates the motion blur based on the motion information, the first imaging parameter, and the second imaging parameter. The imaging device according to claim 1 or 2, wherein the acquisition means calculates and acquires the motion information by comparing images among a plurality of the first captured images. The estimation means determines the motion blur of the subject in the first captured image calculated based on the motion information, and the motion in the second captured image based on the difference between the first imaging parameter and the second imaging parameter. The imaging device according to any one of claims 1 to 3, wherein the motion blur in the second captured image is estimated by converting it into blur. The estimation means estimates motion blur in the second captured image based on the motion information, the time interval between the plurality of images in the first imaging, and the exposure time set as the second imaging parameter. The imaging device according to any one of claims 1 to 4, wherein the image pickup apparatus is characterized by the above. The imaging device according to any one of claims 1 to 5, wherein the notification means notifies the movement blur by displaying information corresponding to the motion blur on a display unit. Any one of claims 1 to 6, wherein the notification means notifies the movement blur by superimposing information corresponding to the movement blur on the first captured image and displaying it on a display unit. The imaging device according to the section. The second imaging according to any one of claims 1 to 7, wherein the second imaging is a main imaging, and the first imaging is a preparatory imaging performed before the main imaging. Imaging device. The imaging device according to any one of claims 1 to 8, wherein the notification means notifies the movement blur by highlighting a partial region in which the movement blur occurs. The imaging device according to claim 9, wherein the notification means notifies the movement blur by displaying a frame whose display form is changed according to the direction of the movement blur. The ninth aspect of the present invention, wherein the notification means highlights the outline of the subject for each partial region where the motion blur occurs, based on at least one of the magnitude and the direction of the motion blur. Imaging device. The imaging device according to any one of claims 9 to 11, wherein the notification means changes the degree of highlighting of a partial region in which the motion blur occurs depending on the direction of the motion blur. The imaging device according to claim 9, wherein the notification means notifies the movement blur by displaying an icon in a partial region where the movement blur occurs. The detecting means for detecting the movement of the imaging device is further provided, and the notification means is based on the result of comparing the direction of the movement of the imaging device detected by the detecting means with the direction of the motion blur. The imaging device according to any one of claims 1 to 13, wherein the display form is changed for each partial region to notify the motion blur. The notification means changes the display form for each of the partial regions based on whether the direction of movement of the image pickup apparatus and the direction of the movement blur are the same direction or the opposite direction. The imaging device according to claim 14, further comprising notifying motion blur. The imaging according to claim 14 or 15, wherein the notification means notifies the motion blur to the partial region where the magnitude of the motion blur is smaller than the magnitude of the motion of the imaging device. apparatus. It is further provided with a detecting means for detecting the movement of the imaging device, and a dividing means for dividing the captured image into the partial regions based on the amount of movement detected by the detecting means and the magnitude of the motion blur. The imaging device according to claim 1. A focal length detecting means for detecting a change in the focal length of the photographing lens is further provided, and the dividing means divides the captured image into the partial regions based on the detected change in the focal length and the amount of motion blur. The imaging apparatus according to claim 17. A distance measuring means for measuring the distance from the image pickup apparatus to the subject is further provided, and the dividing means is characterized in that the captured image is divided into the partial regions based on the distance measured information and the amount of motion blur. The imaging device according to claim 17. An acquisition step of acquiring the first captured image obtained by the first imaging with the first imaging parameter and the movement information of the subject in the first captured image, and
A setting step of setting the second shooting parameter independently of the first shooting parameter, and
Using the motion information and the second imaging parameter, an estimation step of estimating the motion blur of the subject in the second captured image obtained when the second imaging is performed with the second imaging parameter, and
It has a notification process for notifying the estimated movement blur, and
In the notification step, the image pickup apparatus is characterized in that the display form is changed for each partial region to notify the motion blur based on at least one of the magnitude and the direction of the motion blur for each partial region of the image. Control method. A program for causing a computer to execute each step of the control method according to claim 20. A computer-readable storage medium that stores a program for causing a computer to execute each step of the control method according to claim 20.